My useless websites

We recently had a competition at work where you had to create a ‘useless website’. There weren’t many rules to the contest (make it publicly accessible, SSFW, enter as many times as you like), so I decided to hedge my bets and create/submit half a dozen simple sites all using the same concept of randomly generating something.

It was a good example of disposable software as I could churn out an entire site in 10 or 15 minutes including publishing it live on github and didn’t have to worry about tests/technical debt or any such thing. It was really fun.

I ended up with a runner’s up award for the sloth site, the winner was my very talented colleague James and his ‘Potato Simulator 2015‘.

Here’s the six sites I created in a week:

pizzagenerator

Pizza Generator: randomly generate a succulent pizza (with bonus exotic mode)

Drink Tea Every Day: Australian Tea Tally

Drink Tea Every Day: Australian Tea Tally

Are you faster than a sloth?

Are you faster than a sloth?

Quote of the Day

Quote of the Day

Are you taller than a giraffe?

Are you taller than a giraffe?

Ralph says...

Ralph says…

Goodbye Evernote, Hello OneNote

I’ve been using Evernote for a couple of years now (premium for a year) to take excessive amounts of notes and rely on its local offline caching and super-fast indexing/search to never lose a thing. And I loved using it until recently.

They recently released a new version with Work Chat instant messaging built in. Whilst I do write and store notes for home and for work, I have no use for this functionality whatsoever. I don’t want to chat with someone in my note-taking program, I want to take notes!

But Evernote made it so you a) can’t disable this functionality, and b) are required to use this functionality to accomplish simple things like sharing a notebook with someone. It seems this decision has annoyed a lot of their customers and Evernote is not alone – many companies create feature bloat in their existing products – hello Firefox Hello!

In the meantime Microsoft OneNote has been made completely free to use, available on all platforms I need (Mac, Windows 8 and iOS), and the best feature is it doesn’t have any form of chat in built.

I wish more companies would create products that focus on doing one thing and doing that thing well.

I’ve been using OneNote over the last month and haven’t looked back.

Testing beyond requirements? How much is enough?

At the Brisbane Software Testers Meetup last week there was a group discussion about the requirement to test beyond requirements/acceptance criteria and if you’re doing so, how much is enough? Where do you draw the line? It came from an attendee who had a manager pull him up for a production bug that wasn’t found in testing but wasn’t in the requirements. If it wasn’t in the requirements, how could he test it?

In my opinion, testing purely against requirements or acceptance criteria is never enough. Here’s why.

Imagine you have a set of perfectly formed requirements/acceptance criteria, we’ll represent as this blue blob.

Requirements

Then you have a perfectly formed software system your team has built represented by this yellow blob

System

In a perfect, yet non-existent, world, all the requirements/acceptance criteria are covered perfectly by the system, and the system exists of only the requirements/acceptance criteria.

Requirements - System

But in the real world there’s never a perfect overlap. There’s requirements/acceptance criteria that are either missed by the system (part A), or met by the system (part B). These can both be easily verified by requirements or acceptance criteria based testing. But most importantly, there are things in your system that are not specified by any requirements or acceptance criteria (part C).

Requirements - System(1)

These things in part C often exist of requirements that have been made up (assumptions), as well as implicit and unknown requirements.

The biggest flaw about testing against requirements is that you won’t discover these things in part C as they’re not requirements! But, as shown by the example from the tester meetup, even though something may not be specified as a requirement, the business can think they’re a requirement when it effects usage.

Software development should aim to have as few assumptions, implicit and unknown requirements in a system as reasonably possible. Different businesses, systems and software have different tolerances for how much effort is spent on reducing the size of these unknowns, so there’s no one size fits all answer to how much is enough.

But there are two activities that a tester can perform and champion on a team which can drastically reduce the size of these unknown unknowns.

1 – User Story Kick-Offs: I have only worked on agile software development teams over the last number of years so all functionality that I test is developed in the form of a user story. I have found the best way to reduce the number of unknown requirements in a system is to make sure every user story is kicked-off with a BA, tester and developer (often called The Three Amigos) all present and each acceptance criterion is read aloud and understood by the three. At this point, as a tester, I like to raise items that haven’t been thought of so that these can be specified as acceptance criteria and are unlikely to either make it or not make it into the system by other means or assumptions.

2 – Exploratory Testing: As a tester on an agile team I make time to not only test the acceptance criteria and specific user stories, but to explore the system and understand how the stories fit together and to think of scenarios above and beyond what has been specified. Whilst user stories are good at capturing vertical slices of functionality, their weakness, in my opinion, is they are just a ‘slice’ of functionality and often cross-story requirements may be missed or implied. This is where exploratory testing is great for testing these assumptions and raising any issues that may arise across the system.

Summary

I don’t believe there’s a clear answer to how much testing above and beyond requirements/acceptance criteria is enough. There will always be things in a system that weren’t in the requirements and as a team we should strive to reduce the things that fall into that category as much as possible given the resources and time available. It isn’t just the testers role to either just test requirements or be solely responsible/accountable for requirements that aren’t specified, the team should own this risk.

Test your web apps in production? Stylebot can help.

I test in production way too much for my liking (more details in an upcoming blog post).

testinprod

Testing in production is risky, especially because I test in a lot of different environments and they all look the same. I found the only way I could tell which environment I was in was by looking closely at the URL. This was problematic as it led to doing things in a production environment thinking I was using a pre-production or test environment – oops.

I initially thought about putting some environment specific code/CSS into our apps that made the background colour different for each environment, but the solution was complex and it still couldn’t tell me I was using production from a glance.

I recently found the Stylebot extension for Chrome that allows you to locally tweak styles on any websites you visit. I loaded this extension and added our production sites with the background colour set to bright red, so now I immediately know I am using production as it’s bright red, be extra careful.

Stylebot Example

I’ve also set some other environments to be contrasting bright colours (purple, yellow etc.) so I am know from a quick glance what environment I am using.

I like this solution as I haven’t had to change any of our apps at all and it works in all environments: which is just what I needed.

Do you do something similar? Leave a comment below.

Intentionally Disposable Software

Update 11 Feb: Sorry, somehow comments were disabled for this post. This has been resolved now.

There’s a series of code retreats that take place each year where a group of programmers get together to work in groups to solve a problem (kata). They do this in iterations over and over again, and most importantly they delete their entire code at the end of every iteration (typically 45 minutes).

“It’s much easier to walk a new road when the old road is out of sight”

~ Drew Miller

Programmers don’t delete enough production code. Which is funny because I’ve met heaps of programmers, including myself, who love deleting code; it’s a strangely satisfying, cleansing ritual.

What if we could replicate what we do when doing these katas and delete the entire source code of a system every 6 months or a year and start again? Does that sound crazy? Can we do this?

We couldn’t do this with how we currently develop software. It’s because we build software systems that are way too complex, have features that no-one uses and are built to last way too long. We expect our software systems to last 10+ years, and we’re so anxious about long-lasting technology impacts we make overly cautious or conservative decisions that come back to bite us constantly in the years to come. We build technical debt into legacy systems that no one wants to work on. We obsess about re-usability, longevity and salvageability of every piece of code we write. We build massive overly complex regression test suites because we expect the system to work for so long and be maintained for so long we expect it to eventually deteriorate and we want a regression test suite as a ‘safety net’ when it inevitably does.

Maintaining legacy software is like painting the Sydney Harbour Bridge. You start painting it on one side and by the time you get to ‘finish’ it on the other side it’s taken you so long you now need to start repainting the original side again. Wouldn’t it be easier to just build a new bridge?

What we need is Intentionally Disposable Software. We need to build software only designed to last 6 months, a year max. As soon as we deploy it we start immediately on a replacement for it. We put bare minimum effort into maintenance as we’ll just replace what we have in Production as soon as we can: why wash up when you can use fresh paper plates for every meal? As soon as the replacement is ready, we deploy that and completely blow away the old software system. We rinse and repeat.

It’s somewhat similar to planned obsolescence but we don’t do it to annoy our customers and attempt to generate repeat purchases, we do it to refine our software system without any legacy.

We use analytics to tell us exactly what features of the system in production are used and most importantly, what features are little or never used. We don’t build those features into the replacement systems ever again so each system we build is leaner, more focused on the important things it is meant to do and does them better each time. We don’t waste any time on building or supporting unimportant things.

We don’t have time to build up technical debt. We aren’t anxious about choosing a wrong technology. Did we use AngularJS and now hate it? Never fear, we start work immediately on our new system replacement and use ReactJS (or whatever the latest/coolest framework is).

Developer happiness skyrockets! No legacy code! No technical debt! Everyone can work on the latest/best technology to get the job done and want to stick around in our organization to do just that. It’s like being a consultant without being a consultant. Because everyone has already implemented the same thing before, everyone is aware of the gotchas, so whilst people are constantly learning new technology, they’re efficient because they know what they’re actually trying to achieve. And because we’re building such a lean system we’re lean in our approach.

We do need to make sure we use open standards and have an easy way to export/import/migrate data in fast, clean ways – which is good.

The same applies to integration points, we need to be modular enough and use open standards and protocols exclusively to be able to drop out one system and replace it with another that integrates easily.

So what about testing?

If we expect a system to last six months to a year, we need just enough testing. We need just enough testing to make sure the system is built right (doesn’t break), but not too much requirements based testing around building the right system, because we know we won’t build the right system, we’ll be building a ‘more right’ replacement as soon as this one is built.

We need testing that focuses on intention of the system over implementation, because the implementation will constantly change each time we rewrite it. If we write our automated tests in a declarative style devoid of implementation detail we’ll be much better off.

Think


Scenario: Overseas customers must provide ID for expensive purchases
Given I am a non-Australian customer
When I order more than $100 of tea
Then I need to provide additional ID on checkout

Not

Scenario: Overseas customers must provide ID for expensive purchases
Given I go to the Beautiful Tea Home Page
When I click Sign InAnd I enter my username and password
And I click OK
...

Think of Disposable Software like building a house.

You want somewhere to live so you decide to build a house. You design and build a new ‘dream home’ with the best intentions but soon you realize there’s a big difference between what you actually need and what you thought you needed. There’s also some fundamental design flaws you didn’t even realize until it was built and you’re living in it, like mold in the bathroom as it has not enough air-flow and the bedrooms face the wrong direction and are constantly too hot to sleep at night. Plus life has since thrown a new baby into the mix, so 12 months later you find yourself with a house with a lot of design flaws that doesn’t meet your now much clearer, and since expanded, requirements.

So what do you do? Since you’ve invested a lot (financially and emotionally) into your house and you expected it to last ten years of more, you renovate. But the problem with renovating is that you’ve got to work around all the original design flaws, and since the house is already built it’s much more difficult and expensive to make changes to it, and also since you’re living in it, any renovation comes with risk of disruption/displacement to the occupants including an overly sensitive newborn. You find since it’s mainly custom work that the renovations you’re planning will cost nearly as much as the original house.

This is like legacy software. You can’t renovate it easily as it’s already in use, you’re constantly working around its design flaws so it’s much more difficult and costly to make changes to it. Plus it’s really hard to remove the unnecessary parts of your house by renovation.

But what’s the alternative? What if you built the house knowing that come 12 months time you could knock it down, recycle it, and build a new house knowing exactly what you want this time around. You’ll know not to face the bedrooms West. You’ll know the make the bathroom has plenty of air-flow. You’ll even include a bedroom suitable for the baby. But you don’t get too caught up in getting this house ‘perfect’ because come 12 months time you can do it all again. The house could be prefabricated so it’s much cheaper to build off-site in a construction friendly environment, and the migration involves temporarily moving some furniture from the old structure, placing the new house in place with the furniture and recycling the old. You own less objects (clutter) as you know this happens and are prepared for it. As you kids grow up their needs change so instead of doing ‘extensions’ and ‘renovations’ you simply redesign your new house which will be delivered in 12 months time.

This is like Intentionally Disposable Software.

As Alex Bundardzic wrote almost ten years ago in his fantastic Disposable Software post:

“In reality, a good software developer should be able to… deliver a disposable, short lived product that addresses only the pressing needs of the moment, and then simply disappears. No maintenance, no enhancements, no song-and-dance, nothing.”

What do you think?

Never compare your organization’s insides with another organization’s outsides

I once heard the brilliant suggestion that you should never compare your insides with another person’s outsides; because they’re not the same thing. For example, just because someone may seem happy and drive an expensive car, that’s only the outside view of that person and it doesn’t paint the full picture of that person’s insides, which you’re dangerously trying to compare with your own inner thoughts/status.

The same applies with comparing things at your organization to things you’ve heard about other organizations. Countless times, including just this week, have I heard managers and colleagues say things they’ve heard like: “Facebook don’t have testers”, “Google has 10,000+ engineers in 40 offices working on trunk” and “Flickr deploys to production 10 times a day so we can too”. These are examples of comparing our insides to other’s outsides, again.

Yes, Google may have 10,000+ engineers committing to one branch but having spoken to people who work at Google it’s not quite as amazing as it seems. For example, firstly the code-base is broken down into projects (imagine the checkout time without this), each and every change set must be code reviewed, have automated and manual tests performed against it (which can take hours/days) before it is even committed to the trunk, even before it can even be considered for a production release.

I didn’t realize it at the time but the keynote at GTAC last year captured this phenomenon perfectly:

Google from the outside (like a jet plane)Google from the inside (lots of people pushing a broken down car)

Summary

It can not only be really annoying/unhealthy for staff to constantly hear such comparisons, it can also be dangerous because doing something just because Google/Facebook/Twitter/Flickr does it without knowing the inner workings of their organizations will inevitably lead to failure when you try to do it without that context and experience.

So next time you are tempted to drop something you’ve heard from a conference or a blog post about how another company does something better than yours, or to justify that we can/should do it this way, remember, never compare your organization’s insides with another organization’s outsides.

Iterative vs Incremental Software Development

What’s the difference between ‘iterative’ and ‘incremental’ software development?

I know a lot of agile software development teams call their blocks of development time ‘iterations’ instead of ‘sprints’. Does that mean they’re doing iterative software development?

You’ve probably seen the Mona Lisa analogy by Jeff Paton that visually tries to show the difference between the two development approaches:

Incremental Development:

incrementing

Iterative Development:

iterating

But which is better?

Well, if for some (very likely) reason (lack of money, changed business conditions, change in management) we had to stop after iteration/increment one or two, which approach would yield a better outcome?

Mona Lisa

Incremental development gives us a painting of half a lady whereas iterative development gives us an outline of a lady, but both paintings really wouldn’t belong in The Louvre. Perhaps we could have just painted a smaller painting?

This is where I think the Mona Lisa art analogy falls apart. A work of art, like a book, but unlike a piece of software, has a pretty clear definition of done. An artist knows when their piece of art is done: not a single stroke more, not a single stroke less.

But I’ve never worked on a piece of software that was considered done: there’s always more functionality to add/remove/fix.

If we can recognize that software is never done, all we need to do it work out how to get it to where we want it to be (for now).

“We shall not cease from exploration, and the end of all our exploring will be to arrive where we started and know the place for the first time.”
~ T.S. Eliot

If we are driven by time to market we should internally iterate just enough so we can  release ‘increments’ fast and often, and iterate/release again and again.

If we are driven by user experience, we should internally iterate a lot to get things right, release increments only when necessary, and iterate again.

Both approaches are about iterating. Both are also about incrementing. The difference is how soon we release after how many times we iterate.

Compare the beginnings of the two dominant mobile operating systems. Google went for time to market with Android, they released an unpolished, yet feature rich, operating system quickly and made it better by iterating/incrementing again and again over time. Apple took the opposite approach: they released iOS with highly polished features relatively slowly (it took three major iOS releases to get MMS and copy & paste!) but focused on getting things right from the start.

Both approaches are different but neither are wrong: they highlight the differences between Apple and Google and their approach to developing software.

Summary

We can’t build anything without iterating to some degree: no code is written perfectly the second that it is typed or committed. Even if it looks like a company is incrementally building their software: they’re iteratively building it inside.

We can’t release anything without incrementing to some degree: no matter how small a release is, it’s still an incremental change over the last release. Some increments are bigger because they’ve already been internally iterated upon more, some are smaller as they’re less developed and will evolve over time.

So, we develop software iteratively and release incrementally in various sizes over time.

Five automated acceptance test anti-patterns

Whilst being involved with lots of people writing automated acceptance tests using tools like SpecFlow and WebDriver I’ve seen some ‘anti-patterns’ emerge that can make these tests non-deterministic (flaky), very fragile to change and less efficient to run.

Here’s five ‘anti-patterns’ I’ve seen and what you can do instead.

Anti-pattern One: Not using page-objects

Page objects are just a design pattern to ensure automated UI tests use reusable, modular code. Not using them, eg, writing WebDriver code directly in step definitions, means any changes to your UI will require updates in lots of different places instead of the one ‘page’ class.

Bad

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
Driver.FindElement(By.Id("tea-"+typeOfTea)).Click();
Driver.FindElement(By.Id("buy")).Click();
}

Better

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
     MenuPage.BuyTea(typeOfTea);
}

Complicated set up scenarios within the tests themselves

Whilst there’s a place for automated end-to-end scenarios (I call these user journies), I prefer most acceptance tests to jump straight to the point.

Bad

Scenario: Accept Visa and Mastercard for Australia
 Given I am on the home page for Australia
 And I choose the tea menu
 And I select some 'green tea'
 And I add the tea to my basket
 And I choose to checkout
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Better

This usually requires adding some special functionality to your app, but the ability for testing to ‘jump’ to certain pages with data automatically set up makes automated tests much easier to read and maintain.

Scenario: Accept Visa and Mastercard for Australia
 Given I am the checkout page for Australia
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Using complicated x-path or CSS selectors

Using element identification selectors that have long chains from the DOM in them leads to fragile tests, as any change to that chain in the DOM will break your tests.

Bad

private static readonly By TeaTypeSelector =
            By.CssSelector(
                "#input-tea-type > div > div.TeaSearchRow > div.TeaSearchCell.no > div:nth-child(2) > label");

Better

Identify by ‘id’ (unique) or ‘class’. If there’s multiple elements in a group, create a parent container and iterate through them.

private static readonly By TeaTypeSelector = By.Id("teaType");

Directly executing JavaScript

Since WebDriver can directly execute any arbitrary JavaScript, it can be tempting to bypass DOM manipulation and just run the JavaScript.

Bad

public void RemoveTea(string teaType)
{
  (driver as IJavaScriptExecutor).ExecuteScript(string.Format("viewModel.tea.types.removeTeaType(\"{0}\");", teaType));
  }

Better

It is much better to let the WebDriver control the browser elements which should fire the correct JavaScript events and call the JavaScript, as that way you avoid having your ‘test’ JavaScript in sync to your ‘real’ JavaScript.

public void RemoveTea(string teaType)
{
  driver.FindElement(By.Id("remove-"+teaType)).Click();
}

Embedding implementation detail in your features/scenarios

Acceptance test scenarios are meant to convey intention over implementation. If you start seeing things like URLs in your test scenarios you’re focusing on implementation.

Bad


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to 'http://twitter.com/beautifultea'
   And I should see a link to 'https://facebook.com/beautifultea'
 

Better

Hide implementation detail in the steps (or pages, or config) and make your scenarios about the test intention.


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to the Beautiful Tea Twitter account
   And I should see a link to the Beautiful Tea Facebook page
 

I hope you’ve enjoyed these anti-patterns. Leave a comment below if you have any of your own.

Software testers shouldn’t write code

Software testers shouldn’t write code. There I’ve said it.

“If you put too much emphasis on those [automated test] scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.”

~ Joel Spolsky on testers

I used to think that you could/should teach testers to write code (as it will make them better testers), but I’m now at a point where I think that it’s a bad idea to teach testers to code for a number of reasons:

  1. A software tester’s primary responsibility/focus should always be to test software. By including a responsibility to also write code/software takes away from that primary focus. Testers will get into a trap of sorting out their own coding issues over doing their actual job.
  2. If a software tester wants their primary focus to be writing code, they should become a software programmer. A lot of testers want to learn coding not because they’ll be a better tester, but they want to earn more money. These testers should aim to be become programmers/developers if they want to code or think they can earn more money doing that.
  3. Developing automated tests should be done as part of developing the new/changed functionality (not separately). This has numerous benefits such as choosing the best level to test at (unit, integration etc.) at the right time. This means there isn’t a separate team lagging behind the development team for test coverage.
  4. Testers are great at providing input into automated test coverage but shouldn’t be responsible for creating that coverage. A tester working with a developer to create tests is a good way to get this done.

I think the software development industry would be a lot better if we had expectations on programmers to be responsible for self-tested code using automated tests, and testers to be responsible for testing the software and testing the the automated tests. Any tester wanting to code will move towards a programming job that allows them to do that and not try to change what is expected of them in their role.

Update 19th Jan 2015: this post seems to have triggered a lot of emotion, let me clarify some things:

  • A tester having technical skills isn’t bad: the more technical skills the tester has the better – if they can interrogate a database or run a sql trace then they’ll be more efficient/effective at their job – and a tester can be technical without knowing how to code
  • I don’t consider moving from testing into programming by any means the only form of career advancement: some testers hate coding and that’s fine, other’s love coding and I think it would be beneficial for them to become a programmer if they want to code more than they test.
  • I still believe everyone should take responsibility for their own career rather than expecting their employer/boss/industry leader/blogger to do it for them (more about this here).

What is a good ratio of software developers to testers on an agile team?

The developer:tester ratio question comes up a lot and I find most, if not all, answers are “it depends”.

I won’t say “it depends” (it’s annoying). I will tell you what works for me given my extensive experience, but will provide some caveats.

I’ve worked on different agile software development teams as a tester for a number of years and I personally find a ratio of 8:1 developers to tester(s) (me) works well (that’s 4 dev-pairs if pair programming). Any less developers and I am bored; any more and I have too must to test and cycle time is in jeopardy.

Some caveats:

  • I’m an efficient tester and the 8:1 ratio works well when there’s 8 equally efficient programmers on the team – if the devs are too slow, or the user stories are too big, I get bored;
  • Everyone in the team is responsible for quality; I have to make sure that happens;
  • A story must be kicked off with the tester (me) present so I can question any assumptions/anomalies in the acceptance criteria before any code is written;
  • A story is only ready for test if the developer has demonstrated the functionality to me at their workstation (bonus points in an integrated environment) – we call this a ‘shoulder check’ – much the same way as monkeys check each others shoulders for lice;
  • A story is also only ready for test if the developer has created sufficient and passing automated test coverage including unit tests, integration tests (if appropriate) and some acceptance tests; and
  • Bug fixes take priority over new development to ensure flow.

What ratio do you find works for you?