Five automated acceptance test anti-patterns

Whilst being involved with lots of people writing automated acceptance tests using tools like SpecFlow and WebDriver I’ve seen some ‘anti-patterns’ emerge that can make these tests non-deterministic (flaky), very fragile to change and less efficient to run.

Here’s five ‘anti-patterns’ I’ve seen and what you can do instead.

Anti-pattern One: Not using page-objects

Page objects are just a design pattern to ensure automated UI tests use reusable, modular code. Not using them, eg, writing WebDriver code directly in step definitions, means any changes to your UI will require updates in lots of different places instead of the one ‘page’ class.

Bad

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
Driver.FindElement(By.Id("tea-"+typeOfTea)).Click();
Driver.FindElement(By.Id("buy")).Click();
}

Better

[When(@"I buy some '(.*)' tea")]
public void WhenIBuySomeTea(string typeOfTea)
{
     MenuPage.BuyTea(typeOfTea);
}

Complicated set up scenarios within the tests themselves

Whilst there’s a place for automated end-to-end scenarios (I call these user journies), I prefer most acceptance tests to jump straight to the point.

Bad

Scenario: Accept Visa and Mastercard for Australia
 Given I am on the home page for Australia
 And I choose the tea menu
 And I select some 'green tea'
 And I add the tea to my basket
 And I choose to checkout
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Better

This usually requires adding some special functionality to your app, but the ability for testing to ‘jump’ to certain pages with data automatically set up makes automated tests much easier to read and maintain.

Scenario: Accept Visa and Mastercard for Australia
 Given I am the checkout page for Australia
 Then I should see 'visa' is accepted
 And I should see 'mastercard' is accepted

Using complicated x-path or CSS selectors

Using element identification selectors that have long chains from the DOM in them leads to fragile tests, as any change to that chain in the DOM will break your tests.

Bad

private static readonly By TeaTypeSelector =
            By.CssSelector(
                "#input-tea-type > div > div.TeaSearchRow > div.TeaSearchCell.no > div:nth-child(2) > label");

Better

Identify by ‘id’ (unique) or ‘class’. If there’s multiple elements in a group, create a parent container and iterate through them.

private static readonly By TeaTypeSelector = By.Id("teaType");

Directly executing JavaScript

Since WebDriver can directly execute any arbitrary JavaScript, it can be tempting to bypass DOM manipulation and just run the JavaScript.

Bad

public void RemoveTea(string teaType)
{
  (driver as IJavaScriptExecutor).ExecuteScript(string.Format("viewModel.tea.types.removeTeaType(\"{0}\");", teaType));
  }

Better

It is much better to let the WebDriver control the browser elements which should fire the correct JavaScript events and call the JavaScript, as that way you avoid having your ‘test’ JavaScript in sync to your ‘real’ JavaScript.

public void RemoveTea(string teaType)
{
  driver.FindElement(By.Id("remove-"+teaType)).Click();
}

Embedding implementation detail in your features/scenarios

Acceptance test scenarios are meant to convey intention over implementation. If you start seeing things like URLs in your test scenarios you’re focusing on implementation.

Bad


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to 'http://twitter.com/beautifultea'
   And I should see a link to 'https://facebook.com/beautifultea'
 

Better

Hide implementation detail in the steps (or pages, or config) and make your scenarios about the test intention.


 Scenario: Social media links displayed on checkout page
   Given I am the checkout page for Australia
   Then I should see a link to the Beautiful Tea Twitter account
   And I should see a link to the Beautiful Tea Facebook page
 

I hope you’ve enjoyed these anti-patterns. Leave a comment below if you have any of your own.

Software testers shouldn’t write code

Software testers shouldn’t write code. There I’ve said it.

“If you put too much emphasis on those [automated test] scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.”

~ Joel Spolsky on testers

I used to think that you could/should teach testers to write code (as it will make them better testers), but I’m now at a point where I think that it’s a bad idea to teach testers to code for a number of reasons:

  1. A software tester’s primary responsibility/focus should always be to test software. By including a responsibility to also write code/software takes away from that primary focus. Testers will get into a trap of sorting out their own coding issues over doing their actual job.
  2. If a software tester wants their primary focus to be writing code, they should become a software programmer. A lot of testers want to learn coding not because they’ll be a better tester, but they want to earn more money. These testers should aim to be become programmers/developers if they want to code or think they can earn more money doing that.
  3. Developing automated tests should be done as part of developing the new/changed functionality (not separately). This has numerous benefits such as choosing the best level to test at (unit, integration etc.) at the right time. This means there isn’t a separate team lagging behind the development team for test coverage.
  4. Testers are great at providing input into automated test coverage but shouldn’t be responsible for creating that coverage. A tester working with a developer to create tests is a good way to get this done.

I think the software development industry would be a lot better if we had expectations on programmers to be responsible for self-tested code using automated tests, and testers to be responsible for testing the software and testing the the automated tests. Any tester wanting to code will move towards a programming job that allows them to do that and not try to change what is expected of them in their role.

Update 19th Jan 2015: this post seems to have triggered a lot of emotion, let me clarify some things:

  • A tester having technical skills isn’t bad: the more technical skills the tester has the better – if they can interrogate a database or run a sql trace then they’ll be more efficient/effective at their job – and a tester can be technical without knowing how to code
  • I don’t consider moving from testing into programming by any means the only form of career advancement: some testers hate coding and that’s fine, other’s love coding and I think it would be beneficial for them to become a programmer if they want to code more than they test.
  • I still believe everyone should take responsibility for their own career rather than expecting their employer/boss/industry leader/blogger to do it for them (more about this here).

What is a good ratio of software developers to testers on an agile team?

The developer:tester ratio question comes up a lot and I find most, if not all, answers are “it depends”.

I won’t say “it depends” (it’s annoying). I will tell you what works for me given my extensive experience, but will provide some caveats.

I’ve worked on different agile software development teams as a tester for a number of years and I personally find a ratio of 8:1 developers to tester(s) (me) works well (that’s 4 dev-pairs if pair programming). Any less developers and I am bored; any more and I have too must to test and cycle time is in jeopardy.

Some caveats:

  • I’m an efficient tester and the 8:1 ratio works well when there’s 8 equally efficient programmers on the team – if the devs are too slow, or the user stories are too big, I get bored;
  • Everyone in the team is responsible for quality; I have to make sure that happens;
  • A story must be kicked off with the tester (me) present so I can question any assumptions/anomalies in the acceptance criteria before any code is written;
  • A story is only ready for test if the developer has demonstrated the functionality to me at their workstation (bonus points in an integrated environment) – we call this a ‘shoulder check’ – much the same way as monkeys check each others shoulders for lice;
  • A story is also only ready for test if the developer has created sufficient and passing automated test coverage including unit tests, integration tests (if appropriate) and some acceptance tests; and
  • Bug fixes take priority over new development to ensure flow.

What ratio do you find works for you?

The value of automated testing

I recently saw an email thread discussing the value of test automation and asking how to justify it to management. It just seems such a strange concept to me. Justifying the value of something that is so essential to writing good quality software, is like justifying the value of exercise to a human body: it’s benefits are so obvious it’s almost a waste of time attempting to justify it.

But the main reason I find justifying the value of automated testing so strange is I don’t really see any viable alternative. It’s like justifying to your boss the benefit of flying from Australia to the USA and back for a conference (~30 hours return) instead of taking a ship (50-80 days).

If we have a critical bug in our production system we need to turn around a fix in less than an hour. To turnaround a fix in less than an hour we need full regression test coverage that can give us feedback that we’re all good and haven’t broken anything else in half that time. To get the same coverage that our automated regression test suite has through 30 minutes of manual test execution would require having about 184 QA staff always available to do regression testing in parallel: that’s not going to happen.

Unless you can wait days/weeks for manual regression testing, or have a sufficiently large team of QA resources always available to test, you really can’t release software quickly with confidence that it’s high quality. With automated tests you can; and that’s the value of automated testing.

Why to avoid t-shirt sizes for user story estimation

The more I work on agile software development teams who use t-shirt sizes (S,M,L,XL etc.) to estimate user stories the more I dislike this approach. Here’s why:

  • In my opinion, the most import thing about user story sizing is relativity, and t-shirt sizes are a subjective measure of relativity: someone in the team might think a large is two times as big as a small, whereas another person might think it’s three times as big. This isn’t helped by the t-shirt analogy where it’s actually hard to determine how much bigger is a large t-shirt than a small one?
  • You can’t create a single measure of team velocity unless you define a scale that converts t-shirt sizes into a numeric size so you can measure t-shirt size relativity and velocity.
  • As soon as create a scale to convert t-shirt sizes into a numeric size you’ve essentially started using story points (in a convoluted way).

TL;DR: Using t-shirt sizes for user story estimation is confusing and ultimately leads the team to using story points so just skip t-shirt sizes and use relative story points instead.

Notes from GTAC 2014 (aka The Flaky Test Conference)

I was lucky enough the attend the Google Test Automation Conference (GTAC) at Google Kirkland in Washington last week. As usual, it was a very well run conference with an interesting mix of talks and attendees.

Whilst there wasn’t an official theme this year, I personally saw two themes emerge throughout the two days: dealing with flaky tests and running automated tests on real mobile devices.

Flaky Tests

There wasn’t too many talks that didn’t mention flaky automated tests (known as ‘flakes’) at some point. Whilst there seemed to be some suggestions for dealing with flaky tests (like Facebook running new tests x times to see if they fail and classify them as flaky and assign to the owner to fix), there didn’t seem to be a lot of solutions for avoiding the creation of flaky tests in the first place which I would have liked to see.

Real Mobile Devices

The obsession of running mobile automated tests on real devices continued from last year’s conference with talks about mobile devices as a service. I personally think we’d be better spending the time and effort on making more realistic mobile emulators that we can scale rather than continuing the real device test obsession.

Key Takeaway

My key takeaway was even highly innovative companies like Google, Facebook and Netflix still struggle balancing software quality and velocity. In Australia, these companies don’t have a strong presence here, and often the IT management of smaller companies here like to say things like “Google does x” or “Facebook does y”. The problem with this is they only know these companies from the outside. Ankit Mehta’s slides at the beginning of his keynote captured this perfectly and hence were my favorite slides of the conference:

Google from the outside (like a jet plane) Google from the inside (lots of people pushing a broken down car)Jonathan Rasmusson also recently wrote about something similar on his blog:

“When we see or admire what one company is doing, we are only seeing it through the lens of the end result. And not the underlying foundations that were setup to get them there.”

Well done to the GTAC committee and Google for putting on another excellent conference. Here’s some photos I took of Seattle/Kirkland during my time there.

Lock down your browser versions if you run WebDriver tests

Recently our WebDriver tests that run in Chrome via a Windows service all suddenly stopped working even though we hadn’t made any changes to our tests. It seems Chrome had automatically updated itself on our WebDriver agents introducing a Chromium 38 bug meaning WebDriver won’t work at all (full details here and here). Getting these tests running again has been very painful, mainly due to Google not having standalone Chrome installers for any previous versions of Chrome publicly available.

If you run any WebDriver tests I highly recommend you lock down your browser versions to stop this happening to you in the future. Here’s how:

Firefox

Firefox is fantastic in this regard as they make every back version easily accessible as well as a simple way on all platforms to stop automatic upgrades. I tend to lock down to Firefox ESRs (Extended Support Releases) such as versions 24 and 31 which are listed on this comprehensive Wikipedia page.

To stop updates all you do is open preferences, advanced, update and select ‘never’.

Firefox Prevent Updates

Chrome

Chrome is a P.I.T.A. in both being able to install a previous version or lock down the currently installed one. Google prefer a Chrome web installer which always installs the latest version of Chrome, and if you want a specific version you need the alternate (offline) installer (for all users if you use a Windows service), but they only provide the latest installer. It’s hard if not impossible to find older alternate (offline) installers on the web, even oldapps.com can’t host them.

Once you have a version of Chrome on Windows that you want to keep, you need to download a group policy template, and disable automatic updates before running Chrome (so it doesn’t automatically update before you set the group policy). I won’t go into full details but you should be able to find all details here. Some sites mention using a plugin to stop updates but this doesn’t work so you’ll need to go down the group policy path.

Summary

Locking down browser versions avoids having to suddenly work out why your entire WebDriver test suite fails.

An ongoing stream of articles I find interesting

Much like the collection of quotes that I regularly update, I have now created a collection of links/articles that I find interesting at links.watirmelon.com which I will reguarly update. Like articles on this blog, these are automatically published to this follow-only twitter account: @watirmelon

My favorite article I have read lately is the U.S. Digital Services Playbook: a collection of 13 key “plays” about how to deliver great digital services. I was surprised to see this great stuff coming from a government agency!

Digital Service Plays

  1. Understand what people need
  2. Address the whole experience, from start to finish
  3. Make it simple and intuitive
  4. Build the service using agile and iterative practices
  5. Structure budgets and contracts to support delivery
  6. Assign one leader and hold that person accountable
  7. Bring in experienced teams
  8. Choose a modern technology stack
  9. Deploy in a flexible hosting environment
  10. Automate testing and deployments
  11. Manage security and privacy through reusable processes
  12. Use data to drive decisions
  13. Default to open

Enjoy.

Faking Geolocation in Selenium WebDriver with Firefox

I recently worked on some Selenium WebDriver tests that needed to provide a Geolocation to a HTML5 site so it could display some results. There’s a couple of things to keep in mind when doing this:

  1. There are two Firefox about:config preferences that you must both set in your WebDriver Firefox profile to always supply the Geolocation information to your test (instead of displaying a prompt). These are:
    1. set ‘geo.prompt.testing’ to true
    2. set ‘geo.prompt.testing.allow’ to true
  2. If you want to use a specific location you need to override an about:config preference to a JSON file containing that location. This is only supported in very recent versions of Firefox (I tested version 31). Whilst this can be manually done using a data:application/json string, Firefox using WebDriver in C# completely ignores this so the workaround is to create a JSON file with the location and reference using using a file:/// prefix
    1. Create C:\Dev\Brussels.json etc containing something like:
      {
          "status": "OK",
          "accuracy": 10.0,
          "location": {
              "lat": 50.850780,
              "lng": 4.358138,
              "latitude": 50.850780,
              "longitude": 4.358138,
              "accuracy": 10.0
          }
      }
    2. set ‘geo.wifi.uri’ to ‘file:///C:/Dev/brussels.json’ (or wherever your file is)

Once you’ve done this you should be able to test Geolocation without seeing the prompt or being able to completely override the location. Some example C# WebDriver scripts are below.


using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;

namespace GeoLocation.Tests.Acceptance
{
    [TestClass]
    public class WebDriverGeoLocation
    {
        [TestMethod]
        public void CanOverrideLocationInNewerVersionsOfFirefoxLike31()
        {
            var profile = new FirefoxProfile {EnableNativeEvents = false};
            profile.SetPreference("geo.prompt.testing", true);
            profile.SetPreference("geo.prompt.testing.allow", true);
            profile.SetPreference("geo.wifi.uri", "file:///C:/Dev/brussels.json");
            var driver = new FirefoxDriver(profile);
            driver.Navigate().GoToUrl("http://html5demos.com/geo");
            new WebDriverWait(driver, TimeSpan.FromSeconds(15)).Until(ExpectedConditions.ElementExists((By.ClassName("success"))));
            Assert.AreEqual("found you!", driver.FindElement(By.Id("status")).Text);
            driver.Close();
        }

        [TestMethod]
        public void CantOverrideLocationInOlderVersionsOfFirefoxLike24()
        {
            var profile = new FirefoxProfile { EnableNativeEvents = false };
            profile.SetPreference("geo.prompt.testing", true);
            profile.SetPreference("geo.prompt.testing.allow", true);
            var driver = new FirefoxDriver(profile);
            driver.Navigate().GoToUrl("http://html5demos.com/geo");
            new WebDriverWait(driver, TimeSpan.FromSeconds(15)).Until(ExpectedConditions.ElementExists((By.ClassName("success"))));
            Assert.AreEqual("found you!", driver.FindElement(By.Id("status")).Text);
            driver.Close();
        }
    }
}

Enjoy your Geolocation testing!