An ongoing stream of articles I find interesting

Much like the collection of quotes that I regularly update, I have now created a collection of links/articles that I find interesting at links.watirmelon.com which I will reguarly update. Like articles on this blog, these are automatically published to this follow-only twitter account: @watirmelon

My favorite article I have read lately is the U.S. Digital Services Playbook: a collection of 13 key “plays” about how to deliver great digital services. I was surprised to see this great stuff coming from a government agency!

Digital Service Plays

  1. Understand what people need
  2. Address the whole experience, from start to finish
  3. Make it simple and intuitive
  4. Build the service using agile and iterative practices
  5. Structure budgets and contracts to support delivery
  6. Assign one leader and hold that person accountable
  7. Bring in experienced teams
  8. Choose a modern technology stack
  9. Deploy in a flexible hosting environment
  10. Automate testing and deployments
  11. Manage security and privacy through reusable processes
  12. Use data to drive decisions
  13. Default to open

Enjoy.

Faking Geolocation in Selenium WebDriver with Firefox

I recently worked on some Selenium WebDriver tests that needed to provide a Geolocation to a HTML5 site so it could display some results. There’s a couple of things to keep in mind when doing this:

  1. There are two Firefox about:config preferences that you must both set in your WebDriver Firefox profile to always supply the Geolocation information to your test (instead of displaying a prompt). These are:
    1. set ‘geo.prompt.testing’ to true
    2. set ‘geo.prompt.testing.allow’ to true
  2. If you want to use a specific location you need to override an about:config preference to a JSON file containing that location. This is only supported in very recent versions of Firefox (I tested version 31). Whilst this can be manually done using a data:application/json string, Firefox using WebDriver in C# completely ignores this so the workaround is to create a JSON file with the location and reference using using a file:/// prefix
    1. Create C:\Dev\Brussels.json etc containing something like:
      {
          "status": "OK",
          "accuracy": 10.0,
          "location": {
              "lat": 50.850780,
              "lng": 4.358138,
              "latitude": 50.850780,
              "longitude": 4.358138,
              "accuracy": 10.0
          }
      }
    2. set ‘geo.wifi.uri’ to ‘file:///C:/Dev/brussels.json’ (or wherever your file is)

Once you’ve done this you should be able to test Geolocation without seeing the prompt or being able to completely override the location. Some example C# WebDriver scripts are below.


using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;

namespace GeoLocation.Tests.Acceptance
{
    [TestClass]
    public class WebDriverGeoLocation
    {
        [TestMethod]
        public void CanOverrideLocationInNewerVersionsOfFirefoxLike31()
        {
            var profile = new FirefoxProfile {EnableNativeEvents = false};
            profile.SetPreference("geo.prompt.testing", true);
            profile.SetPreference("geo.prompt.testing.allow", true);
            profile.SetPreference("geo.wifi.uri", "file:///C:/Dev/brussels.json");
            var driver = new FirefoxDriver(profile);
            driver.Navigate().GoToUrl("http://html5demos.com/geo");
            new WebDriverWait(driver, TimeSpan.FromSeconds(15)).Until(ExpectedConditions.ElementExists((By.ClassName("success"))));
            Assert.AreEqual("found you!", driver.FindElement(By.Id("status")).Text);
            driver.Close();
        }

        [TestMethod]
        public void CantOverrideLocationInOlderVersionsOfFirefoxLike24()
        {
            var profile = new FirefoxProfile { EnableNativeEvents = false };
            profile.SetPreference("geo.prompt.testing", true);
            profile.SetPreference("geo.prompt.testing.allow", true);
            var driver = new FirefoxDriver(profile);
            driver.Navigate().GoToUrl("http://html5demos.com/geo");
            new WebDriverWait(driver, TimeSpan.FromSeconds(15)).Until(ExpectedConditions.ElementExists((By.ClassName("success"))));
            Assert.AreEqual("found you!", driver.FindElement(By.Id("status")).Text);
            driver.Close();
        }
    }
}

Enjoy your Geolocation testing!

Connecting to a Windows 8.1 box from Mac OS X

I have a new Windows 8.1 machine for work and couldn’t work out how to connect to it via the Microsoft Remote Desktop Connection client on Mac OS X. No matter what I did I got the following error:

Verify Identity of Computer

When trying to work out the reason for this I stumbled across a link to this page with the solution deeply embedded in another page.

Here’s my summary:

On your Windows 8.1 box:

  1. Open the Management Console (mmc.exe)
  2. Add a Snap-In (File Menu) and choose ‘Group Policy Object Editor’ and then ‘Local Computer’ on the subsequent screen
  3. Navigate using the tree to:
    • Computer Configuration
      • Administrative Templates
        • Windows Components
          • Remote Desktop Services
            • Remote Desktop Session Host
              • Security
  4. Change the following two settings from their default:
    1. ‘Require use of specific for remote desktop (RDP) connections’ from ‘Default’ to ‘Enabled’, then select ‘RDP’ in the options pane
    2. ‘Require user authentication for remote connections by using Network Level Authentications’ to ‘Disabled’
  5. Restart the ‘Remote Desktop Service’ or restart the Windows 8.1 machine

I hope this helps you should you come across this problem yourself.

Deciding to have lots of children and lots of tests is still fun later on

I recently saw a paraphrased quote by James Bach from a testing meetup in Sydney.

Deciding to have lots of (automated) checks [sic: tests] is like deciding to have lots of children. It’s fun at first, but later…

I read it a number of times and each time I read it I disagreed with it a little more.

As a proud father of three beautiful boys, I truly believe having lots of children is fun at first AND fun later on. Sure, having lots of kids is hardest thing you’ll ever do and continues to be hard as each day goes by, but hard and fun aren’t opposites or mutually exclusive whatsoever1; I’ve actually found them to be strongly correlated (think of your funnest job: was it easy?). So don’t let anybody put you off having lots of kids ever, because they are still loads of fun later on (assuming you’re not scared of hard work). I love my boys: they’re the funnest people I know and they get funner every day.

As a developer of software, I also believe having lots of automated tests is fun later on, on the proviso that you’ve put thought into them upfront. I truly believe the only way to make sustainable software that you can change and refactor with confidence is to develop it using self-testing code. Sure, having too many automated e2e tests can be a PITA2 but I’d choose lots of automated tests over no or very few automated tests any day of the week3. Again, don’t let someone put you off having lots of automated tests: just do them right!


Addendum

I asked James Bach on Twitter about his quote (and how many children he has, the answer is one), and in the typical self-righteous context driven testing ‘community’ style I was called ‘reckless’ for choosing to have three beautiful boys with my lovely wife.

It didn’t end there with other members of the ‘community’ doing what they do4 and taking the opportunity to jump in uninvited, attack me for even wondering how someone with only one child can comment on having lots of children, and try to intimidate me by accusing me of using ‘ad-hominem’ falacies/attacks against James Bach (they like big words).

This entire episode reaffirms my choice to have nothing whatsoever to do with the context driven testing ‘community’ and anyone who associates themselves with it (which started by me deleting my twitter account so they can’t attack me or have anything to do with me).

My final word of warning to those of you who still consider yourself part of that ‘community’, a comment about ‘context-driven testing':

“I chose not to engage in those dogmatic discussions. I once had a job interview where the term context-driven led one of the devs to do some googling. I had to defend myself for affiliating as he’d found some right contentious and dogmatic stuff and wondered if I were some kind of extremist for including that term in my resume. It’s no longer in my resume, FWIW.”

[source]

Footnotes

[1] I recently read that happiness and unhappiness aren’t actually the opposite of one another: you can be both happy and unhappy at the same time.

[2] In case you didn’t know: PITA means ‘pain in the ass’, and lots of end to end tests are a pain in the ass. There’s lots of articles on here about why, the most recent one being about Salesforce.com and its 100,000 e2e tests.

[3] FWIW most codebases I have worked on have had zero to little automated tests, so I don’t think having too many automated tests is our common industry problem.

[4] It’s not hard to find examples of where members of this ‘community’ rally against and intimidate a particular person they disagree with on twitter, for examples: here, here, here, here, here, etc. I personally know a fellow tester who had a very similar negative experience to me a couple of years ago and has since distanced herself also.

5 books / 5 slides / 5 minutes

At the last Brisbane Software Testers meetup I volunteered to do a 5 minute lightning talk. Since I’ve read a lot of books lately I thought I would share what I had read and some of the key snippets and set my set a challenge of talking about 5 books using 5 slides in 5 minutes.

Unfortunately some of the other volunteers for lightning talks withdrew so I had a longer window and ended up talking way longer (including some bonus slides about Think Like A Freak).

I am keen to try this again using 5 books I have since read to see if it’s actually possible to communicate this amount of information. My slides are below and are also available in PDF format (accessible).

5 books - 5 slides - 5 minutes (1) 5 books - 5 slides - 5 minutes (2) 5 books - 5 slides - 5 minutes (3) 5 books - 5 slides - 5 minutes (4) 5 books - 5 slides - 5 minutes

 

100,000 e2e selenium tests? Sounds like a nightmare!

This story begins with a promo email I received from Sauce Labs…

“Ever wondered how an Enterprise company like Salesforce runs their QA tests? Learn about Salesforce’s inventory of 100,000 Selenium tests, how they run them at scale, and how to architect your test harness for success”

saucelabs email

100,000 end-to-end selenium tests and success in the same sentence? WTF? Sounds like a nightmare to me!

I dug further and got burnt by the molten lava: the slides confirmed my nightmare was indeed real:

Salesforce Selenium Slide

“We test end to end on almost every action.”

Ouch! (and yes, that is an uncredited image from my blog used in the completely wrong context)

But it gets worse. Salesforce have 7500 unique end-to-end WebDriver tests which are run on 10 browsers (IE6, IE7, IE8, IE9, IE10, IE11, Chrome, Firefox, Safari & PhantomJS) on 50,000 client VMs that cost multiple millions of dollars, totaling 1 million browser tests executed per day (which equals 20 selenium tests per day, per machine, or over 1 hour to execute each test).

Salesforce UI Testing Portfolio

My head explodes! (and yes, another uncredited image from this blog used out of context and with my title removed).

But surely that’s only one place right? Not everyone does this?

A few weeks later I watched David Heinemeier Hansson say this:

“We recently had a really bad bug in Basecamp where we actually lost some data for real customers and it was incredibly well tested at the unit level, and all the tests passed, and we still lost data. How the f*#% did this happen? It happened because we were so focused on driving our design from the unit test level we didn’t have any system tests for this particular thing.
…And after that, we sort of thought, wait a minute, all these unit tests are just focusing on these core objects in the system, these individual unit pieces, it doesn’t say anything about whether the whole system works.”

~ David Heinemeier Hansson – Ruby on Rails creator

and read that he had written this:

“…layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests (Which btw do not need to be so slow any more, thanks to advances in parallelization and cloud runner infrastructure).”

~ David Heinemeier Hansson – Ruby on Rails creator

I started to get very worried. David is the creator of Ruby on Rails and very well respected within the ruby community (despite being known to be very provocative and anti-intellectual: the ‘Fox News’ of the ruby world).

But here is dhh telling us to replace lower level tests with higher level ‘system’ (end to end) tests that use something like Capybara to drive a browser because unit tests didn’t find a bug and because it’s now possible to parallelize these ‘slow’ tests? Seriously?

Speed has always seen as the Achille’s heel of end to end tests because everyone knows that fast feedback is good. But parallelization solves this right? We just need 50,000 VMs like Salesforce?

No.

Firstly, parallelization of end to end tests actually introduces its own problems, such as what to do with tests that you can’t run in parallel (for example, ones that change global state of a system such as a system message that appears to all users), and it definitely makes test data management trickier. You’ll be surprised the first time you run an existing suite of sequential e2e tests in parallel, as a lot will fail for unknown reasons.

Secondly, the test feedback to someone who’s made a change still isn’t fast enough to enable confidence in making a change (by the time your app has been deployed and the parallel end-to-end tests have run; the person who made the change has most likely moved onto something else).

But the real problem with end to end tests isn’t actually speed. The real problem with end to end tests is that when end to end tests fail, most of the time you have no idea what went wrong so you spend a lot of time trying to find out why. Was it the server? Was it the deployment? Was it the data? Was it the actual test? Maybe a browser update that broke Selenium? Was the test flaky (non-deterministic or non-hermetic)?

Rachel Laycock and Chirag Doshi from ThoughtWorks explain this really well in their recent post on broken UI tests:

“…unlike unit tests, the functional tests don’t tell you what is broken or where to locate the failure in the code base. They just tell you something is broken. That something could be the test, the browser, or a race condition. There is no way to tell because functional tests, by definition of being end-to-end, test everything.”

So what’s the answer? You have David’s FUD about unit testing not catching a major bug in BaseCamp. On the other hand you need to face the issue of having a large suite of end to end tests will most likely result in you spending all your time investigating test failures instead of delivering new features quickly.

If I had to choose just one, I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week.

Why? Because it’s much easier to supplement comprehensive unit testing with human exploratory end-to-end system testing (and you should anyway!) than trying to manually verify units function from the higher system level, and it’s much easier to know why a unit test is broken as explained above. And it’s also much easier to add automated end-to-end tests later than trying to retrofit unit tests later (because your code probably won’t be testable and making it testable after-the-fact can introduce bugs).

To answer our question, let’s imagine for a minute that you were responsible for designing and building a new plane. You obviously need to test that your new plane works. You build a plane by creating parts (units), putting these together into components, and then putting all the components together to build the (hopefully) working plane (system).

If you only focused on unit tests, like David mentioned in his Basecamp example, you could be pretty confident that each piece of the plane would be have been tested well and works correctly, but wouldn’t be confident it would fly!

If you only focussed on end to end tests, you’d need to fly the plane to check the individual units and components actually work (which is expensive and slow), and even then, if/when it crashed, you’d need to examine the black-box to hopefully understand which unit or component didn’t work, as we currently do when end-to-end tests fail.

But, obviously we don’t need to choose just one. And that’s exactly what Airbus does when it’s designing and building the new Airbus A350:

As with any new plane, the early design phases were riddled with uncertainty. Would the materials be light enough and strong enough? Would the components perform as Airbus desired? Would parts fit together? Would it fly the way simulations predicted? To produce a working aircraft, Airbus had to systematically eliminate those risks using a process it calls a “testing pyramid.” The fat end of the pyramid represents the beginning, when everything is unknown. By testing materials, then components, then systems, then the aircraft as a whole, ever-greater levels of complexity can be tamed. “The idea is to answer the big questions early and the little questions later,” says Stefan Schaffrath, Airbus’s vice president for media relations.

The answer, which has been the answer all along, is to have a balanced set of automated tests across all levels, with a disciplined approach to having a larger number of smaller specific automated unit/component tests and a smaller number of larger general end-to-end automated tests to ensure all the units and components work together. (My diagram below with attribution)

Automated Testing Pyramid

Having just one level of tests, as shown by the stories above, doesn’t work (but if it did I would rather automated unit tests). Just like having a diet of just chocolate doesn’t work, nor does a diet that deprives you of anything sweet or enjoyable (but if I had to choose I would rather a diet of healthy food only than a diet of just chocolate).

Now if we could just convince Salesforce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change and stop David from continuing on his anti-unit pro-system testing anti-intellectual rampage which will result in more damage to our industry than it’s worth.