Waiting for AJAX calls in WebDriver C#

I was trying to work out how to wait for AJAX calls to complete in C# WebDriver before continuing a test.

Whilst I believe that your UI should visually indicate that AJAX activity is occurring (such as a spinner) and in this case you should be able to wait until such an indicator changes, if you don’t have a visual indicator and you use JQuery for your AJAX calls, you can use a JavaScript call to jQuery.active to determine if there are any active AJAX requests, and wait until this value is zero.

I wrapped this into a WebDriver extension method on Driver, so you can call it like this:


The actual extension method looks like this:

public static void WaitForAjax(this IWebDriver driver, int timeoutSecs = 10, bool throwException=false)
  for (var i = 0; i < timeoutSecs; i++)
    var ajaxIsComplete = (bool)(driver as IJavaScriptExecutor).ExecuteScript("return jQuery.active == 0");
    if (ajaxIsComplete) return;
  if (throwException)
    throw new Exception("WebDriver timed out waiting for AJAX call to complete");

I hope you find this helpful if you’re ever in the same situation.

Extensive post release testing is sign of an unhealthy testing process

Does your organization conduct extensive post-release testing in production environments?

If you do, then it shows you probably have an unhealthy testing process, and you’ve fallen into the “let’s just test it in production” trap.

If testing in non-production environments was reflective of production behaviour, there would be no need to do production testing at all. But often testing isn’t reflective of real production behaviour, so we test in production to mitigate the risk of things going wrong.

It’s also the case that often issues are found in a QA environment don’t appear in a local development environment.

But it makes much more sense to test in an environment as close to where the code was written as possible: it’s much cheaper, easier and more efficient to find and fix bugs early.

For example, say you were testing a feature and how it behaves across numerous times of day across numerous time zones. As you progress through different test environments this becomes increasingly difficult to test:

In a local development environment: you could fake the time and timezone to see how your application behaves.
In a CI or QA environment: you could change a single server time and restart your application to see how your application behaves under various time scenarios: not as easy as ‘faking’ the time locally but still fairly easy to do.
In a pre-production environment: you’ll probably have clustered web servers so you’ll be looking at changing something like 6 or 8 server times to test this feature. Plus it will effect anyone else utilizing this system.
In a production environment: you’ll need to wait until the actual time to test the feature as you won’t be able to change the server times in production.

Clearly it’s cheaper, easier and more efficient to test changing times in an environment closer to where the code was written.

You should aim to conduct as much testing as you can in earlier test environments and taper this off so by the time you can a change into production you’ll be confident that it’s been tested comprehensively. This probably requires some change to your testing process though.

Tests Performed per Environment

How to Remedy A ‘Test in Production’ Culture

As soon as you find an issue in a later environment, ask why wasn’t this found in an earlier environment? Ultimately ask: why can’t we reproduce this in a local environment?

Some Hypothetical Examples

Example One: our tests fail in CI because of JavaScript errors that don’t reproduce on a local development environment. Looking into this we realize this is because the JavaScript is minified in CI but not in a local development environment. We make a change to enable local development environments to run tests in minified mode which reproduces these issues.

Example Two: our tests failed in pre-production that didn’t fail in QA because pre-production has a regular back up of the production database whereas QA often gets very out of date. We schedule a task to periodically restore the QA database from a production snapshot to ensure the data is reflective.

Example Three: our tests failed in production that didn’t fail in pre-production as email wasn’t being sent in production and we couldn’t test it in pre-production/QA as we didn’t want to accidentally send real emails. We configure our QA environments to send emails, but only to a white-list of specified email addresses we use for testing to stop accidental emails. We can be confident that changes to emails are tested in QA.


It’s easy to fall into a trap of just testing things in production even though it’s much more difficult and risky: things often go wrong with real data, the consequences are more severe and it’s generally more difficult to comprehensively test in production as you can’t change or fake things as easily.

Instead of just accepting “we’ll test it in production”, try instead to ask, “how can we test this much earlier whilst being confident our changes are reflective of actual behaviour?”

You’ll be much less stressed, your testing will be much more efficient and effective, and you’ll have a healthier testing process.

Testing beyond requirements? How much is enough?

At the Brisbane Software Testers Meetup last week there was a group discussion about the requirement to test beyond requirements/acceptance criteria and if you’re doing so, how much is enough? Where do you draw the line? It came from an attendee who had a manager pull him up for a production bug that wasn’t found in testing but wasn’t in the requirements. If it wasn’t in the requirements, how could he test it?

In my opinion, testing purely against requirements or acceptance criteria is never enough. Here’s why.

Imagine you have a set of perfectly formed requirements/acceptance criteria, we’ll represent as this blue blob.


Then you have a perfectly formed software system your team has built represented by this yellow blob


In a perfect, yet non-existent, world, all the requirements/acceptance criteria are covered perfectly by the system, and the system exists of only the requirements/acceptance criteria.

Requirements - System

But in the real world there’s never a perfect overlap. There’s requirements/acceptance criteria that are either missed by the system (part A), or met by the system (part B). These can both be easily verified by requirements or acceptance criteria based testing. But most importantly, there are things in your system that are not specified by any requirements or acceptance criteria (part C).

Requirements - System(1)

These things in part C often exist of requirements that have been made up (assumptions), as well as implicit and unknown requirements.

The biggest flaw about testing against requirements is that you won’t discover these things in part C as they’re not requirements! But, as shown by the example from the tester meetup, even though something may not be specified as a requirement, the business can think they’re a requirement when it effects usage.

Software development should aim to have as few assumptions, implicit and unknown requirements in a system as reasonably possible. Different businesses, systems and software have different tolerances for how much effort is spent on reducing the size of these unknowns, so there’s no one size fits all answer to how much is enough.

But there are two activities that a tester can perform and champion on a team which can drastically reduce the size of these unknown unknowns.

1 – User Story Kick-Offs: I have only worked on agile software development teams over the last number of years so all functionality that I test is developed in the form of a user story. I have found the best way to reduce the number of unknown requirements in a system is to make sure every user story is kicked-off with a BA, tester and developer (often called The Three Amigos) all present and each acceptance criterion is read aloud and understood by the three. At this point, as a tester, I like to raise items that haven’t been thought of so that these can be specified as acceptance criteria and are unlikely to either make it or not make it into the system by other means or assumptions.

2 – Exploratory Testing: As a tester on an agile team I make time to not only test the acceptance criteria and specific user stories, but to explore the system and understand how the stories fit together and to think of scenarios above and beyond what has been specified. Whilst user stories are good at capturing vertical slices of functionality, their weakness, in my opinion, is they are just a ‘slice’ of functionality and often cross-story requirements may be missed or implied. This is where exploratory testing is great for testing these assumptions and raising any issues that may arise across the system.


I don’t believe there’s a clear answer to how much testing above and beyond requirements/acceptance criteria is enough. There will always be things in a system that weren’t in the requirements and as a team we should strive to reduce the things that fall into that category as much as possible given the resources and time available. It isn’t just the testers role to either just test requirements or be solely responsible/accountable for requirements that aren’t specified, the team should own this risk.

The value of automated testing

I recently saw an email thread discussing the value of test automation and asking how to justify it to management. It just seems such a strange concept to me. Justifying the value of something that is so essential to writing good quality software, is like justifying the value of exercise to a human body: it’s benefits are so obvious it’s almost a waste of time attempting to justify it.

But the main reason I find justifying the value of automated testing so strange is I don’t really see any viable alternative. It’s like justifying to your boss the benefit of flying from Australia to the USA and back for a conference (~30 hours return) instead of taking a ship (50-80 days).

If we have a critical bug in our production system we need to turn around a fix in less than an hour. To turnaround a fix in less than an hour we need full regression test coverage that can give us feedback that we’re all good and haven’t broken anything else in half that time. To get the same coverage that our automated regression test suite has through 30 minutes of manual test execution would require having about 184 QA staff always available to do regression testing in parallel: that’s not going to happen.

Unless you can wait days/weeks for manual regression testing, or have a sufficiently large team of QA resources always available to test, you really can’t release software quickly with confidence that it’s high quality. With automated tests you can; and that’s the value of automated testing.

100,000 e2e selenium tests? Sounds like a nightmare!

This story begins with a promo email I received from Sauce Labs…

“Ever wondered how an Enterprise company like Salesforce runs their QA tests? Learn about Salesforce’s inventory of 100,000 Selenium tests, how they run them at scale, and how to architect your test harness for success”

saucelabs email

100,000 end-to-end selenium tests and success in the same sentence? WTF? Sounds like a nightmare to me!

I dug further and got burnt by the molten lava: the slides confirmed my nightmare was indeed real:

Salesforce Selenium Slide

“We test end to end on almost every action.”

Ouch! (and yes, that is an uncredited image from my blog used in the completely wrong context)

But it gets worse. Salesforce have 7500 unique end-to-end WebDriver tests which are run on 10 browsers (IE6, IE7, IE8, IE9, IE10, IE11, Chrome, Firefox, Safari & PhantomJS) on 50,000 client VMs that cost multiple millions of dollars, totaling 1 million browser tests executed per day (which equals 20 selenium tests per day, per machine, or over 1 hour to execute each test).

Salesforce UI Testing Portfolio

My head explodes! (and yes, another uncredited image from this blog used out of context and with my title removed).

But surely that’s only one place right? Not everyone does this?

A few weeks later I watched David Heinemeier Hansson say this:

“We recently had a really bad bug in Basecamp where we actually lost some data for real customers and it was incredibly well tested at the unit level, and all the tests passed, and we still lost data. How the f*#% did this happen? It happened because we were so focused on driving our design from the unit test level we didn’t have any system tests for this particular thing.
…And after that, we sort of thought, wait a minute, all these unit tests are just focusing on these core objects in the system, these individual unit pieces, it doesn’t say anything about whether the whole system works.”

~ David Heinemeier Hansson – Ruby on Rails creator

and read that he had written this:

“…layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests (Which btw do not need to be so slow any more, thanks to advances in parallelization and cloud runner infrastructure).”

~ David Heinemeier Hansson – Ruby on Rails creator

I started to get very worried. David is the creator of Ruby on Rails and very well respected within the ruby community (despite being known to be very provocative and anti-intellectual: the ‘Fox News’ of the ruby world).

But here is dhh telling us to replace lower level tests with higher level ‘system’ (end to end) tests that use something like Capybara to drive a browser because unit tests didn’t find a bug and because it’s now possible to parallelize these ‘slow’ tests? Seriously?

Speed has always seen as the Achille’s heel of end to end tests because everyone knows that fast feedback is good. But parallelization solves this right? We just need 50,000 VMs like Salesforce?


Firstly, parallelization of end to end tests actually introduces its own problems, such as what to do with tests that you can’t run in parallel (for example, ones that change global state of a system such as a system message that appears to all users), and it definitely makes test data management trickier. You’ll be surprised the first time you run an existing suite of sequential e2e tests in parallel, as a lot will fail for unknown reasons.

Secondly, the test feedback to someone who’s made a change still isn’t fast enough to enable confidence in making a change (by the time your app has been deployed and the parallel end-to-end tests have run; the person who made the change has most likely moved onto something else).

But the real problem with end to end tests isn’t actually speed. The real problem with end to end tests is that when end to end tests fail, most of the time you have no idea what went wrong so you spend a lot of time trying to find out why. Was it the server? Was it the deployment? Was it the data? Was it the actual test? Maybe a browser update that broke Selenium? Was the test flaky (non-deterministic or non-hermetic)?

Rachel Laycock and Chirag Doshi from ThoughtWorks explain this really well in their recent post on broken UI tests:

“…unlike unit tests, the functional tests don’t tell you what is broken or where to locate the failure in the code base. They just tell you something is broken. That something could be the test, the browser, or a race condition. There is no way to tell because functional tests, by definition of being end-to-end, test everything.”

So what’s the answer? You have David’s FUD about unit testing not catching a major bug in BaseCamp. On the other hand you need to face the issue of having a large suite of end to end tests will most likely result in you spending all your time investigating test failures instead of delivering new features quickly.

If I had to choose just one, I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week.

Why? Because it’s much easier to supplement comprehensive unit testing with human exploratory end-to-end system testing (and you should anyway!) than trying to manually verify units function from the higher system level, and it’s much easier to know why a unit test is broken as explained above. And it’s also much easier to add automated end-to-end tests later than trying to retrofit unit tests later (because your code probably won’t be testable and making it testable after-the-fact can introduce bugs).

To answer our question, let’s imagine for a minute that you were responsible for designing and building a new plane. You obviously need to test that your new plane works. You build a plane by creating parts (units), putting these together into components, and then putting all the components together to build the (hopefully) working plane (system).

If you only focused on unit tests, like David mentioned in his Basecamp example, you could be pretty confident that each piece of the plane would be have been tested well and works correctly, but wouldn’t be confident it would fly!

If you only focussed on end to end tests, you’d need to fly the plane to check the individual units and components actually work (which is expensive and slow), and even then, if/when it crashed, you’d need to examine the black-box to hopefully understand which unit or component didn’t work, as we currently do when end-to-end tests fail.

But, obviously we don’t need to choose just one. And that’s exactly what Airbus does when it’s designing and building the new Airbus A350:

As with any new plane, the early design phases were riddled with uncertainty. Would the materials be light enough and strong enough? Would the components perform as Airbus desired? Would parts fit together? Would it fly the way simulations predicted? To produce a working aircraft, Airbus had to systematically eliminate those risks using a process it calls a “testing pyramid.” The fat end of the pyramid represents the beginning, when everything is unknown. By testing materials, then components, then systems, then the aircraft as a whole, ever-greater levels of complexity can be tamed. “The idea is to answer the big questions early and the little questions later,” says Stefan Schaffrath, Airbus’s vice president for media relations.

The answer, which has been the answer all along, is to have a balanced set of automated tests across all levels, with a disciplined approach to having a larger number of smaller specific automated unit/component tests and a smaller number of larger general end-to-end automated tests to ensure all the units and components work together. (My diagram below with attribution)

Automated Testing Pyramid

Having just one level of tests, as shown by the stories above, doesn’t work (but if it did I would rather automated unit tests). Just like having a diet of just chocolate doesn’t work, nor does a diet that deprives you of anything sweet or enjoyable (but if I had to choose I would rather a diet of healthy food only than a diet of just chocolate).

Now if we could just convince Salesforce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change and stop David from continuing on his anti-unit pro-system testing anti-intellectual rampage which will result in more damage to our industry than it’s worth.

My thoughts on tddGate

If you’ve somehow managed to miss the keynote, blog post and subsequent shitstorm about it, David Heinemeier Hansson (dhh), creator of ruby on rails, has recently come out and declared test-driven development (TDD) dead. I’ve dubbed it ‘tddGate‘.

I find it rather ironic that David advocates the importance of clarity of code in his keynote, yet his objections to TDD through his keynote and posts are anything but clear (to me at least).

For example:

  • I don’t fully comprehend his science/pseudoscience/diet analogy in his keynote: he claims TDD is science-based because it uses metrics and coverage, but it’s also like a diet in that most people can’t make it work so it’s pseudoscience, but he also believes information system development isn’t science because it’s actually more like writing French poetry? Very confusing.
  • He interchangeably uses TDD to mean Test Driven Development and Test Driven Design.
  • He seems to imply you can only do TDD if you’re writing unit tests and you can only write unit tests if they are isolated by using dependency injection (DI) and mocks. He also seems fairly negative on unit testing, DI and mocks, therefore negative on TDD, and wants it dead so he can write (slower) system tests without using TDD, mocks or DI.
  • David gives an example of why unit tests aren’t valuable because they didn’t catch a BaseCamp bug to do with attachments (hint: the issue isn’t to do with unit testing per se, but having only one style of tests).
  • Because David thinks TDD is about unit testing, he sees driving system design from units is bad because people don’t care about units, they care about the whole thing, and doesn’t see the importance of testability.
  • Most importantly, he seems to not fully understand TDD (or at least doesn’t communicate his understanding very well):

 “TDD was what I was supposed to do. With TDD I was supposed to write all my tests first and then I would be allowed to write my code. It just didn’t work.” 25:29

The one subject that I wholeheartedly agree with David on is the importance of reading other people’s code. Writers read much more than they write, so should programmers.

So, here’s some of my current thoughts on TDD:

  • I have met few programmers who write unit tests, let alone who practice TDD.
  • Self testing code (eg. automated testing) is critically important to the health of a codebase as it allows someone to confidently make changes and/or perform refactoring without worrying they may have inadvertently broken something.
  • One way to achieve self testing code is via TDD, but it’s by no means the only way. You can easily achieve a self testing codebase by writing tests after code (or even having someone else write tests).
  • There are circumstances where it doesn’t make sense to write tests first (see some examples here).
  • It’s common to practice TDD by writing unit tests but it’s not the only way to practice TDD (for example: you could write an integration test first or an acceptance test first).
  • It’s common to write ‘isolated’ unit tests using DI and test doubles (so they’re fast and decoupled)  but it’s not the only way to write unit tests (you can interact with your database and you can test real dependencies, they’re not isolated unit tests, but are still unit tests nonetheless).
  • I personally find practicing TDD and writing unit tests first does result in a clearer, more well designed API as you’re calling your own API and you can design it how you like, but it isn’t the only way to achieve a clear API.
  • I also find practicing TDD is very effective for bug fixes as it’s easy to write a failing test and have confidence you’ve fixed the problem (and not created any others) when the test finally passes.
  • I don’t trust a test I haven’t seen fail: and this is much easier to do with TDD. You can also achieve this after the fact by (temporarily) changing your code to not work.
  • Unlike David, I strongly believe in the value of testability.
  • I believe it’s important to have the right mix of different types of automated tests for your context. Most often this means more unit tests and less end to end tests, but there are some cases where this is skewed. A diet of just one, like eating only chocolate, or completely banning sweet foods, is unhealthy and unsustainable.
  • Do what works for you personally and in your context. If you love the flow you achieve doing TDD that’s great, if you can get self testing code another way, that’s equally good.
  • If you don’t enjoy it and it doesn’t work for you, don’t make yourself do something like TDD just because someone else says to do it. But don’t stop something like TDD if you like it just because someone else declares it ‘dead’.

A ruby testing framework, from scratch, in 15 minutes

As part of my talk last week at the Brisbane Testers Meetup, I gave a live demo (no pre-recorded or pre-written code) of writing a ruby testing framework from scratch in 15 minutes. The idea was to show that most testing frameworks contain so much functionality ‘you ain’t gonna need’, so why not try writing one from scratch and see how we go? It was also a chance to show the testers who hadn’t done automated testing that programming/automated testing is not rocket science.

Since I promised to talk about selenium, I used watir-webdriver, but I would have preferred to just show testing a simple app/class that I would have written from scratch in ruby.

Our testing problem

I wrote a beautifully simple website to welcome the testers to the first ever Brisbane Testers Meetup, and wanted to write some tests to make sure it worked. The site is accessible at data:text/html,<h1 id=”welcome”>Welcome BNE Testers!</h1> and looks something like this:


First I’ll give you a few moments to get over how amazing that web site is… that’s long enough, now, what we need is a couple of tests for it:

  1. Make sure the welcome message exists
  2. Make sure the welcome message is visible
  3. Make sure the welcome message content is correct

Iteration zero

Do the simplest thing that could possibly work. In our case print out the three things we want to check to the screen and we’ll manually verify them.

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
puts b.h1(id: 'welcome').exists?
puts b.h1(id: 'welcome').visible?
puts b.h1(id: 'welcome').text

which outputs:

Welcome BNE Testers!

A good start but not quite a testing framework.

Iteration One

I think it’s time to introduce a method to assert a value is true.

I like to start by writing how I want my tests to look before I write any ‘implementation’ code:

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }


I usually run the my tests to give me a ‘clue’ to what I need to do next. In our case:

 undefined method `assert' for main:Object (NoMethodError)

In our case, it’s simple, we need to write an assert method. Luckily we know exactly what we need: a method that takes a description string and a block of code that should execute returning true, otherwise we have an error. We can simply write this method above our existing tests:

def assert message, &block
		if (block.call)
			puts "Assertion PASSED for #{message}"
			puts "Assertion FAILED for #{message}"
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'"

which gives us this output when we run:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

This is awesome, but it makes me nervous that all of our three tests passed the first time we ran them. Perhaps we hard coded them to pass? Will they ever fail?

There’s an old saying, source unknown, which is ‘never trust a test you didn’t first see fail‘. Let’s apply this here by making all our tests fail. I usually do this by changing the source system, that way you can keep the integrity of your tests intact.

This is fairly easy to do in our case by changing the id of our welcome element.

data:text/html,<h1 id=”hello”>Welcome BNE Testers!</h1>

When we do so, all our tests fail: yipee.

Assertion FAILED for that the welcome message exists
Assertion FAILED for that the welcome message is visible with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'
Assertion FAILED for that the welcome message text is correct with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'

We change it back and they pass again: double yipee.

Iteration Two –

So far all the text output has been in same color, and everyone knows a good test framework uses color. Lucky I know a gem that does color output easily, all we do is:

require 'colorize'

def assert message, &block
		if (block.call)
			puts "Assertion PASSED for #{message}".green
			puts "Assertion FAILED for #{message}".red
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'".red

which gives us some pretty output:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

and a fail now looks like this:

Assertion FAILED for that the welcome message exists


We can put the assert method in its own file which leaves our test file cleaner and easier to read:

require 'watir-webdriver'
require './assertions.rb'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }


Iteration Three

Our final iteration involves making the tests even easier to read by abstracting away the browser. This is typically done using ‘page objects’ and again we’ll write how we would like it to look before implementing that functionality:

require 'watir-webdriver'
require './assertions.rb'


assert('that the welcome message exists') { Homepage.welcome.exists? }
assert('that the welcome message is visible') { Homepage.welcome.visible? }
assert('that the welcome message text is correct') { Homepage.welcome.text == 'Welcome BNE Testers!' }


When we run this, it provides us a hint at what we need to do:

uninitialized constant Homepage (NameError)

We need to create a HomePage class with three methods: visit, welcome and close.

We can simply add this to our tests file to get it working:

class Homepage
	def initialize
		@browser = Watir::Browser.new
		@browser.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

	def self.visit

	def welcome
		@browser.h1(id: 'welcome')

	def close

After we’re confident it is working okay, we simply move it to a file named homepage.rb and our resulting tests look a lot neater:

require './assertions.rb'
require './homepage.rb'

homepage = Homepage.visit

assert('that the welcome message exists') { homepage.welcome.exists? }
assert('that the welcome message is visible') { homepage.welcome.visible? }
assert('that the welcome message text is correct') { homepage.welcome.text == 'Welcome BNE Testers!' }


and when we run them, they’re green as cucumbers:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct


In a very short period of time, we’ve been able to write a fully functional (but not fully featured) testing framework and build on it as necessary. As I mentioned in my talk, so many test frameworks out there are so bloated and complex, sometimes all we need is simple, so if you’re putting test frameworks on pedestals because you find them too complex, start without one and see how you go!

Answer ‘Will it work?’ over ‘Does it work?’

Software teams must continually answer two key questions to ensure they deliver a quality product:

  1. Are we building the correct thing?
  2. Are we building the thing correctly?

In recent times, I’ve noticed a seismic shift of a tester’s role on an agile software team from testing that the team is building the thing correctly to helping the team build the correct thing. That thing can be a user story, a product or even an entire company.

As Trish Khoo recently wrote:

“The more effort I put into testing the product conceptually at the start of the process, the less I effort I had to put into manually testing the product at the end”

It’s more valuable for a tester to answer ‘will it work?‘ at the start than ‘does it work?‘ at the end. This is because if you can determine something isn’t the correct something before development is started, you’ll save the development, testing and rework needed to actually build what is needed (or not needed).

But how do we know it actually does work if we’re focused on will it work? How do we know that we’re building the thing correctly? The answer is automated tests.

Automated tests, written by programmers, alongside testers, during the engineering process validate the software does what it’s meant to do correctly. Behavior driven approaches assist to translate acceptance criteria directly into automated tests.

So, how can a tester be involved to make sure a team is building the correct thing?

  • get involved in writing the acceptance criteria for every story;
  • ensure a kick off for each story happens so the programmer(s) understand(s) what is expected and any edge cases or queries are discussed;
  • work with the programmer(s) to automate tests based upon the acceptance criteria;
  • ensure a handover/walk-through happens as soon as a story is finished in development to ensure that all the acceptance criteria are met and tests have been written;
  • showcase the finished product every iteration to the business.

You’ll soon find you can provide much greater value as a tester determining whether something will work and then working alongside the development team to ensure it works as it is developed.

You probably don’t need a specification framework

I think plain language specification frameworks like SpecFlow and Cucumber are great, but have a lot of overhead and are way overused.

If you don’t have non-technical folk collaborating with you on your specifications, try writing plain automated tests instead. This means using plain NUnit/MSTest over SpecFlow in C# or minitest over Cucumber in Ruby. You’ll avoid the overhead of maintaining a plain language specification framework and be able to focus on developing a great set of tests instead.

It’s easier than you think to add a plain language specification layer to a set of well structured plain tests. So only add the specification layer when you need it, because chances are you ain’t gonna.

Take control of your own career

During my career, I’ve come across numerous testing colleagues with no experience in automated testing who say things like “I’d love to do automated testing”. They expect to be put into an automated testing role so they can learn automated testing.

I don’t think it should work like that. Your employer shouldn’t be solely responsible for you enhancing your skills and progressing your career.

And, the thing is, it’s never been easier to pick up some new technical skills.

If you want to learn programming start by learning something like Ruby. If you want to learn about automated web testing learn Watir. If you want to learn about behavior driven development tools learn Cucumber.

I taught myself Ruby. I taught myself Watir. I taught myself C#, Python, Selenium, Cucumber and Jenkins. The list goes on.

The barrier to entry has never been lower. Try codeacademy, try ruby koans, download the free watir book, buy Cheezy’s cheap eBook about Watir & Cucumber.

So, instead of watching television or going out for drinks, spend your nights and weekends learning some new skills and taking control of your career instead of expecting your employer to hand it to you on a plate.

You’ll then be able to say “I’m learning all about Watir at the moment and I would love to apply that on a project” instead of “I’d love to do automated testing”.