Test in production?

This post is part of the Pride & Paradev series.

With continuous deployment, it is common to release new software into production multiple times a day. A regression test suite, no matter how well designed, may still take over 10 minutes to run, which can lead to bottlenecks in releasing changes to production.

So, do you even need to test before going live? Why not just test changes in production?

Test changes in production

The website for The Guardian, the UK’s third largest newspaper, deploys on average 11 times a day, of which all changes are tested in production.

“Once the code is in production, QA can really start.”

“Sometimes deployments go wrong. We expect that; and we accept it, because people (and machines) go wrong. But the key to dealing with these kind of mistakes is not to lock down the process or extend the breadth, depth and length of regression tests. The solution is to enable people to fix their mistakes quickly, learn, and get back to creating value as soon as possible.”

~ Andy Hume on Real-time QA [The Guardian Developer Blog]

The key to testing changes as soon as they hit production is to have real time, continuous real user experience monitoring. This includes metrics like page views and page load time, which directly correlate to advertising revenue, an incentive to keep these healthy.

More comprehensive automated acceptance tests can be written in a non-destructive style that means they can be run in production. This means that these can be run immediately following a fresh production deployment, and as feedback about the tests is received, any issues can be remedied immediately into production and tested again.

Test changes before production

There are not many businesses that are able to release software without any form of testing into production: whether there be legislative requirements requiring testing, or the risk of introducing errors is too high for its target market.

Whilst automated regression tests do take longer to run than unit or integration tests, there are ways to manage these to ensure the quickest path into production. These strategies include running tests in parallel, only running business critical tests, only running against the single most popular browser, or only running tests that are directly related to your changes.

You can set up a deployment pipeline that runs a selected subset of tests before deploying into production then running the remaining tests (in a test environment). Any of the issues found in subsequent tests are judged to see whether they warrant another immediate release or whether they can be included in the next set of changes being deployed into production.

Whilst you definitely should run tests before deploying to production, it doesn’t mean that this has to drastically hinder your ability to continuously deploy.

SSD: super super drives

If your development machines don’t have solid state drives (SSDs): go buy some.

If your continous integration (build) box doesn’t have a solid state drive: go buy one.

Solid state drives are the quickest and cheapest way to speed software development. As most continuous integrations/builds are IO, solid state drives provide tremendous speed increases: I’ve seen builds and tests run in less than a fifth of the time a traditional moving hard disk would take. Solid state drives have reached a price point where the lower capacity ones are on par with higher capacity hard disks (which you don’t need unless you’re storing movies etc.)

It used to be that the build machine would use some crappy hardware lying around: often an old development machine someone no longer used. This is one of the worst decisions you can make on a project.

Having a super fast SSD based build machine will ensure fast, reliable build results, fast feedback and a happy development team.

Automated WCAG 2.0 accessibility testing in the build pipeline

The web application I am currently working on must meet accessibility standards: WCAG 2.0 AA to be precise. WCAG 2.0 is a collection of guidelines that ensure your site is accessible for people with disabilities.

An example of poor accessibility design is missing an alt tag on an image, or not specifying a language of a document, eg:

<HTML lang="fr">

Building Accessibility In

Whilst later we’ll do doing accessibility assessments with people who are blind or have low vision, we need to make sure we build accessibility in. To do this, we need automated accessibility tests as part of our continuous integration build pipeline. My challenge this week was to do just this.

Automated Accessibility Tests

First I needed to find a tool to validate against the spec. We’re developing the web application locally so we’ll need to run it locally. There’s a tool called TotalValidator which offers a command line tool, the only downside is the licensing cost, as to use the command line version you need to buy at least 6 copies of the tool at £25 per copy, approximately US$240 in total. There’s no trial of command line tool unless you buy at least one copy of the pro tool at £25. I didn’t want to spend money on something that might not work so I kept looking for alternatives.

There are two sites I found that validate accessibility by URL or supplied HTML code: AChecker and WAVE.

AChecker: this tool which works really well. It even supplies an REST API, but I couldn’t find a way to call the API to validate HTML code (instead of by URL) which is what I would like to do. The software behind AChecker is open source (PHP) so you can actually install your own version behind your firewall if you wish.

WAVE: a new tool recently released by WebAim: Web Accessibility in Mind. Again this is an online checker that allows you to validate by URL or HTML code supplied, but unfortunately there’s no API (yet) and the results aren’t as easy to programatically read.

My Solution

The final solution I came up with is a specific tagged web accessibility feature in our acceptance tests project. This has scenarios that use WebDriver to navigate through our application capturing the HTML source code from each page visited. Finally, it visits the AChecker tool online and validates each piece of HTML source code it collected and fails the build if any accessibility problems are found.

AChecker Results

Build Light

We have a specific build light that runs all the automated acceptance tests and accessibility tests. If any of these fail, the light goes red.

Build Status Lights

It’s much better if it looks like this:

Build Lights All Green


It was fairly easy to use an existing free online accessibility checker to validate all HTML code in your local application, and make this status highly visible to the development team. By building accessibility in, we’re reducing the expected number of issues when we conduct more formal accessibility testing.

Bonus Points: faster accessibility feedback

Ideally, as a page is being developed, the developer/tester should be able to check accessibility (rather than waiting for the build to fail). The easiest way I have found behind a firewall is to use the WAVE Firefox extension, which displays errors as you use your site. Fantastic!

(Apple and Microsoft each have one known accessibility problem, Google has nine!)

Apple.com accessibility

Running Watir-WebDriver tests on Travis CI: a distributed build system

I recently came across Travis CI, a distributed build system that has close links to Github. I’ve seen quite a few projects use it as a CI system, but none that run headless browser tests.

Leveraging off the work I had done recently setting up my own Jenkins server in the cloud to run headless Watir-WebDriver tests, I thought I would have a go at running my WatirMelonCucumber and EtsyWatirWebDriver headless browser tests using Travis CI.

What I didn’t realize is how easy it’d be. The only things I had to do was make sure my Gemfile included rake, and also make sure there was some file existance checking happening for some log files, and it pretty much ran straight away. Wow!

Caveat Emptor

This is pretty new territory, so there’s a few things to watch out for:

  • Every now and, Travis complains about not having Firefox installed. I am not sure why this happens, maybe something to do with the different agents in use;
  • The locale of the build agents seems to be German, so when running my Google tests, the page content is in German, so it fails because it can’t find my expected (English) results; and
  • I can’t seem to capture my test results html file nor screenshot pngs, so it’s a red/green only affair at the moment.

But still, neat distributed free headless CI!

Running your watir-webdriver tests in the cloud, for free!

  • What if you could run unlimited Watir WebDriver tests in the cloud? Check.
  • What if the Watir WebDriver tests would run automatically as soon as you pushed a change to github? Check.
  • What if you would have a full visual history of results with embedded screenshots on failure? Check.
  • What if all of this was free?* Checkmate.

I’ve spent a bit of time over the last week working out how to do this. Here are the basics of what you need to do:

And here’s the detailed instructions.

Set up an Amazon EC2 micro instance running Ubuntu.

  1. First you need to sign up for an Amazon AWS account. This means you’re eligible for a free-tier micro instance for a year.
  2. Once you have an account set up, you need to launch a new instance. I found a free tier eligible Ubuntu image (11.04 Natty 64 bit desktop) and launched that.
  3. You will also want to create an elastic IP and associate it to your instance so that if you reboot your machine, you will have the same IP address. This is done through the AWS console under Elastic IPs.
  4. While you’re here, you’ll want to edit your machine’s security group and open up port 22 for SSH, and 80 for HTTP.
  5. This gives you secure shell (SSH) access to this machine using the provided key, and user ‘ubuntu’:
    ssh -i your-key-name.pem ubuntu@your-ip-address
  6. Everything you will do to configure this machine will be through this SSH session, so polish up your unix command line skills!

Set up Jenkins on your machine

There is a useful page for installing Jenkins on Ubuntu.

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo aptitude update
sudo aptitude install jenkins

Make Jenkins available on port 80 so that you don’t need to specify port

Jenkins installs by default on port 80. Ubuntu won’t let applications run on port 80 unless they’re running as root, so it’s best to set up an Apache 2 proxy to port 80 to 8080.

sudo aptitude install apache2
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod vhost_alias
sudo a2dissite default

Then create a file called jenkins in /etc/apache2/sites-available

	ServerAdmin webmaster@localhost
	ServerName ci.company.com
	ServerAlias ci
	ProxyRequests Off
		Order deny,allow
		Allow from all
	ProxyPreserveHost on
	ProxyPass / http://localhost:8080/

Then run the following commands:

sudo a2ensite jenkins
sudo apache2ctl restart

Password Protect Jenkins

You should go to your Jenkins site (accessible directly at your instance’s IP address through a web browser), and create an account, and then configure the security of Jenkins.

Install Jenkins Plugins

You will need to install the following Jenkins plugins

  • Github: to integrate to Github SCM
  • Rake: to run ruby rake tasks that run Watir-WebDriver tests
  • Green balls: because blue balls are just plain wrong

Install RVM for the Jenkins user

First we’ll need to install git

sudo apt-get install git

Jenkins will need to be able to run ruby, so we’ll install RVM as the Jenkins user.

To run as the jenkins user, we’ll use the sudo command, with the -Hiu arguments to load the home directory and bash profile:

sudo -Hiu jenkins

Once we are user Jenkins, we’ll install RVM using Git.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Now we need to work out what Ubuntu packages Ruby needs, which is easily done via RVM.

rvm notes

which gives me something like

For Ruby (MRI, Rubinius, & REE)  you should install the following OS dependencies:
/usr/bin/apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake

So, we can log-out as the Jenkins user (control-D) and install the following as ubuntu

sudo apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake

Once we’ve done this, we’ll want to run the following as the Jenkins user (
sudo -Hiu jenkins) to install Ruby 1.9.2.

rvm pkg install zlib
rvm install 1.9.2 --with-zlib-dir=$rvm_path/usr

Running headless Watir-WebDriver tests

I choose a desktop version of Ubuntu, so it’ll already have Firefox installed, but if you don’t, you can install it by:

sudo apt-get install firefox

To run our Watir WebDriver tests headlessly using the headless gem, we’ll need xvfb

sudo apt-get install xvfb

Configuring Jenkins to run tests via Rake

You add a new build in Jenkins where you can specify the github repository location.

As we’ve installed the rake plugin, we can configure a new Jenkins project to use an RVM ruby install (in my case ruby-1.9.2-p290@watirmelon-cucumber).

I simply set up a default task in rake, which runs all my cucumber tests. This generates a results.html file which is captured as an artifact, and also creates and captures junit xml results, which are used to show test summary information.I also capture any file created under the ‘screenshots’ directory.

Summary and Outcome

I have set up both my WatirMelonCucumber and EtsyWatirWebDriver projects on jenkins.watirmelon.com.

My Jenkins Dashboard looks something like this:

Please feel to leave a comment below and let me know what you think.

* Free for one year using an free tier EC2 micro instance

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: watirwebdriver.com for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at watir.com. If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server = Selenium::Server.new("/path/to/jar", :background => true)
# run your tests

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :chrome
b.goto 'www.google.com'
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"

The only difference for Firefox:

b = Watir::Browser.new :firefox

The only difference for IE:

b = Watir::Browser.new :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b = Watir::Browser.new(:remote, :url => '', :desired_capabilities => capabilities)
b = Watir::Browser.new :firefox
b.goto "www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using browser.windows, but I haven’t tried this out yet.


Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading

Running Watir reliably on a CI machine

I’ve found that Watir generally works well when running under Windows on a Continuous Integration machine (for example, Hudson or TeamCity) but if you use any of the autoit stuff to handle things like security authentication dialogs, it can fail when the machine is locked or not actively logged in.

I’ve found two ways to ensure Watir reliably runs tests without issues.

Run Caffeine to stop the machine locking or screensaver activating

Your Windows sys-admin might not like this one, but caffeine is a tiny executable that’s perfect for ensuring your Windows machine never locks or activates its screensaver, by simulating a keypress every 59 seconds. It’s a matter of installing it and a coffee cup appears in the tray to show you you’re machine is now on caffeine!

Use a command to release any remote desktop sessions to console

Caffeine works well, but if you use remote desktop to control the machine, when you disconnect, the machine will be locked and this will cause problems. What you need tp do is create a batch file, on your desktop say, that releases the RDP connection to console, so it’s ready to run any watir tests. The command you’ll need in the batch file is:

cmd.exe /Q /C "%windir%\System32\tscon.exe 0 /dest:console"

By using the combination of the above items, it’s easy to set up a reliable suite of Watir tests to run on a CI machine that you can easily remote into to see what is running when needed.