Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser = Watir::Browser.new

Before do
  @browser = browser
end

at_exit do
  browser.close
end

To open and close the browser only once in RSpec:

browser = Watir::Browser.new

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }
end

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = 'https://en.wikipedia.org'
end

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"
end

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"
end

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'
end

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'
end

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

Introducing Einstien: the minesweeper robot

As part of the Test Automation Bazaar, I thought it would be fun to have a minesweeper robot contest.

Scott Sims was the ‘default’ winner as he had the only entry, an early interation of his own minesweeper robot that can win beginner level, but not much more.

A colleague of mine, Mark Ryall, and myself developed our own robot ‘Einstein’ and here are some slides and a video* about him. You can check him out on github.

Video

* Soundtrack courtesy of blakerobinson

Turnip: trying to solve Cucumber’s problems with ruby BDD

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i
    end
  end

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000″ as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).

Summary

I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.

Introducing the software testing ice-cream cone (anti-pattern)

As previously explained, I like using the software testing pyramid as a visual way to represent where you should be focusing your testing effort, and often switch between using a cloud or an Eye of Providence to represent the manual session-based tests at the top of the pyramid that you should use to supplement and test your automated tests.

I often see organizations fall into the trap of creating ‘inverted’ pyramids of software testing, and only yesterday did a colleague point out to me that if you invert my pyramid with the cloud, you end up with an ice-cream cone! So, introducing the software testing ice-cream cone (anti-pattern)!

Writing a CoffeeScript web application using TDD

As part of the Test Automation Bazaar minesweeper challenge, a colleague and I developed a CoffeeScript implementation of minesweeper. We wanted to use a test first approach to writing our CoffeeScript, so we decided to use Jasmine.

Setting up a development environment

Even though we were developing the application in CoffeeScript, which generates JavaScript, we found it easy to use a simple ruby environment to boostrap these tools. Our environment therefore looked something like this:

  • Ruby 1.9.2 (specified through a RVM .rvmrc file)
  • CoffeeScript installed via Homebrew
  • guard, guard-coffeescript, guard-sass gems for automatically generating JavaScript from CoffeeScript (and CSS from Sass)
  • jasmine gem: for writing/running tests
  • Sublime Text 2 editor for writing code

Specifying a Guardfile

A Guardfile consists of a bunch of guards that perform actions whenever a file is modified in that location. Our Guardfile looked like:

guard 'coffeescript', :output => 'javascripts' do
  watch /^coffeescripts\/.*[.]coffee/
end

guard 'coffeescript', :output => 'spec/javascripts' do
  watch /^spec\/coffeescripts\/.*[.]coffee/
end

guard 'sass', :input => 'stylesheets'

Writing a Jasmine specification in CoffeeScript

A Jasmine spec in CoffeeScript looked something like this:

describe 'GameState', ->
  game_state = null
  field = Field.new mineCount: 1, rows: 1, cols: 3

  beforeEach ->
    game_state = GameState.new field

  it 'should initialise lost to false', ->
    expect(game_state.lost()).toEqual false

  it 'should initialise won to false', ->
    expect(game_state.won()).toEqual false

  it 'should initialise remaining_mines to mine_count', ->
    expect(game_state.remaining_mines()).toEqual 1

  it 'should initialise remaining_mines to mine_count', ->
    expect(game_state.remaining_cells()).toEqual 2

Running Jasmine tests

The jasmine gem makes it very easy to run your jasmine tests. There’s a rake task called ‘jasmine’ which you can run to launch a jasmine server locally on port 8888. If you browse to that page, you’ll see something like this:

Test First CoffeeScript development

Now that you have Jasmine running, and Guard generating the CoffeeScript, it’s easy to write a new spec, refresh the Jasmine browser page to run all your tests (in our case in a third of a second) and then write the code to make it pass.

Automatically running Jasmine tests on Travis CI

If you’ve got your CoffeeScript and Jasmine on github, it’s trivial to automatically run all your Jasmine tests using the Jasmine::Ci rake task on Travis CI. All you need is a .travis.yml file like:


language: ruby
rvm:
  - 1.9.2
env:
  - DISPLAY=:99.0
before_install: sh -e /etc/init.d/xvfb start

Once you add the project to travis, it’ll automatically run whenever you push. Magic.

Lessons Learned

  • Use Sublime Text 2 for CoffeeScript joy: TextMate 2 forces you to use tabs (4) for CoffeeScript development, we couldn’t find a way to make it stop, and had to switch to Sublime Text 2 (for the better).
  • You still need to know JavaScript: all CoffeeScript does is generate JavaScript for both your application code and Jasmine specs. We found ourselves often delving into the generated code to see what was actually going on, and when using Firebug to debug.
  • Use the jasmine_content div to test the DOM. Jasmine uses a special div with an id of jasmine_content, which we used to inject and test HTML.
  • CoffeeScript classes are odd: we instead used closures to encapsulate state, using a new method on an object, which has the added benefit of looking like ruby.

I hope you enjoy your CoffeeScript test driven development with Jasmine.

Cukepatch: rich editing of feature files on Github

I’ve been a strong advocate of using the built in Github web text editor for editing feature files for some time, as it means that non-technical business users don’t need to worry about having git clients installed and pulling/pushing changes.

The benefit of this method over a publishing system such as Relish is that you can send a subject matter expert a URL to a Github feature file, and if they recognize that the specification is incorrect, they can immediately update it, unlike Relish where they need to go to the source code and push a change.

The downside is that the editor is a pretty basic, meaning no syntax highligting, step completion etc. Until now that is…

Aslak Hellesøy and Julien Biezemans recently announced Cukepatch: rich editing of feature files on Github. There’s some detail on the Cukes Google Group about it, but essentially it provides rich editing (syntax highlighting/validation and step completion) using a Google Chrome user script that reads a cucumber file you create in your public cucumber repository.

I did this for both WatirMelonCucumber and EtsyWatirWebDriver, and the results look like this:

This looks very promising indeed. There’s a few caveats at the moment including the requirement for a backend server, only working with public repositories, having to manually install the user script and being Google Chrome only. As these are overcome, I can see this becoming the de facto way for business users to write and edit specifications. Well done guys.

Visible content locators and i18n in automated tests

I recently read a rebuttal to my post about death to xpath selectors, which raises a point of not using user visible strings in/as selectors to identify elements. The reasoning is that if the time comes to internationalize your site, then your selectors will be brittle as they’re written in a specific language.

Fair point, but if you’re not testing the location of user visible content, then what are you testing? In Australia, I have found it rare (like one project out of about thirty I’ve worked on) that additional languages are supported. But on that one project I used visible user strings to locate objects that weren’t brittle whatsoever. But how? Adam says you can’t do it!

Well, I translate my locators too. That way I am testing both the functionality of the site, the content of the site, and the internationalized content of the site, all at once! No hands.

So how would I do it for my said poor example I used previously?

I’d wrap any selector with something like translate

  @browser.link(:text => translate('Buy')).click

and have a translate method defined in a mix-in:

def translate phrase
  #translate some phrase here using same method as AUT
  phrase
end

Easily capturing response time metrics using the Watir-WebDriver-Performance gem

I don’t like long dedicated performance testing windows at the end of a project. I see response times as non-functional requirements, and like any other requirements, these should be tested as we’re going along. One way of effectively measuring performance testing is by conducting a response time test every time you do a build, if there’s a big degradation in performance: break the build!

A couple of months ago, Tim Koopmans released the Watir-WebDriver-Peformance gem. It’s aimed at providing a set of navigation timing metrics for Watir-WebDriver actions. This is a perfect solution to capture response time metrics, and it’s very straightforward to do (only works in Chrome and IE9 at the moment – no Firefox support).

require 'watir-webdriver'
require 'watir-webdriver-performance'

b = Watir::Browser.new :chrome

10.times do
  b.goto 'http://watir.com'
  puts "Load Time: #{b.performance.summary[:response_time]/1000} seconds."
end  

This produces something like:

Load Time: 3.701 seconds.
Load Time: 0.694 seconds.
Load Time: 1.874 seconds.
Load Time: 1.721 seconds.
Load Time: 2.096 seconds.
Load Time: 0.823 seconds.
Load Time: 2.362 seconds.
Load Time: 1.008 seconds.
Load Time: 1.761 seconds.
Load Time: 2.066 seconds. 

List of available metric groupings

  • :summary
  • :navigation
  • :memory
  • :timing

Summary

The Watir-WebDriver-Performance gem is a great way to capture response time metrics from your Watir-WebDriver automated tests. By instrumenting your existing Watir-WebDriver automated tests that run regularly as part of a continuous integration process, you can start to capture and measure web application performance over time, and be alerted to the possibility of a change being introduced with adverse performance effects. Well done Tim!

Using Opera with Watir-Webdriver (for free)

Selenium 2.0 (WebDriver) was released today, and with it came support for Opera. Since Watir-WebDriver uses the WebDriver bindings, it now supports Opera out of the box.

How to get it working

  1. First make sure you have the latest watir-webdriver (>0.2.5) and selenium-webdriver (>2.0.0) gems installed.
  2. Install Opera if you haven’t already
  3. Download the latest Selenium Standalone Server JAR files to your computer
  4. Create an Environment Variable named SELENIUM_SERVER_JAR and set it to the path to your standalone server JAR file
    (On the mac, you can add something like this to your .bash_profile file:
    export SELENIUM_SERVER_JAR=/Library/Java/selenium-server-standalone-2.0.0.jar
  5. Open irb, require ‘watir-webdriver’, and type something like
    b = Watir::Browser.new :opera

    and ta-da: Opera automated testing.

Yet another software testing pyramid

A fellow ThoughtWorker James Crisp recently wrote an interesting article about his take on an automated test pyramid.

Some of the terminology he used was interesting, which is what I believe led to some questioning comments and a follow up article by another fellow ThoughtWorker, Dean Cornish, who stated the pyramid “oversimplifies a complex problem of how many tests you need to reach a point of feeling satisfied about your test coverage“.

I believe that one of the most unclear areas of James’s pyramid is the use of the term Acceptance tests, which James equates to roughly 10% of the automated test suite. One commenter stated these should instead be called functional tests, but as James points out, aren’t all tests functional in nature? I would also argue that all tests are about acceptance (to different people), so I would rephrase the term to express what is being tested, which in his case is the GUI.

The other fundamental issue I see with James testing pyramid is that it is missing exploratory/session based testing. The only mention of exploratory testing is when James states ‘if defects come to light from exploratory testing, then discover how they slipped through the testing net’, but I feel this could be better represented on the pyramid. Exploratory, or session based testing, ensures confidence in the automated tests that are being developed and run. Without it, an automated testing strategy is fundamentally flawed. That’s why I include it in my automated testing pyramid as the Eye of Providence (I originally got the ‘eye’ idea from another ThoughtWorker Darren Smith).

Show me the Pyramid

Without further ado, here’s my automated test pyramid. It shows what the automated tests use to test: being the GUI, APIs, Integration Points, Components & Units. I’ve put dotted lines between components, integration points and APIs, as these are similar and it might be a case of testing not all of these.

Another way of looking at this, is looking at the intention of the tests. Manual exploratory tests and automated GUI tests are business facing, in that they strive to answer the question: “are we building the right system?”. Unit, integration and component tests are technology facing, in that they strive to answer the question: “are we building the system right?”. So, another version of the automated testing pyramid could simply plot these two styles of tests on the pyramid, showing that you’ll need more technology facing than business facing automated tests, as the business facing tests are more difficult to maintain.

Summary

By removing the term acceptance, and showing what the automated tests test, I believe the first automated test pyramid shows a solid approach to automated testing. Acceptance tests and functional tests can be anywhere in the pyramid, but you should limit your GUI tests, often by increasing your unit test coverage.

The second pyramid is another way to view the intention of the tests, but I believe both resolve most of the issues Dean has with James’s pyramid. Additionally they both include manual session based testing, a key ingredient in an automated test strategy that should be shown on the pyramid so it is not forgotton.

I welcome your feedback.