Turnip: trying to solve Cucumber’s problems with ruby BDD

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i
    end
  end

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000″ as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).

Summary

I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.

Introducing the software testing ice-cream cone

As previously explained, I like using the software testing pyramid as a visual way to represent where you should be focusing your testing effort, and often switch between using a cloud or an Eye of Providence to represent the manual session-based tests at the top of the pyramid that you should use to supplement and test your automated tests.

I often see organizations fall into the trap of creating ‘inverted’ pyramids of software testing, and only yesterday did a colleague point out to me that if you invert my pyramid with the cloud, you end up with an ice-cream cone! So, introducing the software testing ice-cream cone!

Writing a CoffeeScript web application using TDD

As part of the Test Automation Bazaar minesweeper challenge, a colleague and I developed a CoffeeScript implementation of minesweeper. We wanted to use a test first approach to writing our CoffeeScript, so we decided to use Jasmine.

Setting up a development environment

Even though we were developing the application in CoffeeScript, which generates JavaScript, we found it easy to use a simple ruby environment to boostrap these tools. Our environment therefore looked something like this:

  • Ruby 1.9.2 (specified through a RVM .rvmrc file)
  • CoffeeScript installed via Homebrew
  • guard, guard-coffeescript, guard-sass gems for automatically generating JavaScript from CoffeeScript (and CSS from Sass)
  • jasmine gem: for writing/running tests
  • Sublime Text 2 editor for writing code

Specifying a Guardfile

A Guardfile consists of a bunch of guards that perform actions whenever a file is modified in that location. Our Guardfile looked like:

guard 'coffeescript', :output => 'javascripts' do
  watch /^coffeescripts\/.*[.]coffee/
end

guard 'coffeescript', :output => 'spec/javascripts' do
  watch /^spec\/coffeescripts\/.*[.]coffee/
end

guard 'sass', :input => 'stylesheets'

Writing a Jasmine specification in CoffeeScript

A Jasmine spec in CoffeeScript looked something like this:

describe 'GameState', ->
  game_state = null
  field = Field.new mineCount: 1, rows: 1, cols: 3

  beforeEach ->
    game_state = GameState.new field

  it 'should initialise lost to false', ->
    expect(game_state.lost()).toEqual false

  it 'should initialise won to false', ->
    expect(game_state.won()).toEqual false

  it 'should initialise remaining_mines to mine_count', ->
    expect(game_state.remaining_mines()).toEqual 1

  it 'should initialise remaining_mines to mine_count', ->
    expect(game_state.remaining_cells()).toEqual 2

Running Jasmine tests

The jasmine gem makes it very easy to run your jasmine tests. There’s a rake task called ‘jasmine’ which you can run to launch a jasmine server locally on port 8888. If you browse to that page, you’ll see something like this:

Test First CoffeeScript development

Now that you have Jasmine running, and Guard generating the CoffeeScript, it’s easy to write a new spec, refresh the Jasmine browser page to run all your tests (in our case in a third of a second) and then write the code to make it pass.

Automatically running Jasmine tests on Travis CI

If you’ve got your CoffeeScript and Jasmine on github, it’s trivial to automatically run all your Jasmine tests using the Jasmine::Ci rake task on Travis CI. All you need is a .travis.yml file like:


language: ruby
rvm:
  - 1.9.2
env:
  - DISPLAY=:99.0
before_install: sh -e /etc/init.d/xvfb start

Once you add the project to travis, it’ll automatically run whenever you push. Magic.

Lessons Learned

  • Use Sublime Text 2 for CoffeeScript joy: TextMate 2 forces you to use tabs (4) for CoffeeScript development, we couldn’t find a way to make it stop, and had to switch to Sublime Text 2 (for the better).
  • You still need to know JavaScript: all CoffeeScript does is generate JavaScript for both your application code and Jasmine specs. We found ourselves often delving into the generated code to see what was actually going on, and when using Firebug to debug.
  • Use the jasmine_content div to test the DOM. Jasmine uses a special div with an id of jasmine_content, which we used to inject and test HTML.
  • CoffeeScript classes are odd: we instead used closures to encapsulate state, using a new method on an object, which has the added benefit of looking like ruby.

I hope you enjoy your CoffeeScript test driven development with Jasmine.

Cukepatch: rich editing of feature files on Github

I’ve been a strong advocate of using the built in Github web text editor for editing feature files for some time, as it means that non-technical business users don’t need to worry about having git clients installed and pulling/pushing changes.

The benefit of this method over a publishing system such as Relish is that you can send a subject matter expert a URL to a Github feature file, and if they recognize that the specification is incorrect, they can immediately update it, unlike Relish where they need to go to the source code and push a change.

The downside is that the editor is a pretty basic, meaning no syntax highligting, step completion etc. Until now that is…

Aslak Hellesøy and Julien Biezemans recently announced Cukepatch: rich editing of feature files on Github. There’s some detail on the Cukes Google Group about it, but essentially it provides rich editing (syntax highlighting/validation and step completion) using a Google Chrome user script that reads a cucumber file you create in your public cucumber repository.

I did this for both WatirMelonCucumber and EtsyWatirWebDriver, and the results look like this:

This looks very promising indeed. There’s a few caveats at the moment including the requirement for a backend server, only working with public repositories, having to manually install the user script and being Google Chrome only. As these are overcome, I can see this becoming the de facto way for business users to write and edit specifications. Well done guys.

Visible content locators and i18n in automated tests

I recently read a rebuttal to my post about death to xpath selectors, which raises a point of not using user visible strings in/as selectors to identify elements. The reasoning is that if the time comes to internationalize your site, then your selectors will be brittle as they’re written in a specific language.

Fair point, but if you’re not testing the location of user visible content, then what are you testing? In Australia, I have found it rare (like one project out of about thirty I’ve worked on) that additional languages are supported. But on that one project I used visible user strings to locate objects that weren’t brittle whatsoever. But how? Adam says you can’t do it!

Well, I translate my locators too. That way I am testing both the functionality of the site, the content of the site, and the internationalized content of the site, all at once! No hands.

So how would I do it for my said poor example I used previously?

I’d wrap any selector with something like translate

  @browser.link(:text => translate('Buy')).click

and have a translate method defined in a mix-in:

def translate phrase
  #translate some phrase here using same method as AUT
  phrase
end

Easily capturing response time metrics using the Watir-WebDriver-Performance gem

I don’t like long dedicated performance testing windows at the end of a project. I see response times as non-functional requirements, and like any other requirements, these should be tested as we’re going along. One way of effectively measuring performance testing is by conducting a response time test every time you do a build, if there’s a big degradation in performance: break the build!

A couple of months ago, Tim Koopmans released the Watir-WebDriver-Peformance gem. It’s aimed at providing a set of navigation timing metrics for Watir-WebDriver actions. This is a perfect solution to capture response time metrics, and it’s very straightforward to do (only works in Chrome and IE9 at the moment – no Firefox support).

require 'watir-webdriver'
require 'watir-webdriver-performance'

b = Watir::Browser.new :chrome

10.times do
  b.goto 'http://watir.com'
  puts "Load Time: #{b.performance.summary[:response_time]/1000} seconds."
end 

This produces something like:

Load Time: 3.701 seconds.
Load Time: 0.694 seconds.
Load Time: 1.874 seconds.
Load Time: 1.721 seconds.
Load Time: 2.096 seconds.
Load Time: 0.823 seconds.
Load Time: 2.362 seconds.
Load Time: 1.008 seconds.
Load Time: 1.761 seconds.
Load Time: 2.066 seconds. 

List of available metric groupings

  • :summary
  • :navigation
  • :memory
  • :timing

Summary

The Watir-WebDriver-Performance gem is a great way to capture response time metrics from your Watir-WebDriver automated tests. By instrumenting your existing Watir-WebDriver automated tests that run regularly as part of a continuous integration process, you can start to capture and measure web application performance over time, and be alerted to the possibility of a change being introduced with adverse performance effects. Well done Tim!

Using Opera with Watir-Webdriver (for free)

Selenium 2.0 (WebDriver) was released today, and with it came support for Opera. Since Watir-WebDriver uses the WebDriver bindings, it now supports Opera out of the box.

How to get it working

  1. First make sure you have the latest watir-webdriver (>0.2.5) and selenium-webdriver (>2.0.0) gems installed.
  2. Install Opera if you haven’t already
  3. Download the latest Selenium Standalone Server JAR files to your computer
  4. Create an Environment Variable named SELENIUM_SERVER_JAR and set it to the path to your standalone server JAR file
    (On the mac, you can add something like this to your .bash_profile file:
    export SELENIUM_SERVER_JAR=/Library/Java/selenium-server-standalone-2.0.0.jar
  5. Open irb, require ‘watir-webdriver’, and type something like
    b = Watir::Browser.new :opera

    and ta-da: Opera automated testing.

Yet another software testing pyramid

A fellow ThoughtWorker James Crisp recently wrote an interesting article about his take on an automated test pyramid.

Some of the terminology he used was interesting, which is what I believe led to some questioning comments and a follow up article by another fellow ThoughtWorker, Dean Cornish, who stated the pyramid “oversimplifies a complex problem of how many tests you need to reach a point of feeling satisfied about your test coverage“.

I believe that one of the most unclear areas of James’s pyramid is the use of the term Acceptance tests, which James equates to roughly 10% of the automated test suite. One commenter stated these should instead be called functional tests, but as James points out, aren’t all tests functional in nature? I would also argue that all tests are about acceptance (to different people), so I would rephrase the term to express what is being tested, which in his case is the GUI.

The other fundamental issue I see with James testing pyramid is that it is missing exploratory/session based testing. The only mention of exploratory testing is when James states ‘if defects come to light from exploratory testing, then discover how they slipped through the testing net’, but I feel this could be better represented on the pyramid. Exploratory, or session based testing, ensures confidence in the automated tests that are being developed and run. Without it, an automated testing strategy is fundamentally flawed. That’s why I include it in my automated testing pyramid as the Eye of Providence (I originally got the ‘eye’ idea from another ThoughtWorker Darren Smith).

Show me the Pyramid

Without further ado, here’s my automated test pyramid. It shows what the automated tests use to test: being the GUI, APIs, Integration Points, Components & Units. I’ve put dotted lines between components, integration points and APIs, as these are similar and it might be a case of testing not all of these.

Another way of looking at this, is looking at the intention of the tests. Manual exploratory tests and automated GUI tests are business facing, in that they strive to answer the question: “are we building the right system?”. Unit, integration and component tests are technology facing, in that they strive to answer the question: “are we building the system right?”. So, another version of the automated testing pyramid could simply plot these two styles of tests on the pyramid, showing that you’ll need more technology facing than business facing automated tests, as the business facing tests are more difficult to maintain.

Summary

By removing the term acceptance, and showing what the automated tests test, I believe the first automated test pyramid shows a solid approach to automated testing. Acceptance tests and functional tests can be anywhere in the pyramid, but you should limit your GUI tests, often by increasing your unit test coverage.

The second pyramid is another way to view the intention of the tests, but I believe both resolve most of the issues Dean has with James’s pyramid. Additionally they both include manual session based testing, a key ingredient in an automated test strategy that should be shown on the pyramid so it is not forgotton.

I welcome your feedback.

Removing local page references from Cucumber steps

I’ve been giving some thought about the maintainability of having local page object model references in Cucumber steps. To explain what I mean, here’s some code of mine from an Etsy page model step:

When /^an item is added to the cart$/ do
  @advanced_search_page = EtsyAdvancedSearchPage.new @browser, true
  @search_results_page = @advanced_search_page.search_for 'hat'
  @etsy_item_page = @search_results_page.click_first_result
  @item_title = @etsy_item_page.item_title
  @etsy_cart_page = @etsy_item_page.click_add_to_cart
end

Here I am using instance variables in Cucumber steps to store references to pages that I am interacting with. As you can see, it’s quite easy to lose track of which page is which.

If I want to use one of these pages in another step, I have to refer to the same variable name. The key to this concept is having pages return pages, so you don’t have to initialize pages in your steps (except for the first visit, which normally happens only once per scenario).

A colleague and good friend of mine has been showing me some alternatives. One alternative to this is to actually dynamically refer to pages only when you need them. This means you don’t need to return a page from a page, as they are always initialized when needed. The above method would look like:

When /^an item is added to the cart$/ do
  visit EtsyAdvancedSearchPage do |page| page.search_for 'hat' end
  on EtsySearchResultsPage do |page| page.click_first_result end
  on EtsyItemPage do |page|
    @item_title = page.item_title
    page.click_add_to_cart
  end
end

This introduces two new methods at the Cucumber step level that need to be created: on, and visit. The on method simply creates a new page object so you can call methods from it. As each page initialize checks expected title and expected element, it will raise an error automatically if either are incorrect. The visit method is an extension of on, which actually initalizes the page and visits it.

These are defined as a module in our Cucumber env.rb file, and then mixed into the Cucumber World, so that steps automatically have access to these.

module Browser
  BROWSER = Watir::Browser.new ENV['WEB_DRIVER'] ||:firefox

  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new BROWSER, visit
    block.call page if block
    page
  end
end

World Browser

Summary

By introducing the on and visit methods, it means that we no longer need to have instance variables for page classes in our Cucumber steps. We also no longer need to worry about state transformations in pages, as these can be done in a modular way in the steps themselves. It means that when an error occurs initializing a page, it is more likely to to occur in the correct place. I find the steps more readible, as you only have to initialize the page once using the on block, and then can refer to page.

I have updated my set of Etsy Watir-WebDriver steps if you’d like to take a look.

What do you think? How will this scale?

Update: I have updated the WatirMelonCucumber project to use this style. It’s slightly different in that it supports two sites, and therefore the on method dynamically switches between these.

Cucumber Steps:

When /^I search for a? ?"([^"]*)"$/ do |term|
  on :home do |page|
    page.search_for term
  end
end

The on method:

def on page, visit=false, &block
  page_class = Object.const_get "#{@site}#{page.capitalize}Page"
  page = page_class.new BROWSER, visit
  block.call page if block
  page
end

An automated testing journey

I did a presentation this evening in Brisbane on an automated testing journey that one may embark on. The whole thing was centered around this tube map style diagram I came up with recently: (download in PDF)

Here’s a link to my prezi slides and it should appear below (if you have flash enabled that is).  You can also download them in very printable PDF if you so choose.

I feel the presentation was well received, but I really shouldn’t have tried to squeeze three days of thinking into 30 mins. Oh well.

As always, I welcome your feedback.