You probably don’t need a specification framework

I think plain language specification frameworks like SpecFlow and Cucumber are great, but have a lot of overhead and are way overused.

If you don’t have non-technical folk collaborating with you on your specifications, try writing plain automated tests instead. This means using plain NUnit/MSTest over SpecFlow in C# or minitest over Cucumber in Ruby. You’ll avoid the overhead of maintaining a plain language specification framework and be able to focus on developing a great set of tests instead.

It’s easier than you think to add a plain language specification layer to a set of well structured plain tests. So only add the specification layer when you need it, because chances are you ain’t gonna.

Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser = Watir::Browser.new

Before do
  @browser = browser
end

at_exit do
  browser.close
end

To open and close the browser only once in RSpec:

browser = Watir::Browser.new

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }
end

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = 'https://en.wikipedia.org'
end

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"
end

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"
end

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'
end

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'
end

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

Turnip: trying to solve Cucumber’s problems with ruby BDD

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i
    end
  end

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000″ as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).

Summary

I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.

WatirMelon Spinach

Hello Spinach!

“Spinach is a new awesome BDD framework that features encapsulation and modularity of your step definitions.”

Seems like a good idea, so I thought I’d give it a try, and convert my existing simple Watirmelon Cucumber tests to use Spinach instead (link to my source code). It was very easy to do, here’s some observations:

  • It’s easy to get existing Cukes running using Spinach, but I imagine if you were starting out using Spinach you’d design things a lot differently
  • Steps belong in their own class, but you can include mixins to reuse steps – the ruby way
  • Steps are in a steps directory under features – shorter than step_definitions
  • Goodbye regular expressions in step definitions, which is a bit of a shame, as you can no longer capture values from the step name
  • As you can’t have regular expressions in step names, I find myself repeating steps that are similar but slightly different, this means my example steps have gone from 41 to 57 lines of code
  • Scenario Outlines aren’t supported at the moment, but I have raised this as a feature request
  • Hooks are dramatically improved, so I found them very easy to use and understand
  • There is no cucumber world, so you do a normal include instead, and env.rb is still supported
  • It displays really cool ticks on the command line when you’re running your spins

Well done to Codegram for releasing Spinach. If anything, it creates some great innovations that I imagine may find their way into Cucumber in the future.

Capturing screenshots in Watir-Webdriver & Cucumber

Update: 29 August – Thanks to Rob Hunter for the screenshot embed trick

When you’re running a suite of automated tests and one fails, I find it handy to capture a screenshot of the browser so it’s easier to investigate. Fortunately this is easier than ever before with Watir-Webdriver and it’s inbuilt save_screenshot method.

If you’re using Cucumber, you’ll be looking for something like this:

After do |scenario|
  if scenario.failed?
    Dir::mkdir('screenshots') if not File.directory?('screenshots')
    screenshot = "./screenshots/FAILED_#{scenario.name.gsub(' ','_').gsub(/[^0-9A-Za-z_]/, '')}.png"
    Browser::BROWSER.driver.save_screenshot(screenshot)
    embed screenshot, 'image/png'
  end
end

The neatest thing about this method is that it captures the entire browser page, not just what is displayed without scrolling (see a screenshot of my blog below).

I hope you find this handy, I think it’s awesome.

Removing local page references from Cucumber steps

I’ve been giving some thought about the maintainability of having local page object model references in Cucumber steps. To explain what I mean, here’s some code of mine from an Etsy page model step:

When /^an item is added to the cart$/ do
  @advanced_search_page = EtsyAdvancedSearchPage.new @browser, true
  @search_results_page = @advanced_search_page.search_for 'hat'
  @etsy_item_page = @search_results_page.click_first_result
  @item_title = @etsy_item_page.item_title
  @etsy_cart_page = @etsy_item_page.click_add_to_cart
end

Here I am using instance variables in Cucumber steps to store references to pages that I am interacting with. As you can see, it’s quite easy to lose track of which page is which.

If I want to use one of these pages in another step, I have to refer to the same variable name. The key to this concept is having pages return pages, so you don’t have to initialize pages in your steps (except for the first visit, which normally happens only once per scenario).

A colleague and good friend of mine has been showing me some alternatives. One alternative to this is to actually dynamically refer to pages only when you need them. This means you don’t need to return a page from a page, as they are always initialized when needed. The above method would look like:

When /^an item is added to the cart$/ do
  visit EtsyAdvancedSearchPage do |page| page.search_for 'hat' end
  on EtsySearchResultsPage do |page| page.click_first_result end
  on EtsyItemPage do |page|
    @item_title = page.item_title
    page.click_add_to_cart
  end
end

This introduces two new methods at the Cucumber step level that need to be created: on, and visit. The on method simply creates a new page object so you can call methods from it. As each page initialize checks expected title and expected element, it will raise an error automatically if either are incorrect. The visit method is an extension of on, which actually initalizes the page and visits it.

These are defined as a module in our Cucumber env.rb file, and then mixed into the Cucumber World, so that steps automatically have access to these.

module Browser
  BROWSER = Watir::Browser.new ENV['WEB_DRIVER'] ||:firefox

  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new BROWSER, visit
    block.call page if block
    page
  end
end

World Browser

Summary

By introducing the on and visit methods, it means that we no longer need to have instance variables for page classes in our Cucumber steps. We also no longer need to worry about state transformations in pages, as these can be done in a modular way in the steps themselves. It means that when an error occurs initializing a page, it is more likely to to occur in the correct place. I find the steps more readible, as you only have to initialize the page once using the on block, and then can refer to page.

I have updated my set of Etsy Watir-WebDriver steps if you’d like to take a look.

What do you think? How will this scale?

Update: I have updated the WatirMelonCucumber project to use this style. It’s slightly different in that it supports two sites, and therefore the on method dynamically switches between these.

Cucumber Steps:

When /^I search for a? ?"([^"]*)"$/ do |term|
  on :home do |page|
    page.search_for term
  end
end

The on method:

def on page, visit=false, &block
  page_class = Object.const_get "#{@site}#{page.capitalize}Page"
  page = page_class.new BROWSER, visit
  block.call page if block
  page
end

An automated testing journey

I did a presentation this evening in Brisbane on an automated testing journey that one may embark on. The whole thing was centered around this tube map style diagram I came up with recently: (download in PDF)

Here’s a link to my prezi slides and it should appear below (if you have flash enabled that is).  You can also download them in very printable PDF if you so choose.

I feel the presentation was well received, but I really shouldn’t have tried to squeeze three days of thinking into 30 mins. Oh well.

As always, I welcome your feedback.