Visible content locators and i18n in automated tests

I recently read a rebuttal to my post about death to xpath selectors, which raises a point of not using user visible strings in/as selectors to identify elements. The reasoning is that if the time comes to internationalize your site, then your selectors will be brittle as they’re written in a specific language.

Fair point, but if you’re not testing the location of user visible content, then what are you testing? In Australia, I have found it rare (like one project out of about thirty I’ve worked on) that additional languages are supported. But on that one project I used visible user strings to locate objects that weren’t brittle whatsoever. But how? Adam says you can’t do it!

Well, I translate my locators too. That way I am testing both the functionality of the site, the content of the site, and the internationalized content of the site, all at once! No hands.

So how would I do it for my said poor example I used previously?

I’d wrap any selector with something like translate

  @browser.link(:text => translate('Buy')).click

and have a translate method defined in a mix-in:

def translate phrase
  #translate some phrase here using same method as AUT
  phrase
end

Yet another software testing pyramid

A fellow ThoughtWorker James Crisp recently wrote an interesting article about his take on an automated test pyramid.

Some of the terminology he used was interesting, which is what I believe led to some questioning comments and a follow up article by another fellow ThoughtWorker, Dean Cornish, who stated the pyramid “oversimplifies a complex problem of how many tests you need to reach a point of feeling satisfied about your test coverage“.

I believe that one of the most unclear areas of James’s pyramid is the use of the term Acceptance tests, which James equates to roughly 10% of the automated test suite. One commenter stated these should instead be called functional tests, but as James points out, aren’t all tests functional in nature? I would also argue that all tests are about acceptance (to different people), so I would rephrase the term to express what is being tested, which in his case is the GUI.

The other fundamental issue I see with James testing pyramid is that it is missing exploratory/session based testing. The only mention of exploratory testing is when James states ‘if defects come to light from exploratory testing, then discover how they slipped through the testing net’, but I feel this could be better represented on the pyramid. Exploratory, or session based testing, ensures confidence in the automated tests that are being developed and run. Without it, an automated testing strategy is fundamentally flawed. That’s why I include it in my automated testing pyramid as the Eye of Providence (I originally got the ‘eye’ idea from another ThoughtWorker Darren Smith).

Show me the Pyramid

Without further ado, here’s my automated test pyramid. It shows what the automated tests use to test: being the GUI, APIs, Integration Points, Components & Units. I’ve put dotted lines between components, integration points and APIs, as these are similar and it might be a case of testing not all of these.

Another way of looking at this, is looking at the intention of the tests. Manual exploratory tests and automated GUI tests are business facing, in that they strive to answer the question: “are we building the right system?”. Unit, integration and component tests are technology facing, in that they strive to answer the question: “are we building the system right?”. So, another version of the automated testing pyramid could simply plot these two styles of tests on the pyramid, showing that you’ll need more technology facing than business facing automated tests, as the business facing tests are more difficult to maintain.

Summary

By removing the term acceptance, and showing what the automated tests test, I believe the first automated test pyramid shows a solid approach to automated testing. Acceptance tests and functional tests can be anywhere in the pyramid, but you should limit your GUI tests, often by increasing your unit test coverage.

The second pyramid is another way to view the intention of the tests, but I believe both resolve most of the issues Dean has with James’s pyramid. Additionally they both include manual session based testing, a key ingredient in an automated test strategy that should be shown on the pyramid so it is not forgotton.

I welcome your feedback.

Removing local page references from Cucumber steps

I’ve been giving some thought about the maintainability of having local page object model references in Cucumber steps. To explain what I mean, here’s some code of mine from an Etsy page model step:

When /^an item is added to the cart$/ do
  @advanced_search_page = EtsyAdvancedSearchPage.new @browser, true
  @search_results_page = @advanced_search_page.search_for 'hat'
  @etsy_item_page = @search_results_page.click_first_result
  @item_title = @etsy_item_page.item_title
  @etsy_cart_page = @etsy_item_page.click_add_to_cart
end

Here I am using instance variables in Cucumber steps to store references to pages that I am interacting with. As you can see, it’s quite easy to lose track of which page is which.

If I want to use one of these pages in another step, I have to refer to the same variable name. The key to this concept is having pages return pages, so you don’t have to initialize pages in your steps (except for the first visit, which normally happens only once per scenario).

A colleague and good friend of mine has been showing me some alternatives. One alternative to this is to actually dynamically refer to pages only when you need them. This means you don’t need to return a page from a page, as they are always initialized when needed. The above method would look like:

When /^an item is added to the cart$/ do
  visit EtsyAdvancedSearchPage do |page| page.search_for 'hat' end
  on EtsySearchResultsPage do |page| page.click_first_result end
  on EtsyItemPage do |page|
    @item_title = page.item_title
    page.click_add_to_cart
  end
end

This introduces two new methods at the Cucumber step level that need to be created: on, and visit. The on method simply creates a new page object so you can call methods from it. As each page initialize checks expected title and expected element, it will raise an error automatically if either are incorrect. The visit method is an extension of on, which actually initalizes the page and visits it.

These are defined as a module in our Cucumber env.rb file, and then mixed into the Cucumber World, so that steps automatically have access to these.

module Browser
  BROWSER = Watir::Browser.new ENV['WEB_DRIVER'] ||:firefox

  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new BROWSER, visit
    block.call page if block
    page
  end
end

World Browser

Summary

By introducing the on and visit methods, it means that we no longer need to have instance variables for page classes in our Cucumber steps. We also no longer need to worry about state transformations in pages, as these can be done in a modular way in the steps themselves. It means that when an error occurs initializing a page, it is more likely to to occur in the correct place. I find the steps more readible, as you only have to initialize the page once using the on block, and then can refer to page.

I have updated my set of Etsy Watir-WebDriver steps if you’d like to take a look.

What do you think? How will this scale?

Update: I have updated the WatirMelonCucumber project to use this style. It’s slightly different in that it supports two sites, and therefore the on method dynamically switches between these.

Cucumber Steps:

When /^I search for a? ?"([^"]*)"$/ do |term|
  on :home do |page|
    page.search_for term
  end
end

The on method:

def on page, visit=false, &block
  page_class = Object.const_get "#{@site}#{page.capitalize}Page"
  page = page_class.new BROWSER, visit
  block.call page if block
  page
end

SpecDriver now includes Etsy.com examples

A fellow ThoughtWorker, Paul Hammant, recently announced his intent for QuickTest Professional. And his method? Provide some new technology tutorials for newbies, in the form of a series of open source automated tests for craft site Etsy.com.

He asked me to assist by providing some SpecDriver examples, and since I enjoy doing this stuff I am more than happy to help out.

So, I’ve updated the SpecDriver repository on Github to include some Etsy.com feature files, and working tests. It was a lot of fun to write these tests, because Etsy.com is awesomely testable. Its nicely formatted code with proper ids and class names makes automating against it a real treat.

The benefit of doing this exercise is that it’s meaty enough for me to get some insight into how other people code real automated tests. For example, I’ve had a look into a couple of the code samples and I see a lot of XPath selectors. I personally really dislike XPath stuff, just like I dislike XML as I find it hard to read, and less intuitive than using straight identifiers. I am proud that there isn’t a single XPath used in my Etsy.com examples.

Another thing that stood out to me was how embedded some of the stories are. For example, in the JBehave example the features are located under //etsy-stories/src/main/resources/stories, which took me some time to locate at first, and every time I forget where they are. I much prefer having a features folder in root that makes it pretty obvious what’s in there.

One thing I am very interested in is seeing how people specify tests. I found two of Paul’s scenarios repetitious so I moved them into a single scenario outline. Moving them into a scenario outline made me think about some different ways to test, so I added a new scenario, which didn’t require much more code, but strengthened my feature.

For example, Paul’s two scenarios:


Scenario: Advanced Search for a hat
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for hat
  Then there are search results

Scenario: Advanced Search for a ring
  Given I am searching on Etsy.com
  When I specify the Jewelry sub category
  And I search for ring
  Then there are search results

became my:

Scenario Outline: Advanced Search for items in various categories that exist and don't exist
  Given I am searching on Etsy.com
    When I specify the <Sub Category> sub category
    And I search for <Search Term>
    Then I should see <Search Results> search results for <Search Term> 

Examples:
  | Sub Category | Search Term  | Search Results |
  | Knitting     | 'hat'        | some           |
  | Jewelry      | 'necklace'   | some           |
  | Jewelry      | 'specdriver' | no             |

While I was at it, I noticed there was a spelling correction feature of the advanced search. I also thought it was a good idea to test that this was working correctly.


Scenario: Misspelling a word corrects search automatically
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for 'scalf'
  Then I should see some search results for 'scarf'
  And I should see that the search was for 'scarf' instead of 'scalf'

Now I’ve done this exercise using SpecFlow and WebDriver in C#, I’m keen to do it again using Cucumber & Watir (WebDriver) in Ruby, to compare the implementation. When I am done I will post it on here for you all to see. Enjoy.

TrySelenium.org in Watir-WebDriver

Jason Huggins put out a call for some example scripts to tweet some stuff using Selenium.
Whilst I don’t agree with this being done through the Twitter GUI (there’s an API for this), being a Watir core team member and ruby guy, I thought I would use Watir-WebDriver.

My resulting script is:

require 'rubygems'
require 'highline/import'
require 'watir-webdriver'

begin
 username = ask("What is your twitter username?")
 password = ask("and your password?") { |q| q.echo = false }
 browser=  Watir::Browser.start "twitter.com", :firefox
 browser.span(:text => "Sign in").click
 browser.text_field(:id => "username").set username
 browser.text_field(:id => "password").set password
 browser.button(:id => "signin_submit").click
 raise "Could not log into Twitter: either your password is wrong or a CAPTCHA is needed."   if browser.url.include? "sessions"
 1.times do |attempt|
 browser.text_field(:class => "twitter-anywhere-tweet-box-editor").when_present.set "#SeConf is awesome x #{attempt}! http://tryselenium.org @Watir-WebDriver is even more awesome!"
 browser.link(:class => "tweet-button button").when_present.click
 browser.goto "twitter.com"
 end
ensure
 browser.close
end

And the resulting tweet:

Thanks for Jari Bakken for discussing this with me.

Reducing step explosion by decoupling Cucumber steps from your GUI

Yesterday I wrote about creating a simple page object pattern framework in Cucumber that uses Watir to drive browsers.

Ben Biddington left an excellent comment:

“The thing I like about this pattern is you could run the feature against a completely different search engine by just creating a new page object impl.

This prevents the common “step explosion” problem all too common with cucumber suites.”

Ben highlights an important point, the page object pattern decouples your cucumber steps from your GUI, helping to reduce “step explosion”.

To demonstrate this concept, I very easily modified my feature I wrote yesterday to test both Google and Bing, by using the same step definitions, but using scenario outlines instead of scenarios.

Feature: Internet Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

  Scenario Outline: Search for Watir
    Given I am on the  Home Page
    When I search for "Watir"
    Then I should see at least  results
    Scenarios:
      | search engine | expected number of  |
      | Google        | 100,000             |
      | Bing          |  85,000             |

  Scenario Outline: Do a unit conversion
    Given I am on the  Home Page
    When I convert 10 cm to inches
    Then I should see the conversion result ""
    Scenarios:
      | search engine | as expected                         |
      | Google        | 10 centimeters = 3.93700787 inches  |
      | Bing          | 10 centimetres = 3.937007874 inches |

  Scenario Outline: Do a search using data specified externally
    Given I am on the  Home Page
    When I search for a ridiculously small number of results
    Then I should see at most 5 results
    Scenarios:
      | search engine |
      | Google        |
      | Bing          |

The way I had written my step definitions before weren’t tied to a particular page (besides the naming, which I simply refactored), so all that was needed was to write two new page.rb files with the same methods as the Google pages.

Bing’s home page:

class BingHomePage

  attr_accessor :search_field, :bing_search_button

  URLS = { :production => "http://www.bing.com/" }

  def initialize(browser)
    @browser = browser
    @search_field         = @browser.text_field(:name => "q")
    @bing_search_button  = @browser.button(:name => "go")
  end

  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end

  def visit
    @browser.goto URLS[:production]
  end

  def search_for term
    self.search_field.set term
    self.bing_search_button.click
    bing_results_page = BingResultsPage.new(browser)
    bing_results_page.results.wait_until_present if WEBDRIVER
    bing_results_page
  end

and, Bing’s results page:

class BingResultsPage

  attr_accessor :results, :conversion_result

  def initialize(browser)
    @browser = browser
    @results = @browser.span(:id => "count")
    @conversion_result = @browser.span(:class => "sc_bigLine")
  end

  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end

  def number_search_results
    result = /^[\s\w-]* of ([\d,]+) results$/.match(@results.text)
    raise "Could not determine number of search results from: '#{@results.text}'" if not result
    result.captures[0].gsub(",","").to_i
  end

end

Once I had these pages in place, I didn’t have to require them or anything else, as this is done by Cucumber, so I could now run my cucumber command and get results from two different search engines: Bing and Google. The feature highlighted some subtle differences in how they work: Google had more hits, Bing uses the Australian spelling of centimetres, and adds one more decimal place of precision to conversions.

Summary

As Ben pointed out, and I have demonstrated, by using a page object pattern, you can decouple your steps from your GUI. This allows you to switch GUIs, or indeed testing methods, which in our involved switching search engines, without changing or adding more step definitions, reducing “step explosion”.

Source Code

I have pushed all changes to GitHub: https://github.com/alisterscott/WatirMelonCucumber

Results

My simple Cucumber + Watir page object pattern framework

Introduction

I was very impressed with Jeff Morgan, known as Cheezy, who recently wrote a series of blog posts about how to use Cucumber and Watir, and shared his code on Github.

I love it when people share their ideas this like, so I thought I would share what I have found useful when setting up a very simple Cucumber with Watir (Celerity & Watir-WebDriver) page object pattern framework, and how this compares to what Jeff has proposed.

Show me the code!

Before we begin, I’ll show you my code. It’s all on Github, right now, as we speak, and you can easily fork and clone this repository to play around with it. It uses a simple google search example.

Project Folder Structure

Google Search Feature

Feature: Google Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

Scenario: Search for Watir
  Given I am on the Google Home Page
  When I search for "Watir"
  Then I should see at least 100,000 results

Scenario: Do a unit conversion
  Given I am on the Google Home Page
  When I convert 10 cm to inches
  Then I should see the conversion result
            "10 centimeters = 3.93700787 inches"

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

Page Object Pattern

For our example, we have two page objects. The page classes delegate any methods that don’t exist on the page to the Browser object that is passed in from Cucumber. This ensures you can call browser specific methods (.title, .url etc.) at the page object level without duplicating methods.

class GoogleHomePage

  attr_accessor :search_field, :google_search_button

  URLS = { :production => "http://www.google.com/" }

  def initialize(browser)
    @browser = browser
    @search_field         = @browser.text_field(:name => "q")
    @google_search_button = @browser.button(:name => "btnG")
  end

  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end

  def visit
    @browser.goto URLS[:production]
  end

  def page_title
    @browser.title
  end

  def search_for term
    self.search_field.set term
    self.google_search_button.click
    google_results_page = GoogleResultsPage.new(browser)
    google_results_page.results.wait_until_present if WEBDRIVER
    google_results_page
  end

end

Cucumber Initialization

I have everything to do with initialization in support/env.rb. This initializes a browser once, and then uses the same browser instance for each Cucumber scenario. This is different from Jeff’s hooks.rb file that launches a new browser for every scenario. I have found by reusing the browser, you get much quicker test execution time.

The INDEX_OFFSET is a constant that allows me to use ‘index’ when referring to browser elements across both celerity with its 1-based index and watir-webdriver with its 0-based index.

TEST_DATA_DIR = "./features/test_data"

if ENV["HEADLESS"] then
  require "celerity"
  browser = Celerity::Browser.new
  INDEX_OFFSET = 0
  WEBDRIVER = false
else
  require 'watir-webdriver'
  require 'watir-webdriver/extensions/wait'
  browser = Watir::Browser.new :firefox
  INDEX_OFFSET = -1
  WEBDRIVER = true
end

Before do
  @browser = browser
end

at_exit do
  browser.close
end

Calling the page objects from the steps

As you can see, most of my steps are in a declarative style, so these normally align pretty well to a method on the page object.
I use a given step to create a page object (in our case @google_home_page), and then call methods on this object for when and then steps. I have created a dynamic invocation of the page creation, so it will work for all pages specified in the cucumber step itself. I use the search_for_term method to transition between page objects, as this method returns the new page. As a rule of thumb, I try to keep my step definition code to 1 or 2 lines, and at a fairly high level, which makes it much more readable.

Given /^I am on the (.+)$/ do |page_name|
  @google_home_page = Object.const_get(page_name.gsub(" ","")).new(@browser)
  @google_home_page.visit
end

When /^I search for a? ?"([^"]*)"$/ do |term|
  @google_results_page = @google_home_page.search_for term
end

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Then /^I should see at least ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should >= exp_num_results.gsub(",","").to_i
end

Then /^I should see at most ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should <= exp_num_results.gsub(",","").to_i
end

When /^I convert (.+)$/ do |conversion_statement|
  @google_results_page = @google_home_page.search_for "convert #{conversion_statement}"
end

Then /^I should see the conversion result "([^"]*)"$/ do |exp_conversion_result|
  @google_results_page.conversion_result.text.should == exp_conversion_result
end

Expressing test data external to the features/pages

Unlike Jeff’s recommendation of using default data on page objects, I prefer to store test data externally to the page objects in YAML files. That way I can describe the test data which is directly used in my Cucumber step. For example:

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

The phrase ridiculously small number of results matches directly to a section of my YAML test data file. This keeps the scenario highly readable, with the detailed data specified in the YAML file.

search terms:
  ridiculously small number of results:
    search term: 'macrocryoglobulinemia marvel'

A common method simply retrieves the appropriate test data, and provides it to the step to use. Note here that, as this step doesn’t contain quotation marks, this implies a description of data, rather than a literal string of data that would contain quotation marks around the string.

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Putting it all together
I have set up two profiles in the cucumber.yml file, one is the default that uses Firefox under Watir-Webdriver, and one runs Celerity (headless) and is named ci. It’s then a case of calling "cucumber" or "cucumber -p ci" to run each of these profiles.

I have set up a ./go script that bootstraps the entire process on unix based systems (Mac OSX and Linux).

The final result

Running cucumber provides results as green as a cucumber.

How does this compare to Cheezy’s framework I mentioned?

  • I am using a similar page object pattern to define web application pages, but I haven’t created a helper class to mix into each like he has: I simply use instance variables on a page class to define the GUI elements, instead of creating a series of methods for each object.
  • My page objects also delegate methods to the main Browser object as required, so you can directly call things like .title, .url etc.
  • Instead of using Jeff’s default data pattern, I instead prefer to store this data described in external YAML files instead for maximum flexibility.
  • Jeff talks about using mixins to mix in common objects. I haven’t done this, but it’s easily doable with my framework, and should be done as it scales out.

Summary

Jeff has done an excellent job with his series of posts and code (thank you), and I am hoping here to build on that, and show how it’s easy to set up a simple Cucumber & Watir page object pattern framework.

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: watirwebdriver.com for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at watir.com. If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server = Selenium::Server.new("/path/to/jar", :background => true)
server.start
# run your tests
server.stop

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :chrome
b.goto 'www.google.com'
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

The only difference for Firefox:

b = Watir::Browser.new :firefox

The only difference for IE:

b = Watir::Browser.new :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b = Watir::Browser.new(:remote, :url => 'http://127.0.0.1:4444/wd/hub', :desired_capabilities => capabilities)
b = Watir::Browser.new :firefox
b.goto "www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using browser.windows, but I haven’t tried this out yet.

Summary

Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading

Elegantly handling basic browser authentication with Watir

In programming, I find there’s little that compares to the feeling of deleting code; like finding a one or two-line solution that means you can remove a good-sized chunk of code no longer needed.

A fellow ThoughtWorker (who I shall call the Awesome-est Tom™) showed me a solution to handling the basic browser authentication in Internet Explorer, which effectively meant I could delete lines and lines of ruby code that called the AutoIt dll to handle the modal authentication dialog.

Internet Explorer Basic Authentication Dialog

Internet Explorer Basic Authentication Dialog

The code I had was loosely based upon the solutions available on the Watir wiki, all which roughly use the same technique of locating and authenticating the authentication dialog using AutoIt. Because the dialog is modal, the code I had used to run in a separate thread, and this caused me problems with debugging, and I find if I can avoid AutoIt, then my Watir tests run a lot more reliably.

Tom’s Occam’s Razor solution simply embeds the username and password in the URL that usually triggers an authentication dialog.

Instead of using:

Watir::Browser.start "http://yoururl.com"

you use:

Watir::Browser.start "http://username:password@yoururl.com"

This means you don’t see the authentication dialog at all! No more messy AutoIt code. The only tricky part is ensuring that Internet Explorer allows you to do this. Since Internet Explorer 6, this has been disabled by default, but it’s a simple matter of enabling the functionality by setting two registry keys.

This can easily be done running a .reg file you can create (I’ve uploaded mine ‘IESecurityURL.reg’ here), that sets the required values:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_HTTP_USERNAME_PASSWORD_DISABLE]
"iexplore.exe"=dword:00000000
"explorer.exe"=dword:00000000

You then run this file from the command line: regedit.exe /s IESecurityURL.reg and Robert is your father’s brother.

I’ve updated the wiki with this solution, so hopefully everyone can now use this and avoid the nastiness that is AutoIt :)

Running Watir reliably on a CI machine

I’ve found that Watir generally works well when running under Windows on a Continuous Integration machine (for example, Hudson or TeamCity) but if you use any of the autoit stuff to handle things like security authentication dialogs, it can fail when the machine is locked or not actively logged in.

I’ve found two ways to ensure Watir reliably runs tests without issues.

Run Caffeine to stop the machine locking or screensaver activating

Your Windows sys-admin might not like this one, but caffeine is a tiny executable that’s perfect for ensuring your Windows machine never locks or activates its screensaver, by simulating a keypress every 59 seconds. It’s a matter of installing it and a coffee cup appears in the tray to show you you’re machine is now on caffeine!

Use a command to release any remote desktop sessions to console

Caffeine works well, but if you use remote desktop to control the machine, when you disconnect, the machine will be locked and this will cause problems. What you need tp do is create a batch file, on your desktop say, that releases the RDP connection to console, so it’s ready to run any watir tests. The command you’ll need in the batch file is:

cmd.exe /Q /C "%windir%\System32\tscon.exe 0 /dest:console"

By using the combination of the above items, it’s easy to set up a reliable suite of Watir tests to run on a CI machine that you can easily remote into to see what is running when needed.