Sadly, an anonymous coward attacked myself and the company who I work (even though I don’t mention that company on this blog), so for the first time in this blog’s history, I have had to turn comment moderation on. I am sorry to the other genuine commenters whose comments have been lost in transition, and now have to wait for their new comments to be approved.

Since then I have received numerous emails asking where my article went, and commenting that people found it interesting and worthwhile. So I have decided to repost this article, hopefully with a little less contention this time around, making it clear, this is my opinion and experience: YMMV.

Intro

As a consultant I get to see and work on a lot of automated testing solutions using different automated web testing APIs. Lately I’ve been thinking about how these APIs are different and what makes them so.

My main interest is in ruby, and fortunately ruby has three solid examples of three different kinds of web testing APIs, two of which extend the lowest level API: selenium-webdriver.

I’ll (try to) explain here what I consider to be three kinds of automated web testing APIs and where I consider the sweet spot to be and and why.

A meaty example

As a carnivore, I thought I would explain my concept in terms I can relate to. If you’re a beef eater, there are many different kinds of beef that you can use to make some tasty food to eat. I’ll use three different kinds of beef for my example. The first (rawest) kind would involve getting a beef carcass and filleting it yourself to eventually make some edible food. The second kind of beef you could use is beef that is already in a slightly usable form, but you can then use yourself to make some edible food. For example, you can buy minced beef at a butcher, and then make your own hamburger patties, taco fillings etc from it. The final type of beef you could use is beef that has already been prepared so you can directly consume it, for example, sausages which can be cooked and consumed as is.

I consider these three examples of different kinds of beef to roughly correlate to automated web testing APIs, of which I also consider to be three kinds of.

The first is a Web Driver API, which is the rawest form of an API, its job is to drive a browser by issuing it commands. It provides a high level of user control, but like filleting a beef carcass it’s more ‘work’. An example in ruby of this API is the selenium-webdriver API, which controls the browser using the webdriver drivers.

The second kind of automated web testing API is the Browser API, which is a higher level API but still provides user control. This is the minced beef of APIs, as whilst it’s in a more usable form than a carcass, you still have a lot of control (and potential to what you can do with it). An example in ruby of this API is the watir-webdriver API, which uses the underlying selenium-webdriver carcass to control the browser.

The final kind of automated web testing API is the Web Form DSL (Domain Specific Language) which is a very high level API that provides users with specific methods to automate web forms and their elements. This is the beef sausages of APIs as sometimes you feel like eating something else besides sausages, but it’s difficult to make anything else edible but sausages from sausages. An example in ruby of this Web Form DSL is the Capybara DSL.

Visually, this looks something like this:

Show me the code™

So exactly what do these APIs look like?

I knew you’d ask, that’s why I came prepared.

Say I want to accomplish a fairly basic scenario on my example Google Doc form:

• Start a browser
• Navigate to the watir-webdriver-demo form
• Check whether text field with id ‘entry_0′ exists (this should exist)
• Check whether text field with id ‘entry_99′ exists (this shouldn’t exist)
• Set a text field with id ‘entry_0′ to ’1′
• Set a text field with id ‘entry_0′ to ’2′
• Select ‘Ruby’ from select list with id ‘entry_1′
• Click the Submit button

This is how I would do it in the three different APIs:

# * Start browser
# * Navigate to watir-webdriver-demo form
# * Check whether text field with id 'entry_0' exists
# * Check whether text field with id 'entry_99' exists
# * Set text field with id 'entry_0' to '1'
# * Set text field with id 'entry_0' to '2'
# * Select 'Ruby' from select list with id 'entry_1'
# * Click the Submit button

require 'bench'

benchmark 'selenium-webdriver' do
  require 'selenium-webdriver'

  driver = Selenium::WebDriver.for :firefox
  driver.navigate.to 'http://bit.ly/watir-webdriver-demo'
  begin
    driver.find_element(:id, 'entry_0')
  rescue Selenium::WebDriver::Error::NoSuchElementError
    # doesn't exist
  end
  begin
    driver.find_element(:id, 'entry_99').displayed?
  rescue Selenium::WebDriver::Error::NoSuchElementError
    # doesn't exist
  end
  driver.find_element(:id, 'entry_0').clear
  driver.find_element(:id, 'entry_0').send_keys '1'
  driver.find_element(:id, 'entry_0').clear
  driver.find_element(:id, 'entry_0').send_keys '2'
  driver.find_element(:id, 'entry_1').find_element(:tag_name => 'option', :value => 'Ruby').click
  driver.find_element(:name, 'submit').click
  driver.quit
end

benchmark 'watir-webdriver' do
  require 'watir-webdriver'
  b = Watir::Browser.start 'bit.ly/watir-webdriver-demo', :firefox
  b.text_field(:id => 'entry_0').exists?
  b.text_field(:id => 'entry_99').exists?
  b.text_field(:id => 'entry_0').set '1'
  b.text_field(:id => 'entry_0').set '2'
  b.select_list(:id => 'entry_1').select 'Ruby'
  b.button(:name => 'submit').click
  b.close
end

benchmark 'capybara' do
  require 'capybara'
  session = Capybara::Session.new(:selenium)
  session.visit('http://bit.ly/watir-webdriver-demo')
  session.has_field?('entry_0') # => true
  session.has_no_field?('entry_99') # => true
  session.fill_in('entry_0', :with => '1')
  session.fill_in('entry_0', :with => '2')
  session.select('Ruby', :from => 'entry_1')
  session.click_button 'Submit'
  session.driver.quit
end

run 10

This is how long they took for me to run:

                        user     system      total        real
selenium-webdriver  1.810000   0.840000  22.130000 ( 73.123340)
watir-webdriver     1.940000   0.870000  24.380000 ( 79.388494)
capybara            1.950000   0.890000  24.080000 ( 79.920051)

Note: Capybara doesn’t always require a ‘session’, it’s only for non ruby rack applications, but since my example (Google) is not a rack application, as are most of the applications I test, my example must use the session.

When using ruby, why Watir-WebDriver is my sweet spot

I personally find Watir-WebDriver to be the most elegant solution, as the API is high enough for me to be highly readable/usable, but low enough to be powerful and for me to feel like I’m in control.

For example, being able to select an element by a explicit identifier (name, class name, id, anything) is a huge deal to me. I personally don’t like relying on the API to determine which selector to use: for example Capybara only supports name, id and label, but you can’t tell fill_in which specific one to choose: it appears to try each selector one by one until it finds it.

I have found that Watir-WebDriver also also provides lots of flexibility/neatness. For example: it’s the only API shown here that allows URLs to not have a ‘http://’ prefix (how many people do you know who type in http:// into a browser?).

In my opinion, the high level APIs like Capybara don’t provide enough control (for example – being able to specify the explicit selector), but the low level APIs like webdriver don’t provide enough functionality. This is evident when I am using a language other than ruby (like C#) when I find myself writing a large number of web element extension methods because webdriver doesn’t provide any of them. A .set method is a classic example, even Simon Stewart writes a clearAndType method in his examples even though he wrote webdriver which sadly misses it (you must call .clear, and .send_keys).

My biggest concern about high level field APIs

But my biggest issue with the high level APIs is that I’ve frequently seen them used to write test scripts that are step by step interactions with a web form. Instead of thinking of a business application as that, people see it as a series of forms that you ‘fill in’. This means people create scenarios like Aslak Hellesøy included in his recent post about cucumber web steps (which uses Capybara) and the problems it has created.

Scenario: Successful login
  Given a user "Aslak" with password "xyz"
  And I am on the login page
  And I fill in "User name" with "Aslak"
  And I fill in "Password" with "xyz"
  When I press "Log in"
  Then I should see "Welcome, Aslak"

I’m not saying it’s not possible to end up with something as ugly as above using other APIs, but I am saying the web form DSL style naturally relates to this: as the APIs look so similar to this style because that’s what the DSL was designed for: filling in forms. I’ve seen people frequently write generic, reusable cucumber steps to match the web form DSL like:

When /^I fill in "(.+)" with "(.+)"$/ do |value, field|
  fill_in field, :with => value
end

But this means you end up with less readable, less maintainable test scripts rather than business readable executable specifications.

Summary

Ultimately what I am looking for in an automated web testing API is simplicity and full control. I personally find browser APIs like Watir-WebDriver and Watir give me this, and this is why I love them so. Your mileage may vary, you may like different styles of APIs better, but I’ve seen other APIs so badly abused by people not even thinking about it, so it makes sense to think about what you’re trying to achieve and whether what you’re doing is the right way.

Watir-WebDriver 0.3.1 has recently been released, and now there is a configurable option to specify that you always want to create a fresh reference when using an element.

 Watir::always_locate = true b = Watir::Browser.new 

This will result in a performance hit, but in my experience, it isn’t too bad. Jari Bakken ran WatirSpec using this config which resulted in a 10 second increase in running time (from 145 seconds to 155 seconds) which in my opinion is worth it to have more reliable test results.

Thanks to Jari for working on this. Hope you enjoy.

Update: 15 August 2011: I only just realized this is enabled by default, so you’ll get it automatically when you update your gem. Neat.

Selenium::WebDriver::Error::ObsoleteElementError: Element is no longer attached to the DOM

Now, it’s quite annoying when this happens, and it’s not consistent, but I will try to explain what is happening and how you can try to avoid it.

As Jari Bakken explains on the Watir mailing list, in vanilla Watir, when you define an element, the details of how to locate an element are stored, and the actual finding is done when you need to use the element.

This is dramatically different to Watir-WebDriver that stores an reference to the actual DOM element, so if this changes (the page refreshes or JavaScript updates the DOM for example) then the exception above is raised.

I’ve found there’s a couple of ways to reduce, but not entirely eliminate, the chance of these exceptions being raised. The main idea is to avoid looping over DOM elements, especially when the pages you are testing have a lot of heavy JavaScript.

Instead of looping over the elements like this:

browser.links.each do |link|
  link.click
  browser.goto home
end

I have found it’s better to collect the hrefs of each, and use these collected values:

links = browser.links.collect { |link| link.href }
links.each do |link|
  browser.goto link
  browser.goto home
end

Instead of looping through a element collection to find some element value:

def error_exists? text
  errors = browser.spans(:class => 'error').collect

  errors.each do |span|
    return true if span.text == text
  end
  false
end

I have found it is better to directly find the element itself

def error_exists? text
  browser.span(:class => 'error', :text => text).when_present.exists?
end

Summary

The ObseleteElementError is a definitely a pain, but unless the entire approach to locating elements in Watir-Webdriver is changed, there is at least some way I have shown to minimize the pain.

 browser.send_keys("{ENTER}") 

To do the same thing in Watir-WebDriver, you do this:

 browser.send_keys :enter 

You can also do things like this:

browser.element.send_keys [:control, 'a'], :backspace

The full list of keys are available from here:

• :null
• :cancel
• :help
• :backspace
• :tab
• :clear
• :return
• :enter
• :shift
• :left_shift
• :control
• :left_control
• :alt
• :left_alt
• :pause
• :escape
• :space
• :page_up
• :page_down
• :end
• :home
• :left
• :arrow_left
• :up
• :arrow_up
• :right
• :arrow_right
• :down
• :arrow_down
• :insert
• :delete
• :semicolon
• :equals
• :numpad0
• :numpad1
• :numpad2
• :numpad3
• :numpad4
• :numpad5
• :numpad6
• :numpad7
• :numpad8
• :numpad9
• :multiply
• :add
• :separator
• :subtract
• :decimal
• :divide
• :f1
• :f2
• :f3
• :f4
• :f5
• :f6
• :f7
• :f8
• :f9
• :f10
• :f11
• :f12
• :meta
• :command

Anyone who has used Watir-WebDriver (or WebDriver for that matter) to test Google Chrome will know that it’s been pretty unreliable, and pretty much unusable in the past… until now.

The ChromeDriver executable was released recently which enables WebDriver to control Google Chrome natively using the Wire protocol. This means your Watir-WebDriver tests will run super fast, and super reliably.

For Mac OSX:

1. Get the latest Watir-WebDriver gem

   gem update watir-webdriver

2. Download the ChromeDriver binary from the Chromium site, and copy it to your path (such as /usr/local/bin/)
3. This will work with any current version of Chrome
4. Change your Watir-WebDriver script to use Watir::Browser.new :chrome and voila, super fast Chrome, with no weird first tab opened in the background. This is the key to tell if you’re doing it right, your app should appear in the first tab of Chrome, not the second.

I haven’t done this on Windows yet, so your mileage may vary, but I imagine it wouldn’t be too different.

When /^an item is added to the cart$/ do
  @advanced_search_page = EtsyAdvancedSearchPage.new @browser, true
  @search_results_page = @advanced_search_page.search_for 'hat'
  @etsy_item_page = @search_results_page.click_first_result
  @item_title = @etsy_item_page.item_title
  @etsy_cart_page = @etsy_item_page.click_add_to_cart
end

Here I am using instance variables in Cucumber steps to store references to pages that I am interacting with. As you can see, it’s quite easy to lose track of which page is which.

If I want to use one of these pages in another step, I have to refer to the same variable name. The key to this concept is having pages return pages, so you don’t have to initialize pages in your steps (except for the first visit, which normally happens only once per scenario).

A colleague and good friend of mine has been showing me some alternatives. One alternative to this is to actually dynamically refer to pages only when you need them. This means you don’t need to return a page from a page, as they are always initialized when needed. The above method would look like:

When /^an item is added to the cart$/ do
  visit EtsyAdvancedSearchPage do |page| page.search_for 'hat' end
  on EtsySearchResultsPage do |page| page.click_first_result end
  on EtsyItemPage do |page|
    @item_title = page.item_title
    page.click_add_to_cart
  end
end

This introduces two new methods at the Cucumber step level that need to be created: on, and visit. The on method simply creates a new page object so you can call methods from it. As each page initialize checks expected title and expected element, it will raise an error automatically if either are incorrect. The visit method is an extension of on, which actually initalizes the page and visits it.

These are defined as a module in our Cucumber env.rb file, and then mixed into the Cucumber World, so that steps automatically have access to these.

module Browser
  BROWSER = Watir::Browser.new ENV['WEB_DRIVER'] ||:firefox

  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new BROWSER, visit
    block.call page if block
    page
  end
end

World Browser

Summary

By introducing the on and visit methods, it means that we no longer need to have instance variables for page classes in our Cucumber steps. We also no longer need to worry about state transformations in pages, as these can be done in a modular way in the steps themselves. It means that when an error occurs initializing a page, it is more likely to to occur in the correct place. I find the steps more readible, as you only have to initialize the page once using the on block, and then can refer to page.

I have updated my set of Etsy Watir-WebDriver steps if you’d like to take a look.

What do you think? How will this scale?

Update: I have updated the WatirMelonCucumber project to use this style. It’s slightly different in that it supports two sites, and therefore the on method dynamically switches between these.

Cucumber Steps:

When /^I search for a? ?"([^"]*)"$/ do |term|
  on :home do |page|
    page.search_for term
  end
end

The on method:

def on page, visit=false, &block
  page_class = Object.const_get "#{@site}#{page.capitalize}Page"
  page = page_class.new BROWSER, visit
  block.call page if block
  page
end

This is loosely based upon the great work that Jeff Morgan did in his series of UI testing blog posts, but takes the concept a lot further IMO.

I wanted it to be solid before I released it as a gem, so I wrote unit tests for every method, and wrote yard tagged documentation for each method also. You can see the documentation automatically generated on rdoc.info.

Installation

gem install watir-page-helper

Example

require 'watir-webdriver'
require 'watir-page-helper'

class MyPage
  include WatirPageHelper

  direct_url "http://www.google.com"
  expected_element :text_field, :name => "q"
  expected_title "Google"
  text_field :search_box, :name => "q"
  button :search, :name => "btnG"

  def initialize browser, visit = false
    @browser = browser
    goto if visit

    expected_element if respond_to? :expected_element
    has_expected_title? if respond_to? :has_expected_title?
  end
end

browser = Watir::Browser.new :chrome
page = MyPage.new browser, true
page.search_box = "Watirmelon" #This method is created by WatirPageHelper
page.search #This method is created by WatirPageHelper also
browser.close

The above example is very basic, but shows the concept well. You’re much better off putting the initialize method into a base page class so you don’t need to call the same methods for every page.

What you get

Page Methods

• direct_url: allows you to navigate to a page upon initialization, if visit is set to true
• expected_title: allows you to automatically assert the expected title of the page when it is initialized
• expected_element: allows you to initialize the page by looking for a certain element. This is useful for pages that load dynamic content.

Element Methods

Element methods for a majority of the Watir-WebDriver supported elements which generate useful helper methods.
For example: text_field, select_list, radio_button, form, div, span, h1..h6 etc.

Summary

I’ve been using a watir page helper on a few different projects now and I’m sick of copy-and-pasting the code each time, so I thought it’d be nice to package it as a gem.

I also think it’s important to have unit tests for this kind of stuff, so it was a good opportunity to write unit tests for every method in these classes. The documentation was an added bonus too.

I hope you find this useful, please let me know if anything doesn’t make any sense.

Links/Further Info

• The source code on github
• The gem on rubygems.org.
• The rdoc documentation for each method on rdoc.info.

You can check out and download my ruby code on my Github project page.

• We organized and promoted Watir Day in a very lightweight way, but managed to attract over 60 attendees and 10 sponsors!
• The key to Watir’s success has been its community, focus on building a tool for testers, and not trying to be everything to everyone.
• Watir is seen as a secret sauce for a lot of web companies – they are very hesitant to talk about it publically in case their competitors copy them. But we managed to hear a few of those stories for the first time on the day:
  • Watir is a key part of Facebook’s engineering culture: they have been extremely successful with Watir, after failing with Selenium.
  • Convio has been so successful with in house open source test automation using Watir that they have phased out outsourced testing altogether, and are hiring more automated test engineers than ever.
• An individual’s involvement in an open source project can be inversely proportional to how happy they are in their job – not happy at work = less commits, more happy at work = less commits (via Bret Pettichord)
• The future is bright for Watir. Using WebDriver technology will enable us to deliver a tool designed for testers to testers, but an increasing range of browser and device support.

The questionnaire results speak for themselves:

The slides are available online.