AMA: Trunk Guardian Service?

Sue asks…

I read a LinkedIn blog post from 2015 by Keqiu Hu from LinkedIn about flaky UI tests. He explains how they fixed their flaky UI tests for the LinkedIn app. Among other things they implemented what they called the “Trunk Guardian service” which runs automated UI tests on the last known good build twice and if the test passes on the first run but fails on the second it is marked as ‘flaky’ and disabled and the owner is notified to fix it or get rid of it. I wondered what your thoughts were on such a “Trunk Guardian service” – if the culture / process was in place to solve the other issues that create flaky tests, could such a thing be worth the effort to implement? Article: Test Stability – How We Make UI Tests Stable

My response…

Continue reading “AMA: Trunk Guardian Service?”

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: watirwebdriver.com for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at watir.com. If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server = Selenium::Server.new("/path/to/jar", :background => true)
server.start
# run your tests
server.stop

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :chrome
b.goto 'www.google.com'
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

The only difference for Firefox:

b = Watir::Browser.new :firefox

The only difference for IE:

b = Watir::Browser.new :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b = Watir::Browser.new(:remote, :url => 'http://127.0.0.1:4444/wd/hub', :desired_capabilities => capabilities)
b = Watir::Browser.new :firefox
b.goto "www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using browser.windows, but I haven’t tried this out yet.

Summary

Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading

Cucumber: imperative or declarative? that is the question

I’ve recently been putting some amount of thought into how to write good Cucumber features. One the reasons I love Cucumber so much is its language focus; meaning it makes you think about things like sentence function more than other testing frameworks I have used. One element of sentence function usage in Cucumber that’s important is whether you should write imperative (communicative) or declarative (informative) sentences for your steps. I’ll provide a couple of examples to show the differences.

An imperative (communicative) Cucumber scenario

Scenario: Submit the form with correct data (imperative)
  Given I am on the watir example form
  When I set "What is your name?" to "Alister"
  And I set "What is your story?" to "I like ruby"
  And I set "What testing tool do you like?" to "Watir"
  And I click "Submit"
  Then I should see the message "Thanks! Your response has been recorded."

Each step is a command, or instruction, so it’s pretty clear what is being done.

A declarative (informative) Cucumber scenario

Scenario: Submit the form with correct data (declarative)
  Given I am on the watir example form
  When I submit the form with correct data
  Then I should see the correct message

Each step states an idea, so sometimes it’s not really clear precisely what is being done.

Imperative vs Declarative

There’s some quite good articles about writing Cucumber features/scenarios around. One such article that stood out to me was written in May 2008 about RSpec Story Runner, and whether you should use an imperative or declarative style. The answer is, of course, it depends, but most people tend to lean towards the declarative style, just the way that a lot of programmers prefer declarative programming. I am actually swinging towards imperative style cucumber features and steps; the reasons for which I will explain below.

Running Imperative Features gives you richer output

With the two examples above, I have kinda cheated, in that the step definitions for the declarative style calls imperative steps. For example, this is the declarative step defintion:

When /^I submit the form with correct data$/ do
  When "I set \"What is your name?\" to \"Alister\""
  And "I set \"What is your story?\" to \"I like ruby\""
  And "I set \"What testing tool do you like?\" to \"Watir\""
  And "I click \"Submit\""
end

This is all well and good, as you can now have multiple cucumber steps that call the same thing, it’s DRYish, but the problem is when you run the step “When I submit the form with correct data”, that’s all you get.

Compare this to the imperative step results, which whilst calling exactly the same code, give you a lot more richness.

Writing imperative scenarios allow you to utilise things advanced cucumber functionality

Two table based Cucumber features I like using are scenario outlines, and multiline step arguments, but these don’t make a lot of sense if you’re using a declarative style.
For example, the imperative scenario could be easily adapted to use a scenario outline, whereas the declarative style would mean writing more declarative steps.

Scenario Outline: Submit the form with correct data (imperative - outline)
  Given I am on the watir example form
  When I set "What is your name?" to "<name>"
  And I set "What is your story?" to "<story>"
  And I set "What testing tool do you like?" to "<tool>"
  And I click "Submit"
  Then I should see the message "Thanks! Your response has been recorded."

  Examples:
   | name    | story       | tool     |
   | Alister | I Like ruby | Watir    |
   | Mozart  | I like java | Selenium |

Writing declarative scenarios often means hidden detail

As you move to more and more declarative scenarios, more and more details become hidden in cucumber step definitions and library code. As Rasmus pointed out on Jared Quinert’s blog on does Cucumber suck:

…tests end up looking like the one test to rule them all™:
Given everything is set up
When I submit correct data
Then everything should be OK

Summary

To wrap up, I’m not saying don’t go declarative, what I am saying is that you lose certain things in Cucumber when you do, but whether this is a big deal or not depends on what you’re trying to get out of it. In the meantime, I’ll continue to experiment and see what works and what doesn’t.

How to set up Cucumber and Watir on OSX

These are the steps I had to use to get Cucumber and Watir running on OSX. It’s a shame about step 1, it’s a real pain but I don’t know any way around it.

  1. Install Xcode from OSX installation DVD – this takes about 15 minutes
  2. Open a command prompt and use enter the following commands.
  3. sudo gem update --system
  4. sudo gem install rspec --no-rdoc --no-ri
  5. sudo gem install gherkin --no-rdoc --no-ri
  6. sudo gem install cucumber --no-rdoc --no-ri
  7. sudo gem install firewatir --no-rdoc --no-ri

You should be good to go.

Elegantly handling basic browser authentication with Watir

In programming, I find there’s little that compares to the feeling of deleting code; like finding a one or two-line solution that means you can remove a good-sized chunk of code no longer needed.

A fellow ThoughtWorker (who I shall call the Awesome-est Tom™) showed me a solution to handling the basic browser authentication in Internet Explorer, which effectively meant I could delete lines and lines of ruby code that called the AutoIt dll to handle the modal authentication dialog.

Internet Explorer Basic Authentication Dialog
Internet Explorer Basic Authentication Dialog

The code I had was loosely based upon the solutions available on the Watir wiki, all which roughly use the same technique of locating and authenticating the authentication dialog using AutoIt. Because the dialog is modal, the code I had used to run in a separate thread, and this caused me problems with debugging, and I find if I can avoid AutoIt, then my Watir tests run a lot more reliably.

Tom’s Occam’s Razor solution simply embeds the username and password in the URL that usually triggers an authentication dialog.

Instead of using:

Watir::Browser.start "http://yoururl.com"

you use:

Watir::Browser.start "http://username:password@yoururl.com"

This means you don’t see the authentication dialog at all! No more messy AutoIt code. The only tricky part is ensuring that Internet Explorer allows you to do this. Since Internet Explorer 6, this has been disabled by default, but it’s a simple matter of enabling the functionality by setting two registry keys.

This can easily be done running a .reg file you can create, that sets the required values:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_HTTP_USERNAME_PASSWORD_DISABLE]
"iexplore.exe"=dword:00000000
"explorer.exe"=dword:00000000

You then run this file from the command line: regedit.exe /s IESecurityURL.reg and Robert is your father’s brother.

Rubular & RubyMine: makes Cucumber easier

I’ve spent the last couple of months establishing Acceptance Test Driven Development on a medium sized software delivery project using Watir & Cucumber.

Cucumber is premised on reading feature files and matching strings to determine what to do, which is done by using regular expressions in step files. This means constant use of ruby regular expressions, and in polishing up on my regular expression skills, I have found rubular incredidly useful. I love the simple layout and the ‘Regex Quick Reference’ at the bottom, just where you need it. It’s very well done.

Rubular: A Ruby regular expression editor and tester
Rubular: A Ruby regular expression editor and tester

The other tool I have found incredibly useful is RubyMine. Up until now, I’ve never really bothered with an IDE for Watir stuff. I’ve mainly stuck to text editors before, but since I have started using Cucumber, I have found the RubyMine tool almost critical, as its support for Cucumber feature files and step definitions is superb. It features click through linking for feature files so you know exactly what step you’re calling, and the debugger is awesome; no more puts statements for me :)

So, if you’re thinking about implementing Cucumber, or using Cucumber but are annoyed with the lack of efficiency in managing a large suite of step definitions, I would thoroughly recommend these two tools. Rubular is free, and I believe RubyMine is about $99, but less if you need multiple licenses. There’s also an Early Access Program where you can use RubyMine for free as long as you’re happy to test it along the way.

Running Watir reliably on a CI machine

I’ve found that Watir generally works well when running under Windows on a Continuous Integration machine (for example, Hudson or TeamCity) but if you use any of the autoit stuff to handle things like security authentication dialogs, it can fail when the machine is locked or not actively logged in.

I’ve found two ways to ensure Watir reliably runs tests without issues.

Run Caffeine to stop the machine locking or screensaver activating

Your Windows sys-admin might not like this one, but caffeine is a tiny executable that’s perfect for ensuring your Windows machine never locks or activates its screensaver, by simulating a keypress every 59 seconds. It’s a matter of installing it and a coffee cup appears in the tray to show you you’re machine is now on caffeine!

Use a command to release any remote desktop sessions to console

Caffeine works well, but if you use remote desktop to control the machine, when you disconnect, the machine will be locked and this will cause problems. What you need tp do is create a batch file, on your desktop say, that releases the RDP connection to console, so it’s ready to run any watir tests. The command you’ll need in the batch file is:

cmd.exe /Q /C "%windir%\System32\tscon.exe 0 /dest:console"

By using the combination of the above items, it’s easy to set up a reliable suite of Watir tests to run on a CI machine that you can easily remote into to see what is running when needed.