Tag Archives: Automated Testing

Take control of your own career

During my career, I’ve come across numerous testing colleagues with no experience in automated testing who say things like “I’d love to do automated testing”. They expect to be put into an automated testing role so they can learn automated testing.

I don’t think it should work like that. Your employer shouldn’t be solely responsible for you enhancing your skills and progressing your career.

And, the thing is, it’s never been easier to pick up some new technical skills.

If you want to learn programming start by learning something like Ruby. If you want to learn about automated web testing learn Watir. If you want to learn about behavior driven development tools learn Cucumber.

I taught myself Ruby. I taught myself Watir. I taught myself C#, Python, Selenium, Cucumber and Jenkins. The list goes on.

The barrier to entry has never been lower. Try codeacademy, try ruby koans, download the free watir book, buy Cheezy’s cheap eBook about Watir & Cucumber.

So, instead of watching television or going out for drinks, spend your nights and weekends learning some new skills and taking control of your career instead of expecting your employer to hand it to you on a plate.

You’ll then be able to say “I’m learning all about Watir at the moment and I would love to apply that on a project” instead of “I’d love to do automated testing”.

Which is better: manual or automated testing?

This post is part of the Pride & Paradev series.


Which is better: manual or automated testing?


Manual Testing is better than Automated Testing

Manual testing is better than automated testing. Even when automating a test scenario, you have to manually test it at least once anyway to automate it, so automated testing can’t be done without manual testing. And you have to manually check the automated test results also.

Automated tests can stop working by something as simple as an unexpected pop-up dialog which can be quickly analyzed and dismissed when manually testing.

Manual testing is a sapient activity: one that requires human judgement. As you are testing you are using implicit knowledge to judge whether or not something is working as expected. This enables you to find extra bugs that automated tests would never find. It also allows you to follow smells you find to explore areas that may not have been tested or required.

Manual testing is also helpful for finding layout issues and trivial bugs which wouldn’t be found by an automated test, as you’re fully observing the application as you’re using it. Usability issues are also identifiable by manual testing but can’t be discovered through writing and running automated test scripts.

Automated Testing is better than Manual Testing

Automated testing is better than manual testing. Automated tests are very explicit (black and white) so you have a much higher chance of reproducing a bug if found by an automated test by knowing what the automated test executed to achieve the result. Because the automated tests are explicit, they also execute consistently as they don’t get tired and/or lazy like us humans.

Automated tests are quicker to run than manual tests as there’s no lag time between input and checking, and this means you can run more tests in more browsers more quickly. Manually testing the same functionality in, for example, 8 browsers and 4 devices is tiring, but can easily be achieved with automated tests.

Automated tests also allow you to test things that aren’t manually possible. For example, answering a question like ‘what if I had 200 accounts’, or ‘what if I processed ten transactions simultaneously’ can only be answered efficiently by using automated tests.

Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser = Watir::Browser.new

Before do
  @browser = browser
end

at_exit do
  browser.close
end

To open and close the browser only once in RSpec:

browser = Watir::Browser.new

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }
end

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = 'https://en.wikipedia.org'
end

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"
end

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"
end

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'
end

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'
end

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

My simple Cucumber + Watir page object pattern framework

Introduction

I was very impressed with Jeff Morgan, known as Cheezy, who recently wrote a series of blog posts about how to use Cucumber and Watir, and shared his code on Github.

I love it when people share their ideas this like, so I thought I would share what I have found useful when setting up a very simple Cucumber with Watir (Celerity & Watir-WebDriver) page object pattern framework, and how this compares to what Jeff has proposed.

Show me the code!

Before we begin, I’ll show you my code. It’s all on Github, right now, as we speak, and you can easily fork and clone this repository to play around with it. It uses a simple google search example.

Project Folder Structure

Google Search Feature

Feature: Google Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

Scenario: Search for Watir
  Given I am on the Google Home Page
  When I search for "Watir"
  Then I should see at least 100,000 results

Scenario: Do a unit conversion
  Given I am on the Google Home Page
  When I convert 10 cm to inches
  Then I should see the conversion result 
            "10 centimeters = 3.93700787 inches"

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

Page Object Pattern

For our example, we have two page objects. The page classes delegate any methods that don’t exist on the page to the Browser object that is passed in from Cucumber. This ensures you can call browser specific methods (.title, .url etc.) at the page object level without duplicating methods.

class GoogleHomePage

  attr_accessor :search_field, :google_search_button

  URLS = { :production => "http://www.google.com/" }

  def initialize(browser)
    @browser = browser
    @search_field         = @browser.text_field(:name => "q")
    @google_search_button = @browser.button(:name => "btnG")
  end

  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end

  def visit
    @browser.goto URLS[:production]
  end

  def page_title
    @browser.title
  end

  def search_for term
    self.search_field.set term
    self.google_search_button.click
    google_results_page = GoogleResultsPage.new(browser)
    google_results_page.results.wait_until_present if WEBDRIVER
    google_results_page
  end

end

Cucumber Initialization

I have everything to do with initialization in support/env.rb. This initializes a browser once, and then uses the same browser instance for each Cucumber scenario. This is different from Jeff’s hooks.rb file that launches a new browser for every scenario. I have found by reusing the browser, you get much quicker test execution time.

The INDEX_OFFSET is a constant that allows me to use ‘index’ when referring to browser elements across both celerity with its 1-based index and watir-webdriver with its 0-based index.

TEST_DATA_DIR = "./features/test_data"

if ENV["HEADLESS"] then
  require "celerity"
  browser = Celerity::Browser.new
  INDEX_OFFSET = 0
  WEBDRIVER = false
else
  require 'watir-webdriver'
  require 'watir-webdriver/extensions/wait'
  browser = Watir::Browser.new :firefox
  INDEX_OFFSET = -1
  WEBDRIVER = true
end

Before do
  @browser = browser
end

at_exit do
  browser.close
end

Calling the page objects from the steps

As you can see, most of my steps are in a declarative style, so these normally align pretty well to a method on the page object.
I use a given step to create a page object (in our case @google_home_page), and then call methods on this object for when and then steps. I have created a dynamic invocation of the page creation, so it will work for all pages specified in the cucumber step itself. I use the search_for_term method to transition between page objects, as this method returns the new page. As a rule of thumb, I try to keep my step definition code to 1 or 2 lines, and at a fairly high level, which makes it much more readable.

Given /^I am on the (.+)$/ do |page_name|
  @google_home_page = Object.const_get(page_name.gsub(" ","")).new(@browser)
  @google_home_page.visit
end

When /^I search for a? ?"([^"]*)"$/ do |term|
  @google_results_page = @google_home_page.search_for term
end

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Then /^I should see at least ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should >= exp_num_results.gsub(",","").to_i
end

Then /^I should see at most ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should <= exp_num_results.gsub(",","").to_i
end

When /^I convert (.+)$/ do |conversion_statement|
  @google_results_page = @google_home_page.search_for "convert #{conversion_statement}"
end

Then /^I should see the conversion result "([^"]*)"$/ do |exp_conversion_result|
  @google_results_page.conversion_result.text.should == exp_conversion_result
end

Expressing test data external to the features/pages

Unlike Jeff’s recommendation of using default data on page objects, I prefer to store test data externally to the page objects in YAML files. That way I can describe the test data which is directly used in my Cucumber step. For example:

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

The phrase ridiculously small number of results matches directly to a section of my YAML test data file. This keeps the scenario highly readable, with the detailed data specified in the YAML file.

search terms:
  ridiculously small number of results:
    search term: 'macrocryoglobulinemia marvel'

A common method simply retrieves the appropriate test data, and provides it to the step to use. Note here that, as this step doesn’t contain quotation marks, this implies a description of data, rather than a literal string of data that would contain quotation marks around the string.

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Putting it all together
I have set up two profiles in the cucumber.yml file, one is the default that uses Firefox under Watir-Webdriver, and one runs Celerity (headless) and is named ci. It’s then a case of calling "cucumber" or "cucumber -p ci" to run each of these profiles.

I have set up a ./go script that bootstraps the entire process on unix based systems (Mac OSX and Linux).

The final result

Running cucumber provides results as green as a cucumber.

How does this compare to Cheezy’s framework I mentioned?

  • I am using a similar page object pattern to define web application pages, but I haven’t created a helper class to mix into each like he has: I simply use instance variables on a page class to define the GUI elements, instead of creating a series of methods for each object.
  • My page objects also delegate methods to the main Browser object as required, so you can directly call things like .title, .url etc.
  • Instead of using Jeff’s default data pattern, I instead prefer to store this data described in external YAML files instead for maximum flexibility.
  • Jeff talks about using mixins to mix in common objects. I haven’t done this, but it’s easily doable with my framework, and should be done as it scales out.

Summary

Jeff has done an excellent job with his series of posts and code (thank you), and I am hoping here to build on that, and show how it’s easy to set up a simple Cucumber & Watir page object pattern framework.

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: watirwebdriver.com for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at watir.com. If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server = Selenium::Server.new("/path/to/jar", :background => true)
server.start
# run your tests
server.stop

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :chrome
b.goto 'www.google.com'
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

The only difference for Firefox:

b = Watir::Browser.new :firefox

The only difference for IE:

b = Watir::Browser.new :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b = Watir::Browser.new(:remote, :url => 'http://127.0.0.1:4444/wd/hub', :desired_capabilities => capabilities)
b = Watir::Browser.new :firefox
b.goto "www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using browser.windows, but I haven’t tried this out yet.

Summary

Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading

Upcoming agile testing meetup in Brisbane, Australia

I just got this email from Thoughtworks, and have registered to attend. See you there if you’re in Brisbane.

You are invited to the launch meeting of the Brisbane chapter of the Agile Alliance Australia (AAA). Last year’s inaugural Agile Australia conference established the AAA and it is envisaged that local chapters will provide ongoing education and knowledge-sharing.

ThoughtWorks Global Testing Practice Lead Kristan Vingrys will be our guest speaker discussing the Changing Role of a Tester. Kristan has over 10 years experience in software testing encompassing a wide range of testing practices for both products and applications within a diverse range of industries. He has presented on agile testing and articles – including a chapter in the ThoughtWorks Anthology Book. He has been involved in solution delivery, coaching and mentoring teams with a focus on testing, agile processes and how the two compliment each other.

The Changing Role of a Tester

IT development is getting faster business now expects applications in months not years. Testing has to keep up and can not afford to be seen as a bottleneck. Traditional ways of testing no longer work with the rapid software development methodologies being used; there is no time to do big up-front test case design or multiple cycle test executions. Testers have to become more agile and embrace change while still providing quality information about the application being developed.

Testing on an agile project is different to a waterfall project; it is not just about doing waterfall in smaller iterations. Nor is it focused on finding as many defects as possible, instead the goal is to work as a team delivering quality working software that satisfies customer need. There are some new skills that testers will need to learn, but they do not need to throw away everything they already know. Changing the mindset about how, when and why of testing will help a tester adapt their existing skills to become an invaluable resource on any agile team.

When: 5:30pm, Thursday 13th May
Where: See the meetup site

After the discussion, we will go locally for drinks, food and networking for those interested, location TBA

Please register at the Brisbane Chapter meetup.com site: Brisbane Chapter – First Meeting

Look forward to seeing you there!

Update 19 July 2010:

Link to a great write up by Craig Smith is here, plus slides are here (pdf).

Watir, Selenium & WebDriver

Please also see my new ‘Watir-WebDriver: a detailed introduction‘ post.

Background

Of all the open source automated web testing tools available, Watir and Selenium have been the two most popular ones. Traditionally there has been advantages and disadvantages of each. Selenium’s most useful features have been its support for multiple programming languages, and is support for testing a wide variety of browsers. This is because Selenium uses JavaScript to interact with the browser, and all modern browsers support JavaScript.

Watir

Watir was originally designed to support Internet Explorer only, and was originally designed as a ruby library, meaning you had to use ruby. Watir’s most useful feature was, and is, its neat API, most likely that way because it was designed by testers. Also Watir has been traditionally had more functionality than Selenium, because it was designed to interact directly with IE instead of using JavaScript, as there are limits to what you can do with JavaScript, ie. you can’t script uploading a file.

The limitations of Watir have been addressed over time, for example, by creating new versions that support other browsers (FireWatir, SafariWatir and ChromeWatir for example), but the task of porting Watir to a new browser isn’t an easy one, and the task of keeping every port of Watir in sync, with the same API, is difficult. Because of Watir’s power, there has been lots of interest in it as a tool, particularly from developers who use other languages. Understandably, they have wanted to use the language they are accustomed to, and use for production code, and hence numerous language ports of Watir have also been created: Watij and WatiN to name two.

Selenium & WebDriver

Selenium has had its challenges also. Whilst it has traditionally been flexible in language choice, it has technical limitations because of its JavaScript architecture. It also hasn’t offered a way to control a headless browser. Late last year WebDriver was announced. WebDriver is a common browser automation tool that uses what ever is the most appropriate mechanism to control a browser, but with a common API. WebDriver supports not only real browsers (IE, Chrome & Firefox) but also headless ones (using HtmlUnit). Watir uses have had access to a headless browser by using Celerity. WebDriver was quickly merged with Selenium, to become Selenium 2.0.

Watir & WebDriver

So, what does WebDriver mean to Watir? Some people in the Watir project see an opportunity to leverage the effort put into WebDriver, but to continue to offer the clean neat API that Watir does. Jari Bakken has released an early version of Watir-WebDriver, essentially Watir’s ruby API utilising the WebDriver engine. By using WebDriver, Watir can support any browser that WebDriver does, including a headless HtmlUnit browser, without using Celerity.

What does this mean for the future of Watir?

If you’re a Watir user, it doesn’t really make that much difference. If you think of automated web testing as a car, Watir is the steering wheel and dashboard, which interact with the engine. Allowing Watir users to use WebDriver is like providing an additional engine choice, but keeping the steering wheel and dash the same.

Ultimately, I think that Watir will remain a very popular automated web testing tool, one that has been designed by testers for testers. I can see the usage of WatiN and Watij reducing as more developers move to Selenium 2.0/WebDriver which will offer the same functionality as Watir using a different API and multiple programming languages. If WebDriver can focus on the detail of controlling browsers, ultimately Watir will be a better tool as more effort can be spent on improving the Watir API, upgrading the steering wheel and dash, so to speak.

Why I do automated testing

I often get asked the question: “so why do you do automated testing?” both in job interviews and from people I meet. My answer is always pretty much the same; being that it provides me with exposure to a variety of business domains and technologies, and allows me to write code.

Let me expand a bit further on this. I like writing code but I’ve chosen not to be a software engineer. The reason is that I have met lots of different software engineers that have responsibility for one area of a system. They become the expert in this area of the system, hence when you find a problem in that area, you know who to speak to. Often time these people don’t really know much outside their area of expertise in the system, so when the problem is outside their own area, you need to speak with someone else. There’s nothing wrong with this, but it just isn’t me.

I like to know, or attempt to ‘work out’, how the system functions as a whole. I usually achieve this on a project I join by writing an automated build verification test, or ‘smoke’ test, that verifies the system functions as a whole. I love writing these kinds of tests because you need to know how the entire thing works, not just the isolated components. Software testing is one of the few areas where you get this broad level of exposure, and this keeps things interesting.

Another added bonus of doing software testing, is that your skills are pretty transferable, both across business domains and different technologies. For example, so far in my career I have tested systems that:

  • manage job advertisements, job seeker matches and interviews;
  • process immigration and visa applications;
  • manage disability support and funding;
  • produce electronic school report cards;
  • optimise large scale open cut mines;
  • monitor health of heavy machinery; and
  • manage large funds for people’s retirements.

These systems were developed in various technologies including:

  • Java;
  • Microsoft .NET;
  • Cobol;
  • Powerbuilder;
  • Web;
  • Active X;
  • Terminal Emulators;
  • SOAP web services and;
  • CICS.

The point I am trying to make is that if I decided to be a Java Software Engineer when I finished university, because that’s why I studied, I wouldn’t have had exposure to so many technologies and business areas that I have had as an automated software tester.

Sure, I could become a business analyst, because they work across business domains and technologies, but business analysts don’t code, and I do like writing code, that’s why automated testing is perfect.

The final benefit I see in automated testing is that you can apply your skills in particular testing tools to other tools fairly easily, as the concepts are the same. The biggest issue I see with people doing automated testing isn’t picking up a new tool, but rather developing a maintainable and easy to use test framework. Once you have developed a few frameworks for various business domains and technologies, it is easy to apply the concepts to other testing tools.

So that’s why I do automated testing, and I proud of it.

Celerity: first impressions

Last week at the Test Automation Workshop (TAW) here in Australia, the topic of being able to run automated tests ‘headless’ came up, and I mentioned the Celerity project: a headless Watir port. I decided to have a play with Celerity tonight to see how easy it is to get up and running, and also look at it as a way to quickly run Watir scripts.

Installation

The installation was fairly straightforward. You need a Java 6 JDK, as well as the JRuby binaries. You’ll need to update two environment variables, namely add JRuby to your path, and update your JAVA_HOME variable. Installing Celerity involves a JRuby gem: "jruby -S gem install celerity". You can also install it from github, but I don’t know the difference between the two.

Availability of Gems

Ruby gems are supported in JRuby, but only if they don’t use C libraries. This means that Watir won’t work, nor will Roo. I use Roo to define test cases in spreadsheets, so it means I can’t do a comparison of execution times at the moment, at least until I get these test cases out of spreadsheets.

Using Celerity as a simple load testing tool instead

At TAW, Kelvin Ross thought Celerity sounded promising as a load testing tool, considering it was headless and lightweight. I thought I would give a simple google search script a try, running in both Celerity, and Watir, with 30 concurrent users.

Running under Watir

CPU peaked at 100% for the entire run, and each page varied considerable but took on average 10 seconds to load.

Running under Celerity

CPU use was normal, and each page took just over 1 second to load (minimal variance).

Conclusion

Celerity seems a promising way to execute basic load tests using a headless browser. The benefit of Celerity is support for javascript execution in the browser, the downside at the moment is lack of support for some ruby gems. If you could run Watir under JRuby, I could have used a single script.

Scripts Used
Celerity Script

require 'thread'
require "rubygems"
require "celerity"

def test_google
  browser = Celerity::Browser.new
  browser.goto('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}

Watir Script

require 'thread'
require 'watir'
require 'watir/ie'

def test_google
  browser = Watir::IE.start('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}