When I am creating Watir tests, I write ruby methods to define user tasks, for example, adding a book to a cart becomes def add_book. I then group these ruby methods into ruby modules divided logically by the area of the application I am writing tests for. For example, I would have a ‘Customer’ module and an ‘Admin’ module for the Depot app. The benefit of using modules is you can avoid namespace conflicts as essentially each method is defined by its module’s prefix. This means that you can happily have def Customer.log_on and def Admin.log_on without any conflict or confusion.

As I have mentioned before, I like defining tests outside my code. These tests ultimately need to execute an associatted ruby method (stored in a module) by passing some data in (and getting an outcome and some output back). One way of calling these tests defined external to our code is to have a massive case statement that determines what calls what. This isn’t ideal as it is a maintenance burden, and really it isn’t needed.

In ruby it’s straightforward to dynamically load ruby modules, and then dynamically call individual methods.

require 'temp'

module_name = "Temp"
method_name = "hello_world"

required_module = Kernel.const_get(module_name)
required_method = required_module.method(method_name)
required_method.call('Alister')

This is all well and good if Temp.helloworld() exists, but if it doesn’t, our code throws exceptions:


`const_get': uninitialized constant Kernel::Temp (NameError)

or

`method': undefined method `hello_world' for class `Module' (NameError)


One way to avoid these exceptions is to wrap the code with a rescue clause, but I realised there are some easy ways to check if both modules and methods exist before loading them.

require 'temp'

module_name = "Temp"
method_name = "hello_world"

if Object.const_defined?(module_name)
  required_module = Kernel.const_get(module_name)
  if required_module.respond_to?(method_name) then
    required_method = required_module.method(method_name)
    required_method.call('Alister')
  else
    puts "Invalid method '#{method_name}' for module '#{module_name}'"
  end
else
 puts "Invalid module '#{module_name}'"
end

This ensures that the code continues to execute if the module or method name is specified incorrectly, which is sometimes the case if its specified in a spreadsheet, and especially if someone else has designed the spreadsheet.

Once we are happy about dynamically finding methods in modules, the next step is to make sure that each method is called with the correct number of parameters. This property of a method is called the arity.

The great thing about ruby and arity is that you simply determine the number of parameters and then pass in a correct sized array, using a *, and the receiving method will automatically unpack the array into the parameters specified.

puts required_method.arity()
required_method.call(*parameters)

The flexibility that ruby offers is amazing. I have tried to accomplish this same concept in VBScript but I couldn’t work out how. That’s why I am glad Watir uses ruby, it ultimately means my automated test framework is more efficient and maintainable.

One common challenge I see over and over again is people figuring out how to design a logical and maintainable automated testing framework. I have designed quite a few frameworks for various projects, but one thing that has consistently been a win for me is purposely separating test case and test execution design.

It’s therefore logical that the design of my Watir framework deliberately separates test case design and test execution design so that:

• test case design is done visually in spreadsheets; and
• test execution design is done in ruby methods, because code is the most efficient and maintainable way.

Since I last published details about my framework on this blog, I have started doing assertions using the Test::Unit ruby library. The reasons I chose Test::Unit are:

• it is easy to ‘mix-in’ Test::Unit assertions into modules of ruby code using include Test::Unit::Assertions;
• it is included with ruby;
• ruby scripts with Test::Unit::TestCase are instantly executable, in my case, from SciTE;
• its assertions are easy to understand and use.

I have also made some other improvements to my framework code, including:

• the ability to specify browser types, and spreadsheet sources, as command line arguments (with defaults);
• logging test output to a file;
• no longer attaching to an open browser, the same browser instance is used completely for all tests (and elegantly closed at the end).

The main design has been kept the same, in that a spreadsheet (either excel, openoffice or Google Docs) contains tests grouped by functional area, which call a method in a particular module.

The great thing about my framework is that adding a new test is a matter of designing the test case, and then writing the ruby method: as the methods are called dynamically from the spreadsheet, no extra glue is needed!

Enough talk, here’s the code. The Google spreadsheet is here. You can find a .zip file of all the required files to run it here. It runs on the depot app, which you get here. You will need two gems: Watir (oh duh), and Roo.

Test Driver tc_main.rb

$:.unshift File.join(File.dirname(__FILE__), ".", "lib")
require 'watir'
require 'roo'
require 'test/unit'
require 'customer'
require 'admin'
$stdout = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)
$stderr = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)

class TC_WatirMelon < Test::Unit::TestCase
  @@colmap = {:module_name=>0, :method_name=>1, :comments=>2, :exp_outcome=>3, :exp_error=>4, :first_param=>5}
  @@ss_format = ARGV[0]
  @@specified_browser = ARGV[1]

  def setup
    puts "[Starting at #{Time.now}]\n"
    case @@ss_format
      when "excel"
        @ss = Excel.new("watirmelon.xls")
      when "wiki"
        @ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
      when "gdocs"
        @ss = Google.new("0AtL3mPY2rEqmdEY3XzRqUlZKSmM5Z3EtM21UdFdqb1E")
      else
        @ss = Openoffice.new("watirmelon.ods")
      end
    @ss.default_sheet = @ss.sheets.first
    case @@specified_browser
      when "firefox"
        Watir::Browser.default = 'firefox'
        @browser = Watir::Browser.new
      else
        Watir::Browser.default = 'ie'
        @browser = Watir::Browser.new
        @browser.speed = :zippy
        @browser.visible = true
      end
  end

  def test_run_sheet()
    @ss.first_row.upto(@ss.last_row) do |row|
      #Read row into array
      line = Array.new
      @ss.first_column.upto(@ss.last_column) do |column|
        line << @ss.cell(row, column).to_s.strip
      end

      module_name = line[@@colmap[:module_name]]
      if module_name != "Function" then #if not a header
        method_name = line[@@colmap[:method_name]].downcase.gsub(' ','_') #automatically determine ruby method name based upon data sheet
        exp_outcome = line[@@colmap[:exp_outcome]]
        exp_error = line[@@colmap[:exp_error]]
        first_param = @@colmap[:first_param]
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0 # arity is negative when there is a 'catch all'
        arity = arity-1 # Ignore the first browser parameter
        unless arity == 0
          parameters = line[first_param..first_param+(arity-1)]
        else
          parameters = []
        end
        begin
          act_outcome, act_output = required_method.call(@browser, *parameters)
        rescue Test::Unit::AssertionFailedError => e
          self.send(:add_failure, e.message, e.backtrace)
          act_outcome = false
          act_output = e.message
        end
        if (exp_outcome == 'Success') and act_outcome then
          assert(true, "Expected outcome and actual outcome are the same")
          result = 'PASS'
        elsif (exp_outcome == 'Error') and (not act_outcome) and (exp_error.strip! == act_output.strip!)
          assert(true, "Expected outcome and actual outcome are the same, and error messages match")
          result = 'PASS'
        else
          result = 'FAIL'
          begin
            assert(false,"Row: #{row}: Expected outcome and actual outcome for #{method_name} for #{module_name} do not match, or error messages do not match.")
          rescue Test::Unit::AssertionFailedError => e
            self.send(:add_failure, e.message, e.backtrace)
          end
        end
        puts "###########################################"
        puts "[Running: #{module_name}.#{method_name}]"
        puts "[Expected Outcome: #{exp_outcome}]"
        puts "[Expected Error: #{exp_error}]"
        puts "[Actual Outcome: Success]" if act_outcome
        puts "[Actual Outcome: Error]" if not act_outcome
        puts "[Actual Output: #{act_output}]"
        puts "[RESULT: #{result}]"
        puts "###########################################"
        end
      end
  end

  def teardown
    @browser.close
    puts "[Finishing at #{Time.now}]\n\n"
  end

end

Customer Module customer.rb

require 'test/unit'
include Test::Unit::Assertions

module Customer

  TITLE = 'Pragprog Books Online Store'
  URL = 'http://localhost:3000/store/'

  # Description:: Adds a book named 'book_title' to cart
  def Customer.add_book(browser, book_title)
    browser.goto(URL)
    # Check if title is already in cart - so we can check it was added correctly
    browser.link(:text,'Show my cart').click
    prev_cart_count = 0
    prev_cart_total = 0.00
    if not browser.div(:text,'Your cart is currently empty').exist? then
     # We have a non-empty cart
      for row in browser.table(:index,1)
        if row[2].text == book_title then
          prev_cart_count = row[1].text.to_i
          break
        end
      end
      prev_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f #remove $ sign
      browser.link(:text, 'Continue shopping').click
    end

    found = false
    book_price = 0.00
    1.upto(browser.divs.length) do |index|
      if (browser.div(:index,index).attribute_value('className') == 'catalogentry') and (browser.div(:index,index).h3(:text,book_title).exists?) then
        book_price = browser.div(:index,index).span(:class, 'catalogprice').text[1..-1].to_f #remove $ sign
        browser.div(:index,index).link(:class,'addtocart').click
        found = true
        break
      end
    end
    if not found then
      return false,'Could not locate title in store'
    end

    new_cart_count = 0
    for row in browser.table(:index,1)
      if row[2].text == book_title then
        new_cart_count = row[1].text.to_i
        break
      end
    end
    new_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f # remove $ sign
    assert_equal(new_cart_count,(prev_cart_count+1), "Ensure that new quantity is now one greater than previously")
    assert_equal(new_cart_total,(prev_cart_total + book_price), "Ensure that new cart total is old cart total plus book price")
    browser.link(:text, 'Continue shopping').click
    return true,new_cart_total
  end

  def Customer.check_out(browser, customerName, customerEmail, customerAddress, customerPaymentMethod)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    browser.link(:text,"Checkout").click
    browser.text_field(:id, 'order_name').set(customerName)
    browser.text_field(:id, 'order_email').set(customerEmail)
    browser.text_field(:id, 'order_address').set(customerAddress)
    begin
      browser.select_list(:id, 'order_pay_type').select(customerPaymentMethod)
    rescue Watir::Exception::NoValueFoundException
      flunk('Could not locate customer payment method in drop down list: '+customerPaymentMethod)
    end
    browser.button(:name, 'commit').click
    if browser.div(:id,'errorExplanation').exist? then
      error = ''
      1.upto(browser.div(:id,'errorExplanation').lis.length) do |index|
        error << (browser.div(:id,'errorExplanation').li(:index,index).text + ",")
      end
      browser.link(:text,'Continue shopping').click
      return false, error
    end
    assert_equal(browser.div(:id,'notice').text, 'Thank you for your order.',"Thank you for your order should appear.")
    return true,''
  end

  def Customer.empty_cart(browser)
    browser.goto(URL)
    browser.link(:text,"Show my cart").click
    if browser.div(:text,"Your cart is currently empty").exist? then
      assert('Cart was never empty')
    else
      browser.link(:text,'Empty cart').click
      assert_equal(browser.div(:id, 'notice').text,'Your cart is now empty')
    end
    return true,''
  end

  def Customer.check_cart_total(browser, exp_total)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    act_total = browser.cell(:id, 'totalcell').text[1..-1].to_f
    assert_equal(act_total,exp_total.to_f,"Check that cart total is as expected.")
    return true,act_total
  end
end

Admin Module admin.rb

require 'test/unit'
include Test::Unit::Assertions

module Admin
  TITLE = 'ADMINISTER Pragprog Books Online Store'
  URL = 'http://localhost:3000/admin/'

  def Admin.log_on(browser, username, password)
    browser.goto(URL)
    if browser.link(:text,'Log out').exist? then #if already logged in
      browser.link(:text,'Log out').click
    end
    browser.text_field(:id, 'user_name').set username
    browser.text_field(:id, 'user_password').set password
    browser.button(:value, ' LOGIN ').click
    if browser.div(:id, 'notice').exist? then
      return false,browser.div(:id, 'notice').text
    else
      return true,''
    end
  end

  def Admin.ship_items(browser, name)
    browser.goto(URL)
    browser.link(:text, 'Shipping').click
    num_orders = 0
    index = 0
    browser.form(:action,'/admin/ship').divs.each do |div|
      if div.class_name == "olname"
        index+=1
        if div.text == name then
          browser.form(:action,'/admin/ship').checkbox(:index, index).set
          num_orders+=1
        end
      end
    end

    browser.button(:value, ' SHIP CHECKED ITEMS ').click

    if num_orders == 1 then
      assert_equal(browser.div(:id,"notice").text, "One order marked as shipped","Correct notice")
    elsif num_orders > 1 then
      assert_equal(browser.div(:id,"notice").text, "#{num_orders} orders marked as shipped","Correct notice")
    end
    return true, num_orders.to_s
  end

end

Last week at the Test Automation Workshop (TAW) here in Australia, the topic of being able to run automated tests ‘headless’ came up, and I mentioned the Celerity project: a headless Watir port. I decided to have a play with Celerity tonight to see how easy it is to get up and running, and also look at it as a way to quickly run Watir scripts.

Installation

The installation was fairly straightforward. You need a Java 6 JDK, as well as the JRuby binaries. You’ll need to update two environment variables, namely add JRuby to your path, and update your JAVA_HOME variable. Installing Celerity involves a JRuby gem: "jruby -S gem install celerity". You can also install it from github, but I don’t know the difference between the two.

Availability of Gems

Ruby gems are supported in JRuby, but only if they don’t use C libraries. This means that Watir won’t work, nor will Roo. I use Roo to define test cases in spreadsheets, so it means I can’t do a comparison of execution times at the moment, at least until I get these test cases out of spreadsheets.

Using Celerity as a simple load testing tool instead

At TAW, Kelvin Ross thought Celerity sounded promising as a load testing tool, considering it was headless and lightweight. I thought I would give a simple google search script a try, running in both Celerity, and Watir, with 30 concurrent users.

Running under Watir

CPU peaked at 100% for the entire run, and each page varied considerable but took on average 10 seconds to load.

Running under Celerity

CPU use was normal, and each page took just over 1 second to load (minimal variance).

Conclusion

Celerity seems a promising way to execute basic load tests using a headless browser. The benefit of Celerity is support for javascript execution in the browser, the downside at the moment is lack of support for some ruby gems. If you could run Watir under JRuby, I could have used a single script.

Scripts Used
Celerity Script

require 'thread'
require "rubygems"
require "celerity"

def test_google
  browser = Celerity::Browser.new
  browser.goto('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}

Watir Script

require 'thread'
require 'watir'
require 'watir/ie'

def test_google
  browser = Watir::IE.start('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}

Curiosity got the better of me yesterday when writing about using SciTE, so I did a quick twitter survey to see what other people actually use. I got 13 responses within hours, and they show that there really is no common IDE for Watir.

The complete list of 13 responses:

• Aptana: 1
• Arachno: 1
• eclipse-galileo: 1
• NetBeans: 1
• notepad++: 2
• SCiTE: 4
• TextMate: 1
• vim: 2

I am somewhat embarrassed to admit that I still use SciTE to edit my ruby files for Watir, considering there are a lot more sophisticated IDEs out there that you can use (Netbeans is one).

When I am using a new version of SciTE I notice it seems to get rid of the options to check ruby syntax (often triggered by Control-1). This can be quickly fixed in the ruby.properties file (under ‘Options’) by ensuring the following properties appear after ‘if PLAT_WIN’:

if PLAT_WIN
command.go.*.rb=ruby $(FileNameExt)
command.go.subsystem.*.rb=1
command.go.*.rbw=rubyw $(FileNameExt)
command.go.subsystem.*.rbw=1
command.help.*.rb=$(CurrentWord)!c:\ruby\doc\ProgrammingRuby.chm
command.help.subsystem.*.rb=4
command.help.*.rbw=$(CurrentWord)!c:\ruby\doc\ProgrammingRuby.chm
command.help.subsystem.*.rbw=4

command.name.1.*.rb=Check Syntax
command.1.*.rb=ruby -cw $(FileNameExt)
command.name.1.*.rbw=Check Syntax
command.1.*.rbw=rubyw -cw $(FileNameExt)

command.name.2.*.rb=Code Profiler
command.2.*.rb=ruby -r profile $(FileNameExt)
command.name.2.*.rbw=Code Profiler
command.2.*.rbw=rubyw -r profile $(FileNameExt)

command.name.3.*.rb=Run irb
command.3.*.rb=irb.bat
command.subsystem.3.*.rb=2
command.name.3.*.rbw=Run irb
command.3.*.rbw=irb.bat
command.subsystem.3.*.rbw=2

command.name.4.*.rb=Debug
command.4.*.rb=ruby -d -r debug $(FileNameExt)
command.subsystem.4.*.rb=2
command.name.4.*.rbw=Debug
command.4.*.rbw= ruby -d -r debug $(FileNameExt)
command.subsystem.4.*.rbw=2

Make sure the help command points to your ruby install. That way you also get nice F1 help on any highlighted ruby syntax.

Important update: 22 August 2009:

Make sure the line 3 above reads

command.go.subsystem.*.rb=1

not ‘=2′, otherwise stdout won’t be shown back in SciTE.

Why don’t testers version control their tests?

I am still surprised at how many organizations don’t version control their automated test scripts. I put it down to the following reasons:

• Developers may use expensive version control tools, but sometimes there aren’t enough licenses for testers;
• People don’t realize there are free version control tools available;
• Setting up version control might be considered too difficult for the test team;
• Some people believe you need to own a version control server to version your test scripts; or
• Any combination of the above.

In reality:

• If the software developers’ version control system is available for testers great, but if not, test scripts can be versioned separately;
• There are many different free version control tools available. TortoiseSVN, which uses the Subversion (SVN) protocol, is very popular and very easy to use;
• Setting up a new SVN repository using TortoiseSVN only takes a few minutes; and
• You can set up a SVN repository on a shared network drive, so you don’t need a server (but a server is cool).

How to quickly set up a new SVN respository on a shared network drive (using Windows)

If you haven’t version controlled your test scripts yet, here’s how to do so.

1. Download and install TortoiseSVN from http://tortoisesvn.tigris.org/ (it’s about 19MB, and requires a reboot: bummer )
2. Find a location on a shared network drive where you can store your SVN repository. For example, it could be Q:\SVN  Repositories , create a new directory for your repository (eg. Q:\SVN  Repositories\WatirMelon\ and right click within this new directory in Windows Explorer, and choose TortoiseSVN, and then ‘create repository here’. The path to your new directory will be your SVN repository path.
3. The repository should be created in a matter of seconds, and filled with some directories and files. These files/directories should never be touched, under any circumstances.
4. Now you need to create a local repository and check out the new repository (which will be blank initially). Create a directory on your local drive for your repository, for example C:\watirmelon, and right click within this directory and choose ‘SVN Checkout’.
5. You will need to specify the location of your repository you created in Step 2, but importantly you will need to add file:/// to the front, and change the backslashes into forward slashes.
6. Once you click OK you have a repository (albeit blank) checked out. You would then simply add all your automated test scripts, then do an ‘SVN Add’, and ‘SVN commit’. If you want to use your automated tests on another machine, you simply checkout the repository following steps 4 & 5 above.

Conclusion

So there it is. Now there’s really no excuse not to version control your automated tests, considering it’s free, quick, easy and doesn’t require a server. So go and do it now (if you haven’t already).

With the launch of the new watir.com site, I thought it would be an appropriate time to refresh the Watir logo to be included on the site and all other Watir branded information.

Current Watir Logo by Jacinda Scott

The current logo (above) was designed some time ago by Jacinda Scott, and has served Watir well, but it would be great if someone in the Watir community could propose a refresh of sorts.

One of the issues with the current logo is that we don’t have a high- resolution (or .svg) version, so it is hard to use the logo to full effect.

There has been positive feedback about the current logo, it is round (like the web browser icons, firefox, ie, opera, etc.) and it is blue and ‘water’ like.

I am proposing a logo redesign/refresh competition where you can take the current logo, and propose something fresh. It would be great if it keeps the browser-ish style with some sort of water/blue theme.

Once some designs have been submitted (over the next week – up until say Sunday 9th August), I will compile these and have the Watir community vote on what they think is the best one. The winner shall receive all the praise and glory of having their logo redesign on the watir.com site (with credits of course).

Please email your designs (preferably in .png and/or .svg) to alister dot scott at gmail by 9th August.

I have spent a bit of time over the last few days setting up Watir.com, hosted here on WordPress.

We were originally aiming to host our own version of Confluence and JIRA and use Confluence to serve the Watir.com homepage, but this ended up being a lot more complicated and expensive than originally planned.

The great thing about WordPress is, although it was originally a blogging platform, its functionality also works as a very neat CMS. Whilst wordpress.com has some limitations over wordpress.org, we can live with these limitations for now as we have a free (as in beer) hosted site that the world can see.

Check it out.

I will be presenting at a SIGIST (Special Interest Group in Software Testing) on Watir on May 26 @ the Hilton in Brisbane. Everyone is welcome to attend.

More details here.

• Watif if you don’t want to learn a new language just so you can test your web app?
• Watif you want to kick it old-skool with punch cards?
• Watif you want a fully supported automated test solution running SAS and with in built notifications of results?

Introducing Watif

Web Application Testing in FORTRAN:
web application testing that punches!

Watif is simple

Tests are created using simple, easy to use coding forms, easily followed by business analysts and end users. No more expensive test automation engineers!

Watif Coding Form

Watif is automated

Code is created automatically on punch cards using state of the art FORTRAN compilers, saving you valuable compilation time.

Sample Watif code (automatically generated)

Watif is fully supported

Watirfort is a new company of 1,000 monkeys, available 24/7 worldwide to commercially support Watif and make it a success in your organization.

Fully supported by Watirfort

Watif is SAS

All your tests are run by Watirfort using state of the art punch card processing systems, just like salesforce.com.

State of the art Watirfort labs

In built notification systems

When you purchase Watif services from Watirfort, you can specifiy how many homing pigeons you would like to lease. These homing pigeons are dedicated to delivering your printed Watif output directly to you! Rapid feedback!

Watirfort Homing Pigeons

Planned Additional Browser Support

While Watif 1.0 only initially supports the WorldWideWeb browser, alternative browsers including Netscape Navigator 1.0 are planned for Watif 2.0.

WorldWideWeb fully supported!

Quotes

“I wanted to run around my office punching my hands in the air.”   -   Bec Ferguson

Watif is available for immediate release.

Order now and get a bonus FORTRAN book.