One common challenge I see over and over again is people figuring out how to design a logical and maintainable automated testing framework. I have designed quite a few frameworks for various projects, but one thing that has consistently been a win for me is purposely separating test case and test execution design.

It’s therefore logical that the design of my Watir framework deliberately separates test case design and test execution design so that:

• test case design is done visually in spreadsheets; and
• test execution design is done in ruby methods, because code is the most efficient and maintainable way.

Since I last published details about my framework on this blog, I have started doing assertions using the Test::Unit ruby library. The reasons I chose Test::Unit are:

• it is easy to ‘mix-in’ Test::Unit assertions into modules of ruby code using include Test::Unit::Assertions;
• it is included with ruby;
• ruby scripts with Test::Unit::TestCase are instantly executable, in my case, from SciTE;
• its assertions are easy to understand and use.

I have also made some other improvements to my framework code, including:

• the ability to specify browser types, and spreadsheet sources, as command line arguments (with defaults);
• logging test output to a file;
• no longer attaching to an open browser, the same browser instance is used completely for all tests (and elegantly closed at the end).

The main design has been kept the same, in that a spreadsheet (either excel, openoffice or Google Docs) contains tests grouped by functional area, which call a method in a particular module.

The great thing about my framework is that adding a new test is a matter of designing the test case, and then writing the ruby method: as the methods are called dynamically from the spreadsheet, no extra glue is needed!

Enough talk, here’s the code. The Google spreadsheet is here. You can find a .zip file of all the required files to run it here. It runs on the depot app, which you get here. You will need two gems: Watir (oh duh), and Roo.

Test Driver tc_main.rb

$:.unshift File.join(File.dirname(__FILE__), ".", "lib")
require 'watir'
require 'roo'
require 'test/unit'
require 'customer'
require 'admin'
$stdout = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)
$stderr = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)

class TC_WatirMelon < Test::Unit::TestCase
  @@colmap = {:module_name=>0, :method_name=>1, :comments=>2, :exp_outcome=>3, :exp_error=>4, :first_param=>5}
  @@ss_format = ARGV[0]
  @@specified_browser = ARGV[1]

  def setup
    puts "[Starting at #{Time.now}]\n"
    case @@ss_format
      when "excel"
        @ss = Excel.new("watirmelon.xls")
      when "wiki"
        @ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
      when "gdocs"
        @ss = Google.new("0AtL3mPY2rEqmdEY3XzRqUlZKSmM5Z3EtM21UdFdqb1E")
      else
        @ss = Openoffice.new("watirmelon.ods")
      end
    @ss.default_sheet = @ss.sheets.first
    case @@specified_browser
      when "firefox"
        Watir::Browser.default = 'firefox'
        @browser = Watir::Browser.new
      else
        Watir::Browser.default = 'ie'
        @browser = Watir::Browser.new
        @browser.speed = :zippy
        @browser.visible = true
      end
  end

  def test_run_sheet()
    @ss.first_row.upto(@ss.last_row) do |row|
      #Read row into array
      line = Array.new
      @ss.first_column.upto(@ss.last_column) do |column|
        line << @ss.cell(row, column).to_s.strip
      end

      module_name = line[@@colmap[:module_name]]
      if module_name != "Function" then #if not a header
        method_name = line[@@colmap[:method_name]].downcase.gsub(' ','_') #automatically determine ruby method name based upon data sheet
        exp_outcome = line[@@colmap[:exp_outcome]]
        exp_error = line[@@colmap[:exp_error]]
        first_param = @@colmap[:first_param]
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0 # arity is negative when there is a 'catch all'
        arity = arity-1 # Ignore the first browser parameter
        unless arity == 0
          parameters = line[first_param..first_param+(arity-1)]
        else
          parameters = []
        end
        begin
          act_outcome, act_output = required_method.call(@browser, *parameters)
        rescue Test::Unit::AssertionFailedError => e
          self.send(:add_failure, e.message, e.backtrace)
          act_outcome = false
          act_output = e.message
        end
        if (exp_outcome == 'Success') and act_outcome then
          assert(true, "Expected outcome and actual outcome are the same")
          result = 'PASS'
        elsif (exp_outcome == 'Error') and (not act_outcome) and (exp_error.strip! == act_output.strip!)
          assert(true, "Expected outcome and actual outcome are the same, and error messages match")
          result = 'PASS'
        else
          result = 'FAIL'
          begin
            assert(false,"Row: #{row}: Expected outcome and actual outcome for #{method_name} for #{module_name} do not match, or error messages do not match.")
          rescue Test::Unit::AssertionFailedError => e
            self.send(:add_failure, e.message, e.backtrace)
          end
        end
        puts "###########################################"
        puts "[Running: #{module_name}.#{method_name}]"
        puts "[Expected Outcome: #{exp_outcome}]"
        puts "[Expected Error: #{exp_error}]"
        puts "[Actual Outcome: Success]" if act_outcome
        puts "[Actual Outcome: Error]" if not act_outcome
        puts "[Actual Output: #{act_output}]"
        puts "[RESULT: #{result}]"
        puts "###########################################"
        end
      end
  end

  def teardown
    @browser.close
    puts "[Finishing at #{Time.now}]\n\n"
  end

end

Customer Module customer.rb

require 'test/unit'
include Test::Unit::Assertions

module Customer

  TITLE = 'Pragprog Books Online Store'
  URL = 'http://localhost:3000/store/'

  # Description:: Adds a book named 'book_title' to cart
  def Customer.add_book(browser, book_title)
    browser.goto(URL)
    # Check if title is already in cart - so we can check it was added correctly
    browser.link(:text,'Show my cart').click
    prev_cart_count = 0
    prev_cart_total = 0.00
    if not browser.div(:text,'Your cart is currently empty').exist? then
     # We have a non-empty cart
      for row in browser.table(:index,1)
        if row[2].text == book_title then
          prev_cart_count = row[1].text.to_i
          break
        end
      end
      prev_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f #remove $ sign
      browser.link(:text, 'Continue shopping').click
    end

    found = false
    book_price = 0.00
    1.upto(browser.divs.length) do |index|
      if (browser.div(:index,index).attribute_value('className') == 'catalogentry') and (browser.div(:index,index).h3(:text,book_title).exists?) then
        book_price = browser.div(:index,index).span(:class, 'catalogprice').text[1..-1].to_f #remove $ sign
        browser.div(:index,index).link(:class,'addtocart').click
        found = true
        break
      end
    end
    if not found then
      return false,'Could not locate title in store'
    end

    new_cart_count = 0
    for row in browser.table(:index,1)
      if row[2].text == book_title then
        new_cart_count = row[1].text.to_i
        break
      end
    end
    new_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f # remove $ sign
    assert_equal(new_cart_count,(prev_cart_count+1), "Ensure that new quantity is now one greater than previously")
    assert_equal(new_cart_total,(prev_cart_total + book_price), "Ensure that new cart total is old cart total plus book price")
    browser.link(:text, 'Continue shopping').click
    return true,new_cart_total
  end

  def Customer.check_out(browser, customerName, customerEmail, customerAddress, customerPaymentMethod)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    browser.link(:text,"Checkout").click
    browser.text_field(:id, 'order_name').set(customerName)
    browser.text_field(:id, 'order_email').set(customerEmail)
    browser.text_field(:id, 'order_address').set(customerAddress)
    begin
      browser.select_list(:id, 'order_pay_type').select(customerPaymentMethod)
    rescue Watir::Exception::NoValueFoundException
      flunk('Could not locate customer payment method in drop down list: '+customerPaymentMethod)
    end
    browser.button(:name, 'commit').click
    if browser.div(:id,'errorExplanation').exist? then
      error = ''
      1.upto(browser.div(:id,'errorExplanation').lis.length) do |index|
        error << (browser.div(:id,'errorExplanation').li(:index,index).text + ",")
      end
      browser.link(:text,'Continue shopping').click
      return false, error
    end
    assert_equal(browser.div(:id,'notice').text, 'Thank you for your order.',"Thank you for your order should appear.")
    return true,''
  end

  def Customer.empty_cart(browser)
    browser.goto(URL)
    browser.link(:text,"Show my cart").click
    if browser.div(:text,"Your cart is currently empty").exist? then
      assert('Cart was never empty')
    else
      browser.link(:text,'Empty cart').click
      assert_equal(browser.div(:id, 'notice').text,'Your cart is now empty')
    end
    return true,''
  end

  def Customer.check_cart_total(browser, exp_total)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    act_total = browser.cell(:id, 'totalcell').text[1..-1].to_f
    assert_equal(act_total,exp_total.to_f,"Check that cart total is as expected.")
    return true,act_total
  end
end

Admin Module admin.rb

require 'test/unit'
include Test::Unit::Assertions

module Admin
  TITLE = 'ADMINISTER Pragprog Books Online Store'
  URL = 'http://localhost:3000/admin/'

  def Admin.log_on(browser, username, password)
    browser.goto(URL)
    if browser.link(:text,'Log out').exist? then #if already logged in
      browser.link(:text,'Log out').click
    end
    browser.text_field(:id, 'user_name').set username
    browser.text_field(:id, 'user_password').set password
    browser.button(:value, ' LOGIN ').click
    if browser.div(:id, 'notice').exist? then
      return false,browser.div(:id, 'notice').text
    else
      return true,''
    end
  end

  def Admin.ship_items(browser, name)
    browser.goto(URL)
    browser.link(:text, 'Shipping').click
    num_orders = 0
    index = 0
    browser.form(:action,'/admin/ship').divs.each do |div|
      if div.class_name == "olname"
        index+=1
        if div.text == name then
          browser.form(:action,'/admin/ship').checkbox(:index, index).set
          num_orders+=1
        end
      end
    end

    browser.button(:value, ' SHIP CHECKED ITEMS ').click

    if num_orders == 1 then
      assert_equal(browser.div(:id,"notice").text, "One order marked as shipped","Correct notice")
    elsif num_orders > 1 then
      assert_equal(browser.div(:id,"notice").text, "#{num_orders} orders marked as shipped","Correct notice")
    end
    return true, num_orders.to_s
  end

end