Category Archives: Business Driven

CukeSalad: not so yummy yummy

Posted on
Reply

I’ve been playing around with CukeSalad: Cucumber specs / tests without step-definitions:

Cucumber, washed and ready to eat for Friction-free ATDD/BDD

In creating an example of my simple WatirMelonCucumber framework using CukeSalad, here’s some of the things I’ve found. Please note: these are my personal observations and are not direct criticisms of the huge effort that has gone into making this tool.

It’s not really no step definitions: they’re just called different things. Instead of writing steps, you’re writing roles and tasks, both of which are mandatory, so I found I actually wrote more code (in more files) than if I just wrote step definitions in the first place.

CukeSalad forces you to use predefined step syntax: which to me defeats the purpose of having features in business language. For example, as far as I can tell, Given statements are purely used for setting the roles, and therefore a business person can’t use it to define a precondition which I would normally use Given statements for. Another example is when you want to capture some data from the test, this must be on the end of a sentence following a semicolon (or comma)

When I search for: the phrase 'Watir'

I spent ages trying to work out why steps don’t match: as you still use Cucumber to run your features, Cucumber will only tell you that there’s no matching steps. This doesn’t help when there’s no steps whatsoever! At some point, I realized that Then steps follow a different syntax, and they can’t use placeholders like Whens, but must have an expected value on the end in single quotes.

Then I should see results greater than '30,000'

It’s very confusing what you’re actually asserting on a Then step: I couldn’t work out a way to capture the expected value from the feature (in the example above ’30,000′), and CukeSalad just asserts the task returns the actual expected result, which doesn’t help if you’re not testing for equality!

It’s very confusing trying to convert an existing Cucumber test suite: I wouldn’t bother. I found I had to completely rewrite my feature to comply to the syntactical requirements of CukeSalad, so I can’t this working on an existing feature set.

Cucumber world is not supported: so just do a normal include instead in env.rb.

Summary

Sadly, CukeSalad doesn’t offer me anything over using Cucumber. Whilst it sells itself on having no step definitions, I found there was greater effort in writing the roles and tasks. I don’t like the way it forces me to write sentences using somewhat awkward constraints, and how Given, When, & Then statements all behave differently. It certainly gave me a lot of WTF moments, which isn’t what you want when writing automated tests.

I’d be very interested to hear whether anyone has had ongoing success with CukeSalad.

Turnip: trying to solve Cucumber’s problems with ruby BDD

Posted on
5

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i
    end
  end

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000″ as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).

Summary

I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.

WatirMelon Spinach

Posted on
5

Hello Spinach!

“Spinach is a new awesome BDD framework that features encapsulation and modularity of your step definitions.”

Seems like a good idea, so I thought I’d give it a try, and convert my existing simple Watirmelon Cucumber tests to use Spinach instead (link to my source code). It was very easy to do, here’s some observations:

  • It’s easy to get existing Cukes running using Spinach, but I imagine if you were starting out using Spinach you’d design things a lot differently
  • Steps belong in their own class, but you can include mixins to reuse steps – the ruby way
  • Steps are in a steps directory under features – shorter than step_definitions
  • Goodbye regular expressions in step definitions, which is a bit of a shame, as you can no longer capture values from the step name
  • As you can’t have regular expressions in step names, I find myself repeating steps that are similar but slightly different, this means my example steps have gone from 41 to 57 lines of code
  • Scenario Outlines aren’t supported at the moment, but I have raised this as a feature request
  • Hooks are dramatically improved, so I found them very easy to use and understand
  • There is no cucumber world, so you do a normal include instead, and env.rb is still supported
  • It displays really cool ticks on the command line when you’re running your spins

Well done to Codegram for releasing Spinach. If anything, it creates some great innovations that I imagine may find their way into Cucumber in the future.

Creating a Watir framework using Test::Unit & Roo

Posted on
5

One common challenge I see over and over again is people figuring out how to design a logical and maintainable automated testing framework. I have designed quite a few frameworks for various projects, but one thing that has consistently been a win for me is purposely separating test case and test execution design.

It’s therefore logical that the design of my Watir framework deliberately separates test case design and test execution design so that:

  • test case design is done visually in spreadsheets; and
  • test execution design is done in ruby methods, because code is the most efficient and maintainable way.

Since I last published details about my framework on this blog, I have started doing assertions using the Test::Unit ruby library. The reasons I chose Test::Unit are:

  • it is easy to ‘mix-in’ Test::Unit assertions into modules of ruby code using include Test::Unit::Assertions;
  • it is included with ruby;
  • ruby scripts with Test::Unit::TestCase are instantly executable, in my case, from SciTE;
  • its assertions are easy to understand and use.

I have also made some other improvements to my framework code, including:

  • the ability to specify browser types, and spreadsheet sources, as command line arguments (with defaults);
  • logging test output to a file;
  • no longer attaching to an open browser, the same browser instance is used completely for all tests (and elegantly closed at the end).

The main design has been kept the same, in that a spreadsheet (either excel, openoffice or Google Docs) contains tests grouped by functional area, which call a method in a particular module.

The great thing about my framework is that adding a new test is a matter of designing the test case, and then writing the ruby method: as the methods are called dynamically from the spreadsheet, no extra glue is needed!

Enough talk, here’s the code. The Google spreadsheet is here. You can find a .zip file of all the required files to run it here. It runs on the depot app, which you get here. You will need two gems: Watir (oh duh), and Roo.

Test Driver tc_main.rb


$:.unshift File.join(File.dirname(__FILE__), ".", "lib")
require 'watir'
require 'roo'
require 'test/unit'
require 'customer'
require 'admin'
$stdout = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)
$stderr = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)

class TC_WatirMelon < Test::Unit::TestCase
  @@colmap = {:module_name=>0, :method_name=>1, :comments=>2, :exp_outcome=>3, :exp_error=>4, :first_param=>5}
  @@ss_format = ARGV[0]
  @@specified_browser = ARGV[1]

  def setup
    puts "[Starting at #{Time.now}]\n"
    case @@ss_format
      when "excel"
        @ss = Excel.new("watirmelon.xls")
      when "wiki"
        @ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
      when "gdocs"
        @ss = Google.new("0AtL3mPY2rEqmdEY3XzRqUlZKSmM5Z3EtM21UdFdqb1E")
      else
        @ss = Openoffice.new("watirmelon.ods")
      end
    @ss.default_sheet = @ss.sheets.first
    case @@specified_browser
      when "firefox"
        Watir::Browser.default = 'firefox'
        @browser = Watir::Browser.new
      else
        Watir::Browser.default = 'ie'
        @browser = Watir::Browser.new
        @browser.speed = :zippy
        @browser.visible = true
      end
  end

  def test_run_sheet()
    @ss.first_row.upto(@ss.last_row) do |row|
      #Read row into array
      line = Array.new
      @ss.first_column.upto(@ss.last_column) do |column|
        line << @ss.cell(row, column).to_s.strip
      end

      module_name = line[@@colmap[:module_name]]
      if module_name != "Function" then #if not a header
        method_name = line[@@colmap[:method_name]].downcase.gsub(' ','_') #automatically determine ruby method name based upon data sheet
        exp_outcome = line[@@colmap[:exp_outcome]]
        exp_error = line[@@colmap[:exp_error]]
        first_param = @@colmap[:first_param]
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0 # arity is negative when there is a 'catch all'
        arity = arity-1 # Ignore the first browser parameter
        unless arity == 0
          parameters = line[first_param..first_param+(arity-1)]
        else
          parameters = []
        end
        begin
          act_outcome, act_output = required_method.call(@browser, *parameters)
        rescue Test::Unit::AssertionFailedError => e
          self.send(:add_failure, e.message, e.backtrace)
          act_outcome = false
          act_output = e.message
        end
        if (exp_outcome == 'Success') and act_outcome then
          assert(true, "Expected outcome and actual outcome are the same")
          result = 'PASS'
        elsif (exp_outcome == 'Error') and (not act_outcome) and (exp_error.strip! == act_output.strip!)
          assert(true, "Expected outcome and actual outcome are the same, and error messages match")
          result = 'PASS'
        else
          result = 'FAIL'
          begin
            assert(false,"Row: #{row}: Expected outcome and actual outcome for #{method_name} for #{module_name} do not match, or error messages do not match.")
          rescue Test::Unit::AssertionFailedError => e
            self.send(:add_failure, e.message, e.backtrace)
          end
        end
        puts "###########################################"
        puts "[Running: #{module_name}.#{method_name}]"
        puts "[Expected Outcome: #{exp_outcome}]"
        puts "[Expected Error: #{exp_error}]"
        puts "[Actual Outcome: Success]" if act_outcome
        puts "[Actual Outcome: Error]" if not act_outcome
        puts "[Actual Output: #{act_output}]"
        puts "[RESULT: #{result}]"
        puts "###########################################"
        end
      end
  end

  def teardown
    @browser.close
    puts "[Finishing at #{Time.now}]\n\n"
  end

end

Customer Module customer.rb

require 'test/unit'
include Test::Unit::Assertions

module Customer

  TITLE = 'Pragprog Books Online Store'
  URL = 'http://localhost:3000/store/'

  # Description:: Adds a book named 'book_title' to cart
  def Customer.add_book(browser, book_title)
    browser.goto(URL)
    # Check if title is already in cart - so we can check it was added correctly
    browser.link(:text,'Show my cart').click
    prev_cart_count = 0
    prev_cart_total = 0.00
    if not browser.div(:text,'Your cart is currently empty').exist? then
     # We have a non-empty cart
      for row in browser.table(:index,1)
        if row[2].text == book_title then
          prev_cart_count = row[1].text.to_i
          break
        end
      end
      prev_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f #remove $ sign
      browser.link(:text, 'Continue shopping').click
    end

    found = false
    book_price = 0.00
    1.upto(browser.divs.length) do |index|
      if (browser.div(:index,index).attribute_value('className') == 'catalogentry') and (browser.div(:index,index).h3(:text,book_title).exists?) then
        book_price = browser.div(:index,index).span(:class, 'catalogprice').text[1..-1].to_f #remove $ sign
        browser.div(:index,index).link(:class,'addtocart').click
        found = true
        break
      end
    end
    if not found then
      return false,'Could not locate title in store'
    end

    new_cart_count = 0
    for row in browser.table(:index,1)
      if row[2].text == book_title then
        new_cart_count = row[1].text.to_i
        break
      end
    end
    new_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f # remove $ sign
    assert_equal(new_cart_count,(prev_cart_count+1), "Ensure that new quantity is now one greater than previously")
    assert_equal(new_cart_total,(prev_cart_total + book_price), "Ensure that new cart total is old cart total plus book price")
    browser.link(:text, 'Continue shopping').click
    return true,new_cart_total
  end

  def Customer.check_out(browser, customerName, customerEmail, customerAddress, customerPaymentMethod)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    browser.link(:text,"Checkout").click
    browser.text_field(:id, 'order_name').set(customerName)
    browser.text_field(:id, 'order_email').set(customerEmail)
    browser.text_field(:id, 'order_address').set(customerAddress)
    begin
      browser.select_list(:id, 'order_pay_type').select(customerPaymentMethod)
    rescue Watir::Exception::NoValueFoundException
      flunk('Could not locate customer payment method in drop down list: '+customerPaymentMethod)
    end
    browser.button(:name, 'commit').click
    if browser.div(:id,'errorExplanation').exist? then
      error = ''
      1.upto(browser.div(:id,'errorExplanation').lis.length) do |index|
        error << (browser.div(:id,'errorExplanation').li(:index,index).text + ",")
      end
      browser.link(:text,'Continue shopping').click
      return false, error
    end
    assert_equal(browser.div(:id,'notice').text, 'Thank you for your order.',"Thank you for your order should appear.")
    return true,''
  end

  def Customer.empty_cart(browser)
    browser.goto(URL)
    browser.link(:text,"Show my cart").click
    if browser.div(:text,"Your cart is currently empty").exist? then
      assert('Cart was never empty')
    else
      browser.link(:text,'Empty cart').click
      assert_equal(browser.div(:id, 'notice').text,'Your cart is now empty')
    end
    return true,''
  end

  def Customer.check_cart_total(browser, exp_total)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    act_total = browser.cell(:id, 'totalcell').text[1..-1].to_f
    assert_equal(act_total,exp_total.to_f,"Check that cart total is as expected.")
    return true,act_total
  end
end

Admin Module admin.rb


require 'test/unit'
include Test::Unit::Assertions

module Admin
  TITLE = 'ADMINISTER Pragprog Books Online Store'
  URL = 'http://localhost:3000/admin/'

  def Admin.log_on(browser, username, password)
    browser.goto(URL)
    if browser.link(:text,'Log out').exist? then #if already logged in
      browser.link(:text,'Log out').click
    end
    browser.text_field(:id, 'user_name').set username
    browser.text_field(:id, 'user_password').set password
    browser.button(:value, ' LOGIN ').click
    if browser.div(:id, 'notice').exist? then
      return false,browser.div(:id, 'notice').text
    else
      return true,''
    end
  end

  def Admin.ship_items(browser, name)
    browser.goto(URL)
    browser.link(:text, 'Shipping').click
    num_orders = 0
    index = 0
    browser.form(:action,'/admin/ship').divs.each do |div|
      if div.class_name == "olname"
        index+=1
        if div.text == name then
          browser.form(:action,'/admin/ship').checkbox(:index, index).set
          num_orders+=1
        end
      end
    end

    browser.button(:value, ' SHIP CHECKED ITEMS ').click

    if num_orders == 1 then
      assert_equal(browser.div(:id,"notice").text, "One order marked as shipped","Correct notice")
    elsif num_orders > 1 then
      assert_equal(browser.div(:id,"notice").text, "#{num_orders} orders marked as shipped","Correct notice")
    end
    return true, num_orders.to_s
  end

end

Easily define Watir tests in excel, OO, wikis and Google docs using Roo

Posted on
6

I spent this evening playing with Roo, the ruby library for reading data from spreadsheets and I am very impressed. In a very small amount of time I was able to define tests in four different forms/places and could execute my tests from each of these:

  • An Excel file (.xls): stored locally
  • An OpenOffice (.ods): stored locally
  • An Excel file (.xls) stored in a Confluence wiki page with Confluence Office Connector; and
  • A Google Docs spreadsheet.

The great thing about Roo is that you don’t actually need Excel; Roo simply reads the file, unlike the ruby Excel COM WIN32 API I have used previously.

The spreadsheet (embedded in Confluence) looks like this:

excel-in-confluence

The cool thing about embedding it in Confluence is that you can click the title of the spreadsheet to edit it (in OpenOffice in my case).

I made some minor changes to my existing code that executed my depot tests from a wiki page, and it was as easy as that. A data driven Watir solution with four possible ways to define test cases. Cool.

You can find all the code needed below.


require 'watir'
require 'rubygems'
require 'roo'
require './Customer.rb'
require './Common.rb'

case ARGV[0]
when "excel"
	ss = Excel.new("watirmelon.xls")
when "wiki"
	ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
when "gdocs"
	ss = Google.new("http://spreadsheets.google.com/ccc?key=pEcLrW3b2djraE8JF_2fJWA")
else
	ss = Openoffice.new("watirmelon.ods")
end

ss.default_sheet = ss.sheets.first
ss.first_row.upto(ss.last_row) do |line|
	if ss.cell(line,1).strip != "Function" then #We have an executable test
		begin
			module_name = ss.cell(line,1).strip
			method_name = ss.cell(line,2).downcase.strip.sub(' ','_') # automatically determine function name based upon method name.
			comments = ss.cell(line,3).strip
			expected_outcome = ss.cell(line,4).strip
			expected_error = ss.cell(line,5).strip
			required_module = Kernel.const_get(module_name)
			required_method = required_module.method(method_name)
			arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
			arity = ((arity * -1) - 1) if arity < 0
			parameters = []
			1.upto(arity) do |p|
				parameters.push(ss.cell(line,p+5))
			end
			actual_outcome, actual_output = required_method.call(*parameters)
			# determine the result.
			if (expected_outcome = 'Success') and actual_outcome then
			    result = "PASS"
			elsif (expected_outcome = 'Error') and (not actual_outcome) and (expected_error = actual_output) then
			    result = "PASS"
			else
			    result = "FAIL"
			end
			puts "\nRunning Test: #{method_name} for #{module_name}."
			puts "Expected Outcome: #{expected_outcome}."
			puts "Expected Error: #{expected_error}."
			puts "Actual Outcome: #{actual_outcome}."
			puts "Actual Output: #{actual_output}."
			puts "RESULT: #{result}"
		rescue
			puts "An error occurred: #{$!}"
		end
	end
end

See the full test code below the break.

Continue reading →

Running watir tests from a confluence wiki page

Posted on
3

Background Information

Defining test cases in a wiki has many benefits:

  • It’s easy to write, read and update;
  • It’s centrally located and easily accessible;
  • There’s in-built version control.

I know that FitNesse/Faucets is a wiki built for this purpose, but I have used it before and have a few issues with it:

  • It defines the test scripts in wiki tables which means it’s difficult to write logic;
  • The FitNesse wiki functionality is basic compared to other wiki software available;
  • I couldn’t find a way to store the historical results of test runs; and
  • Most organisations already have a enterprise wiki in place so having another wiki is duplicating effort.

Its just as easy to use your enterprise wiki and its API to run your Watir tests, and then write back the results (but that’s another post).

My favourite enterprise wiki software is Confluence, not just because it’s a rock solid product but also because I love Atlassian’s philosophies. This includes providing the software to open source projects, such as Watir, and also providing free personal licenses which I have used to provide this example.

Writing Watir tests

I have already discussed how I usually organise my Watir tests. In this example I am using three methods I have already written for the Depot Rails Application.

Customer.add_book(bookTitle)
Customer.check_out(customerName, customerEmail, customerAddress, customerPaymentMethod)
Customer.empty_cart()

The full code for each method is at the bottom of this post.

Defining test cases in Confluence

I have created a very basic wiki page that defines some test cases in wiki tables. In this example I have three different tests, but in reality I would have more permutations in each table. (Click image for large version)

Confluence Wiki Page with Watir Tests Defined

Note that each test case has a Function, Test and test data associated with it. The reason the test data is on the right hand side of the tables is to support varying amounts of test data for various test cases. Also note that the table supports both positive tests and negative test cases (testing for errors).

Another benefit of using a wiki page is that you can easily add additional information. For example, you may have a manual verification step at the end of your tests that you can also document in the same wiki page: it needn’t be in a different place. You can even insert pictures and attachements.

Writing a basic script to run Watir tests defined in Confluence

After a wiki page has been created it’s a matter of writing some ruby code to parse the page and run the tests. Fortunately there is already an excellent ruby library already written for this purpose: Confluence4r.

So really it is just a matter of reading each line in the wiki page and looking for a line that is a table row (not a header), determining which ruby method to run and running it.


require 'watir'
require './confluence4r.rb' # for wiki
require './Customer.rb'
require './Common.rb'

server = Confluence::Server.new('http://localhost:8080')
server.login('admin', 'password')
wiki_page = server.getPage("Watir","WatirMelon")
page_content = wiki_page['content']
page_content.each do |line|
    if line =~ /^\|{1}[^\|]/ then # if line begins with a | then we have an executable test.
        begin
        line.strip! # remove newline markers.
        cells = line[1..-1].split('|') #split line by | excluding first character.
        raise 'Not enough parameters.' if cells.length < 4
        cells.each { |cell| cell.strip! }
        module_name = cells[0]
        method_name = cells[1].downcase.strip.sub(' ','_') # automatically determine function name based upon method name.
        comments = cells[2]
        expected_outcome = cells[3]
        expected_error = cells[4]
        parameters = cells[5..-1] # the remaining cells.
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0
        if cells[5..-1].length != arity then
            raise 'Number of parameters supplied do not match method.'
        end
        actual_outcome, actual_output = required_method.call(*parameters)
        # determine the result.
        if (expected_outcome = 'Success') and actual_outcome then
            result = "PASS"
        elsif (expected_outcome = 'Error') and (not actual_outcome) and (expected_error = actual_output) then
            result = "PASS"
        else
            result = "FAIL"
        end
        puts "\nRunning Test: #{method_name} for #{module_name}."
        puts "Expected Outcome: #{expected_outcome}."
        puts "Expected Error: #{expected_error}."
        puts "Actual Outcome: #{actual_outcome}."
        puts "Actual Output: #{actual_output}."
        puts "RESULT: #{result}"
        rescue
            puts "An error occurred: #{$!}"
        end
    end
end

Note that I have dynamically determined the module and method name from the wiki and I have then dynamically called it with the appropriate number of arguments using arity.

When I run it the ruby code the output is:

Running Test: empty_cart for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

Running Test: add_book for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

Running Test: check_out for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

This could be substantially improved but it’s a start. For example, we could easily add timings for each test case.

Next Steps

The next step is to dynamically create a child results page on the Confluence wiki for each time you run a test. That way there is a historical record so you can then schedule your tests to run automatically and check the results later on using the wiki.

I’ll be posting some more information and code in the coming weeks on how to do this.

Click below to see the Watir code I wrote and used.

Continue reading →

Business Driven Testing

Posted on
1

Watir is a great library, but to use it to its full potential you need to create your own framework to drive it.

As with any context driven approach, the framework/solution you decide to implement has to suit your own environment. One approach that I have used successfully in multiple projects (with tweaks for each) is the business driven approach.

All the software projects I have worked on have had one main purpose, that is to support a business need. This business need may be to allow people to easily travel from country to country, or in contrast, allow enthusiasts to efficiently buy books online. Creating a test suite that is focussed around these real business activities will clarify the link from these to the the user tests you are writing and running. This also compliments a risk based testing approach as it is often easy to get business people to express the important and risky areas of the business, this being what will be tested first.

The concept of my framework is that the functionality of the software is divided into functions, each of which has user tests defined and scripted. These user tests can be easily strung together to test entire business scenarios, or soap opera scenarios if you are so inclined.

The first thing to do is split the areas of the software up into functions. You can then define user tests for each function. User tests are something that a user will do or currently does. They have to be real world. They usually consist of some inputs (data), some processing and an outcome. If the outcome is positive, usually there is output, for example, an ID. If the outcome is negative, usually there is an error message.

Performing this activity for the Depot Rails Application creates something like:

Function: Customer
User Tests: Empty Cart, Add Book to Cart, Check Out

Function: Administration
User Tests: Log On, Ship All Book Orders, Ship Some Book Orders, Add User, Delete User

This activity can be done for any kind of requirements and even if requirements don’t exist. You can do this from a functional requirements specification, use cases/user stories (easier), and you can do this (albeit less easily) from an existing system interface or prototype.

Your functions then become modules of Watir code, and the user tests become methods in these modules. Each method takes some data, and returns an outcome (it worked or it didn’t) and optionally an error or an output value.

For example, this is an empty Customer module with just the methods defined.

Customer.rb

module Customer
   URL = 'http://localhost:3000/store/'

   def Customer.add_book(book_title)
   end

   def Customer.check_out(customer_name, customer_email, customer_address, customer_payment_method)
   end

  def Customer.empty_cart
  end
end

For efficiency and usability, I always ensure that each user test is home based, meaning that it starts and finishes on the same home page/screen. This avoids ‘state’ problems occurring, so a series of user tests can be called to test a business scenario without worrying about where each user test starts and ends. This also avoids repetitious and inefficient startup/logging in/logging out for each user test. Don’t laugh, I have often seen this done.Your user tests should accommodate positive and negative tests in the same method. For example, our ‘Customer.check_out(…)’ user test should be able to test both valid and invalid credit card numbers. This is done by making the method return the outcome and then determing the result outside the method depending on your expected outcome.For example, although the method may return an error, we could be expecting this in our negative test so therefore our test actually passes.I have seen many people write specific methods to test specific error messages. Don’t be tempted to write ‘Customer.check_out_valid_card(…)’ and ‘Customer.check_out_invalid_card(…)’. This leads to an unmaintainable set of user tests due to the repetition of code required. Limiting the number of methods also makes it easier to define business scenarios as there are a limited number of methods for a business scenario tester to choose from.
Once you have defined modules and methods you need to define the business scenarios which involve running user stories providing data (positive and negative) as well as expected outcomes. It is best to use a data presentation language to do this.

Excel is very common data presentation language for designing user tests and business scenarios but versioning excel files can be problematic due to their binary nature.

A wiki is excellent for defining user tests and business scenarios in wiki tables as wikis have in built versioning, a centralised, accessible and flexible nature, and are generally easy to use.

In the coming weeks I will discuss how to set up a wiki page in Confluence as a business scenario which includes a series of user tests and then how to dynamically call the ruby methods and determine the test results from the method outcomes.