Background Information

Defining test cases in a wiki has many benefits:

• It’s easy to write, read and update;
• It’s centrally located and easily accessible;
• There’s in-built version control.

I know that FitNesse/Faucets is a wiki built for this purpose, but I have used it before and have a few issues with it:

• It defines the test scripts in wiki tables which means it’s difficult to write logic;
• The FitNesse wiki functionality is basic compared to other wiki software available;
• I couldn’t find a way to store the historical results of test runs; and
• Most organisations already have a enterprise wiki in place so having another wiki is duplicating effort.

Its just as easy to use your enterprise wiki and its API to run your Watir tests, and then write back the results (but that’s another post).

My favourite enterprise wiki software is Confluence, not just because it’s a rock solid product but also because I love Atlassian’s philosophies. This includes providing the software to open source projects, such as Watir, and also providing free personal licenses which I have used to provide this example.

Writing Watir tests

I have already discussed how I usually organise my Watir tests. In this example I am using three methods I have already written for the Depot Rails Application.

Customer.add_book(bookTitle)
Customer.check_out(customerName, customerEmail, customerAddress, customerPaymentMethod)
Customer.empty_cart()

The full code for each method is at the bottom of this post.

Defining test cases in Confluence

I have created a very basic wiki page that defines some test cases in wiki tables. In this example I have three different tests, but in reality I would have more permutations in each table. (Click image for large version)

Note that each test case has a Function, Test and test data associated with it. The reason the test data is on the right hand side of the tables is to support varying amounts of test data for various test cases. Also note that the table supports both positive tests and negative test cases (testing for errors).

Another benefit of using a wiki page is that you can easily add additional information. For example, you may have a manual verification step at the end of your tests that you can also document in the same wiki page: it needn’t be in a different place. You can even insert pictures and attachements.

Writing a basic script to run Watir tests defined in Confluence

After a wiki page has been created it’s a matter of writing some ruby code to parse the page and run the tests. Fortunately there is already an excellent ruby library already written for this purpose: Confluence4r.

So really it is just a matter of reading each line in the wiki page and looking for a line that is a table row (not a header), determining which ruby method to run and running it.

require 'watir'
require './confluence4r.rb' # for wiki
require './Customer.rb'
require './Common.rb'

server = Confluence::Server.new('http://localhost:8080')
server.login('admin', 'password')
wiki_page = server.getPage("Watir","WatirMelon")
page_content = wiki_page['content']
page_content.each do |line|
    if line =~ /^\|{1}[^\|]/ then # if line begins with a | then we have an executable test.
        begin
        line.strip! # remove newline markers.
        cells = line[1..-1].split('|') #split line by | excluding first character.
        raise 'Not enough parameters.' if cells.length < 4
        cells.each { |cell| cell.strip! }
        module_name = cells[0]
        method_name = cells[1].downcase.strip.sub(' ','_') # automatically determine function name based upon method name.
        comments = cells[2]
        expected_outcome = cells[3]
        expected_error = cells[4]
        parameters = cells[5..-1] # the remaining cells.
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0
        if cells[5..-1].length != arity then
            raise 'Number of parameters supplied do not match method.'
        end
        actual_outcome, actual_output = required_method.call(*parameters)
        # determine the result.
        if (expected_outcome = 'Success') and actual_outcome then
            result = "PASS"
        elsif (expected_outcome = 'Error') and (not actual_outcome) and (expected_error = actual_output) then
            result = "PASS"
        else
            result = "FAIL"
        end
        puts "\nRunning Test: #{method_name} for #{module_name}."
        puts "Expected Outcome: #{expected_outcome}."
        puts "Expected Error: #{expected_error}."
        puts "Actual Outcome: #{actual_outcome}."
        puts "Actual Output: #{actual_output}."
        puts "RESULT: #{result}"
        rescue
            puts "An error occurred: #{$!}"
        end
    end
end

Note that I have dynamically determined the module and method name from the wiki and I have then dynamically called it with the appropriate number of arguments using arity.

When I run it the ruby code the output is:

Running Test: empty_cart for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

Running Test: add_book for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

Running Test: check_out for Customer.
Expected Outcome: Success.
Expected Error: .
Actual Outcome: true.
Actual Output: .
RESULT: PASS

This could be substantially improved but it’s a start. For example, we could easily add timings for each test case.

Next Steps

The next step is to dynamically create a child results page on the Confluence wiki for each time you run a test. That way there is a historical record so you can then schedule your tests to run automatically and check the results later on using the wiki.

I’ll be posting some more information and code in the coming weeks on how to do this.

Click below to see the Watir code I wrote and used.

Continue reading →