Capturing page performance times with my WatirMelonCucumber framework

I recently wrote about Tim Koopman’s watir-webdriver-performance gem and how it makes it very easy to collect browser performance stats when you’re running automated tests. Unfortunately it’s Chrome (and IE9) only at the moment, but it looks very promising for when Firefox also supports it (version 7 I believe).

One of my ideas when I first saw this gem was to collect page performance statistics when running automated tests using Cucumber and a page object model, and store a summary of response times for each run. The idea is that you can see how your app is performing over time and whether you have any issues being introduced.

Today, I implemented this functionality in both the WatirMelonCucumber framework, and also my suite of EtsyWatirWebDriver tests.

It’s a fairly simple concept. First I have this class that collects the stats:

require 'yaml'

class PageMetrics
  def initialize
    @pages = {}
  end

  def add_page class_name, performance
    @pages[class_name] ||= []
    @pages[class_name] << performance
  end

  def summary
    summary = []
    @pages.each do |page, metrics|
      average = metrics.inject(0) { |sum, metric| sum + metric.summary[:response_time] }/metrics.size/1000
      result = {
        :page => page.to_s,
        :visits => metrics.size,
        :average_time => average
      }
      summary << result
    end
    summary.to_yaml
  end
end

Then I make sure the class is instantiated and accessible in my cucumber env.rb file:

module Metrics
  def self.page_metrics
    @page_metrics ||= PageMetrics.new
  end
end

Then I make my base page class collect the page metrics (if the browser allows). This uses a method called .with_performance which I recently wrote for the watir-webdriver-performance gem, and is in there from 0.1.3.

class BasePageClass
  include WatirPageHelper

  def initialize browser, page_metrics, visit = false
    @browser = browser
    goto if visit
    expected_element if respond_to? :expected_element
    has_expected_title? if respond_to? :has_expected_title?
    @browser.with_performance {|performance| page_metrics.add_page self.class, performance } unless visit
  end
end

Finally, when I finish running my cucumber tests, I have a hook in my cucumber env.rb file that summarises the page performance times.

at_exit do
  puts Metrics.page_metrics.summary
  File.open('pagemetrics.yml', 'w') { |file| file.puts Metrics.page_metrics.summary }
  Browser::BROWSER.close
end

The yaml file produced contains content that looks something like this:

- :page: EtsyAdvancedSearchPage
  :visits: 8
  :average_time: 0.72075
- :page: EtsySearchResultsPage
  :visits: 6
  :average_time: 1.9331666666666667
- :page: EtsyHomePage
  :visits: 1
  :average_time: 0.789
- :page: EtsyBuyPage
  :visits: 1
  :average_time: 0.741
- :page: EtsyTreasuryPage
  :visits: 1
  :average_time: 1.97
- :page: EtsyCartPage
  :visits: 4
  :average_time: 1.23825
- :page: EtsyItemPage
  :visits: 1
  :average_time: 1.61

You can check out the entire solution in either github repository.

Thanks to Tim for releasing the gem and accepting my pull requests.

Let me know what you think.

Easily capturing response time metrics using the Watir-WebDriver-Performance gem

I don’t like long dedicated performance testing windows at the end of a project. I see response times as non-functional requirements, and like any other requirements, these should be tested as we’re going along. One way of effectively measuring performance testing is by conducting a response time test every time you do a build, if there’s a big degradation in performance: break the build!

A couple of months ago, Tim Koopmans released the Watir-WebDriver-Peformance gem. It’s aimed at providing a set of navigation timing metrics for Watir-WebDriver actions. This is a perfect solution to capture response time metrics, and it’s very straightforward to do (only works in Chrome and IE9 at the moment – no Firefox support).

require 'watir-webdriver'
require 'watir-webdriver-performance'

b = Watir::Browser.new :chrome

10.times do
  b.goto 'http://watir.com'
  puts "Load Time: #{b.performance.summary[:response_time]/1000} seconds."
end  

This produces something like:

Load Time: 3.701 seconds.
Load Time: 0.694 seconds.
Load Time: 1.874 seconds.
Load Time: 1.721 seconds.
Load Time: 2.096 seconds.
Load Time: 0.823 seconds.
Load Time: 2.362 seconds.
Load Time: 1.008 seconds.
Load Time: 1.761 seconds.
Load Time: 2.066 seconds. 

List of available metric groupings

  • :summary
  • :navigation
  • :memory
  • :timing

Summary

The Watir-WebDriver-Performance gem is a great way to capture response time metrics from your Watir-WebDriver automated tests. By instrumenting your existing Watir-WebDriver automated tests that run regularly as part of a continuous integration process, you can start to capture and measure web application performance over time, and be alerted to the possibility of a change being introduced with adverse performance effects. Well done Tim!