Checking an image is actually visible in WebDriverJs

I recently discovered a gap in one of my e2e automated tests where I was checking the existence of an uploaded image in the DOM, but not that the image was actually displayed.

driver.isElementPresent( By.css( `img[alt='upload.jpg']` ) ).then( function( present ) {
  assert.equal( present, true, 'Image not displayed' );
} );

If the DOM has a reference to the image, but it isn’t actually rendered this test will pass. This isn’t ideal.

I remembered my post about how to check that an image is actually rendered using WebDriver in C# and so I used the same JavaScript script which WebDriverJs sends to the driver:

driver.findElement( By.css( `img[alt='upload.jpg']` ) ).then( function( element ) {
  driver.executeScript( 'return (typeof arguments[0].naturalWidth!=\"undefined\" && arguments[0].naturalWidth>0)', element ).then( function( present ) {
    assert.equal( present, true, 'Image not displayed' );
  } );
} );

This works a treat. I’ve moved it into a helper function so I can use this anywhere without repeating it also.

Software testers shouldn’t write code

Software testers shouldn’t write code. There I’ve said it.

“If you put too much emphasis on those [automated test] scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.”

~ Joel Spolsky on testers

I used to think that you could/should teach testers to write code (as it will make them better testers), but I’m now at a point where I think that it’s a bad idea to teach testers to code for a number of reasons:

  1. A software tester’s primary responsibility/focus should always be to test software. By including a responsibility to also write code/software takes away from that primary focus. Testers will get into a trap of sorting out their own coding issues over doing their actual job.
  2. If a software tester wants their primary focus to be writing code, they should become a software programmer. A lot of testers want to learn coding not because they’ll be a better tester, but they want to earn more money. These testers should aim to be become programmers/developers if they want to code or think they can earn more money doing that.
  3. Developing automated tests should be done as part of developing the new/changed functionality (not separately). This has numerous benefits such as choosing the best level to test at (unit, integration etc.) at the right time. This means there isn’t a separate team lagging behind the development team for test coverage.
  4. Testers are great at providing input into automated test coverage but shouldn’t be responsible for creating that coverage. A tester working with a developer to create tests is a good way to get this done.

I think the software development industry would be a lot better if we had expectations on programmers to be responsible for self-tested code using automated tests, and testers to be responsible for testing the software and testing the the automated tests. Any tester wanting to code will move towards a programming job that allows them to do that and not try to change what is expected of them in their role.

Update 19th Jan 2015: this post seems to have triggered a lot of emotion, let me clarify some things:

  • A tester having technical skills isn’t bad: the more technical skills the tester has the better – if they can interrogate a database or run a sql trace then they’ll be more efficient/effective at their job – and a tester can be technical without knowing how to code
  • I don’t consider moving from testing into programming by any means the only form of career advancement: some testers hate coding and that’s fine, other’s love coding and I think it would be beneficial for them to become a programmer if they want to code more than they test.
  • I still believe everyone should take responsibility for their own career rather than expecting their employer/boss/industry leader/blogger to do it for them (more about this here).

The value of automated testing

I recently saw an email thread discussing the value of test automation and asking how to justify it to management. It just seems such a strange concept to me. Justifying the value of something that is so essential to writing good quality software, is like justifying the value of exercise to a human body: it’s benefits are so obvious it’s almost a waste of time attempting to justify it.

But the main reason I find justifying the value of automated testing so strange is I don’t really see any viable alternative. It’s like justifying to your boss the benefit of flying from Australia to the USA and back for a conference (~30 hours return) instead of taking a ship (50-80 days).

If we have a critical bug in our production system we need to turn around a fix in less than an hour. To turnaround a fix in less than an hour we need full regression test coverage that can give us feedback that we’re all good and haven’t broken anything else in half that time. To get the same coverage that our automated regression test suite has through 30 minutes of manual test execution would require having about 184 QA staff always available to do regression testing in parallel: that’s not going to happen.

Unless you can wait days/weeks for manual regression testing, or have a sufficiently large team of QA resources always available to test, you really can’t release software quickly with confidence that it’s high quality. With automated tests you can; and that’s the value of automated testing.

Take control of your own career

During my career, I’ve come across numerous testing colleagues with no experience in automated testing who say things like “I’d love to do automated testing”. They expect to be put into an automated testing role so they can learn automated testing.

I don’t think it should work like that. Your employer shouldn’t be solely responsible for you enhancing your skills and progressing your career.

And, the thing is, it’s never been easier to pick up some new technical skills.

If you want to learn programming start by learning something like Ruby. If you want to learn about automated web testing learn Watir. If you want to learn about behavior driven development tools learn Cucumber.

I taught myself Ruby. I taught myself Watir. I taught myself C#, Python, Selenium, Cucumber and Jenkins. The list goes on.

The barrier to entry has never been lower. Try codeacademy, try ruby koans, download the free watir book, buy Cheezy’s cheap eBook about Watir & Cucumber.

So, instead of watching television or going out for drinks, spend your nights and weekends learning some new skills and taking control of your career instead of expecting your employer to hand it to you on a plate.

You’ll then be able to say “I’m learning all about Watir at the moment and I would love to apply that on a project” instead of “I’d love to do automated testing”.

Which is better: manual or automated testing?

This post is part of the Pride & Paradev series.


Which is better: manual or automated testing?


Manual Testing is better than Automated Testing

Manual testing is better than automated testing. Even when automating a test scenario, you have to manually test it at least once anyway to automate it, so automated testing can’t be done without manual testing. And you have to manually check the automated test results also.

Automated tests can stop working by something as simple as an unexpected pop-up dialog which can be quickly analyzed and dismissed when manually testing.

Manual testing is a sapient activity: one that requires human judgement. As you are testing you are using implicit knowledge to judge whether or not something is working as expected. This enables you to find extra bugs that automated tests would never find. It also allows you to follow smells you find to explore areas that may not have been tested or required.

Manual testing is also helpful for finding layout issues and trivial bugs which wouldn’t be found by an automated test, as you’re fully observing the application as you’re using it. Usability issues are also identifiable by manual testing but can’t be discovered through writing and running automated test scripts.

Automated Testing is better than Manual Testing

Automated testing is better than manual testing. Automated tests are very explicit (black and white) so you have a much higher chance of reproducing a bug if found by an automated test by knowing what the automated test executed to achieve the result. Because the automated tests are explicit, they also execute consistently as they don’t get tired and/or lazy like us humans.

Automated tests are quicker to run than manual tests as there’s no lag time between input and checking, and this means you can run more tests in more browsers more quickly. Manually testing the same functionality in, for example, 8 browsers and 4 devices is tiring, but can easily be achieved with automated tests.

Automated tests also allow you to test things that aren’t manually possible. For example, answering a question like ‘what if I had 200 accounts’, or ‘what if I processed ten transactions simultaneously’ can only be answered efficiently by using automated tests.

Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser = Watir::Browser.new

Before do
  @browser = browser
end

at_exit do
  browser.close
end

To open and close the browser only once in RSpec:

browser = Watir::Browser.new

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }
end

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = 'https://en.wikipedia.org'
end

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"
end

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"
end

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'
end

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'
end

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

My simple Cucumber + Watir page object pattern framework

Introduction

I was very impressed with Jeff Morgan, known as Cheezy, who recently wrote a series of blog posts about how to use Cucumber and Watir, and shared his code on Github.

I love it when people share their ideas this like, so I thought I would share what I have found useful when setting up a very simple Cucumber with Watir (Celerity & Watir-WebDriver) page object pattern framework, and how this compares to what Jeff has proposed.

Show me the code!

Before we begin, I’ll show you my code. It’s all on Github, right now, as we speak, and you can easily fork and clone this repository to play around with it. It uses a simple google search example.

Project Folder Structure

Google Search Feature

Feature: Google Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

Scenario: Search for Watir
  Given I am on the Google Home Page
  When I search for "Watir"
  Then I should see at least 100,000 results

Scenario: Do a unit conversion
  Given I am on the Google Home Page
  When I convert 10 cm to inches
  Then I should see the conversion result 
            "10 centimeters = 3.93700787 inches"

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

Page Object Pattern

For our example, we have two page objects. The page classes delegate any methods that don’t exist on the page to the Browser object that is passed in from Cucumber. This ensures you can call browser specific methods (.title, .url etc.) at the page object level without duplicating methods.

class GoogleHomePage

  attr_accessor :search_field, :google_search_button

  URLS = { :production => "http://www.google.com/" }

  def initialize(browser)
    @browser = browser
    @search_field         = @browser.text_field(:name => "q")
    @google_search_button = @browser.button(:name => "btnG")
  end

  def method_missing(sym, *args, &block)
    @browser.send sym, *args, &block
  end

  def visit
    @browser.goto URLS[:production]
  end

  def page_title
    @browser.title
  end

  def search_for term
    self.search_field.set term
    self.google_search_button.click
    google_results_page = GoogleResultsPage.new(browser)
    google_results_page.results.wait_until_present if WEBDRIVER
    google_results_page
  end

end

Cucumber Initialization

I have everything to do with initialization in support/env.rb. This initializes a browser once, and then uses the same browser instance for each Cucumber scenario. This is different from Jeff’s hooks.rb file that launches a new browser for every scenario. I have found by reusing the browser, you get much quicker test execution time.

The INDEX_OFFSET is a constant that allows me to use ‘index’ when referring to browser elements across both celerity with its 1-based index and watir-webdriver with its 0-based index.

TEST_DATA_DIR = "./features/test_data"

if ENV["HEADLESS"] then
  require "celerity"
  browser = Celerity::Browser.new
  INDEX_OFFSET = 0
  WEBDRIVER = false
else
  require 'watir-webdriver'
  require 'watir-webdriver/extensions/wait'
  browser = Watir::Browser.new :firefox
  INDEX_OFFSET = -1
  WEBDRIVER = true
end

Before do
  @browser = browser
end

at_exit do
  browser.close
end

Calling the page objects from the steps

As you can see, most of my steps are in a declarative style, so these normally align pretty well to a method on the page object.
I use a given step to create a page object (in our case @google_home_page), and then call methods on this object for when and then steps. I have created a dynamic invocation of the page creation, so it will work for all pages specified in the cucumber step itself. I use the search_for_term method to transition between page objects, as this method returns the new page. As a rule of thumb, I try to keep my step definition code to 1 or 2 lines, and at a fairly high level, which makes it much more readable.

Given /^I am on the (.+)$/ do |page_name|
  @google_home_page = Object.const_get(page_name.gsub(" ","")).new(@browser)
  @google_home_page.visit
end

When /^I search for a? ?"([^"]*)"$/ do |term|
  @google_results_page = @google_home_page.search_for term
end

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Then /^I should see at least ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should >= exp_num_results.gsub(",","").to_i
end

Then /^I should see at most ([\d,]+) results$/ do |exp_num_results|
  @google_results_page.number_search_results.should <= exp_num_results.gsub(",","").to_i
end

When /^I convert (.+)$/ do |conversion_statement|
  @google_results_page = @google_home_page.search_for "convert #{conversion_statement}"
end

Then /^I should see the conversion result "([^"]*)"$/ do |exp_conversion_result|
  @google_results_page.conversion_result.text.should == exp_conversion_result
end

Expressing test data external to the features/pages

Unlike Jeff’s recommendation of using default data on page objects, I prefer to store test data externally to the page objects in YAML files. That way I can describe the test data which is directly used in my Cucumber step. For example:

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 5 results

The phrase ridiculously small number of results matches directly to a section of my YAML test data file. This keeps the scenario highly readable, with the detailed data specified in the YAML file.

search terms:
  ridiculously small number of results:
    search term: 'macrocryoglobulinemia marvel'

A common method simply retrieves the appropriate test data, and provides it to the step to use. Note here that, as this step doesn’t contain quotation marks, this implies a description of data, rather than a literal string of data that would contain quotation marks around the string.

When /^I search for a?n? ?([^"].+[^"])$/ do |term|
  term = Common.get_search_term_data term
  @google_results_page = @google_home_page.search_for term
end

Putting it all together
I have set up two profiles in the cucumber.yml file, one is the default that uses Firefox under Watir-Webdriver, and one runs Celerity (headless) and is named ci. It’s then a case of calling "cucumber" or "cucumber -p ci" to run each of these profiles.

I have set up a ./go script that bootstraps the entire process on unix based systems (Mac OSX and Linux).

The final result

Running cucumber provides results as green as a cucumber.

How does this compare to Cheezy’s framework I mentioned?

  • I am using a similar page object pattern to define web application pages, but I haven’t created a helper class to mix into each like he has: I simply use instance variables on a page class to define the GUI elements, instead of creating a series of methods for each object.
  • My page objects also delegate methods to the main Browser object as required, so you can directly call things like .title, .url etc.
  • Instead of using Jeff’s default data pattern, I instead prefer to store this data described in external YAML files instead for maximum flexibility.
  • Jeff talks about using mixins to mix in common objects. I haven’t done this, but it’s easily doable with my framework, and should be done as it scales out.

Summary

Jeff has done an excellent job with his series of posts and code (thank you), and I am hoping here to build on that, and show how it’s easy to set up a simple Cucumber & Watir page object pattern framework.