AMA: more details about our JS e2e tests

Satyajit Malugu asks…

Saw your recent post on JS vs Ruby. As a recent migrant from Ruby to JS for mobile test automation – couple of questions 1) how are you doing page objects (ES6 works too) 2) Are you able to run your E2E tests in parallel? 3)Is there anything like binding.pry for debugging JS whilst execution?)

My response…

  1. This post explains how we’re doing our page objects in ES6, and since all our e2e tests are open source, you can see for yourself :)
  2. We use CircleCI which supports parallel test execution, which I enabled in the past, but it’s currently disabled as it was consuming too many build containers which are shared across our organization. Since we have a lot more containers available now I have it on my list to renable these.
  3. I believe WebStorm supports debugging Node Mocha tests; I’ve been meaning to investigate this over console.log statements; I’ll report back when I get around to it.

AMA: R.Y.O. Page Objects 2.0

Michael Karlovich asks…

Do you have any updated thoughts on rolling your own page objects with Watir? The original post is almost 4 years old but is still the basis (loosely) of every page object framework I’ve built since then.

My response…

Wow, I can’t believe that post is almost four years old. I have also have used this for the basis of every page object framework I have built since then.

I recently had a look at our JavaScript (ES2015) code of page objects and despite ES2015 not having meta-programming support like ruby, our classes are remarkably similar to what I was proposing ~4 years ago.

I believe this is because some patterns are classic and therefore almost timeless, they can be applied over and over again to different contexts. There’s a huge amount of negativity towards best practices of late, but I could seriously say that page objects are a best practise for test automation of ui systems, which isn’t saying they will be exactly the same in every context, but there’s a common best-practice pattern there which you most likely should be using.

Page objects, as a pattern, typically:

  • Inherit from a base page object/container which stores common actions like:
    • instantiating the object looking for a known element that defines that page’s existence
    • optionally allow a ‘visit’ to the page during instantiation using some defined URL/path
    • provides actions and properties common to all pages, for example: waiting for the page, checking the page is displayed, getting the title/url of the page, and checking cookies and local storage items for that page;
  • Define actions as methods which are ways of interacting with that page (such as logging in);
  • Do not expose internals about the page outside the page – for example they typically don’t expose elements or element selectors which should only be used within actions/methods for that page which are exposed; and
  • Can also be modelled as components for user interfaces that are built using components to give greater reusability of the same components across different pages.

The biggest benefit I have found from using page objects as a pattern is having more deterministic end-to-end tests since instantiating a page I know I am on that page, so my tests will fail more reliably with a better understanding of what went wrong.

Are there any other pattern attributes you would consider vital for page objects?

AMA: Page Object Best Practices

Anonymous asks..

What do you think is best practice for storing element locations? I’ve heard about having an “Element map”, a single place where all element IDs or methods for location are stored. Is that a good idea? Should each Page Object have it’s own element map?

My response…

First let me preface my answer with a quote:

“best practices are useful reference points, but they must come with a warning label: the more you rely on external intelligence, the less you will value an internal idea. And this is the age of the idea”

~  Gyan Nagpal, Talent Economics

I tend to store the element locators, or selectors, in the page objects themselves. Typically if I am going to use an element locator more than once within a page object I will store this as a property of the page object, otherwise I will just use it within a method.

So I would typically do this in ES6:

import { By } from 'selenium-webdriver';
import config from 'config';

import BaseContainer from '../base-container.js';
import * as driverHelper from '../driver-helper.js';

export default class LoginPage extends BaseContainer {
	constructor( driver, visit ) {
		const loginURL = `${config.get( 'authURL' )}?redirect_to=${config.get( 'baseURL' )}`;
		super( driver, By.css( '#loginform' ), visit, loginURL );
	}

	login( username, password ) {
		driverHelper.setWhenSettable( this.driver, By.css( '#user_login' ), username );
		driverHelper.setWhenSettable( this.driver, By.css( '#user_pass' ), password, { secureValue: true } );
		return driverHelper.clickWhenClickable( this.driver, By.css( '#wp-submit' ) );
	}
}

over

import { By } from 'selenium-webdriver';
import config from 'config';

import BaseContainer from '../base-container.js';
import * as driverHelper from '../driver-helper.js';

export default class LoginPage extends BaseContainer {
	constructor( driver, visit ) {
		const loginURL = `${config.get( 'authURL' )}?redirect_to=${config.get( 'baseURL' )}`;
		super( driver, By.css( '#loginform' ), visit, loginURL );
		this.userNameSelector = By.css( '#user_login' );
		this.passwordSelector = By.css( '#user_pass' );
		this.submitSelector = By.css( '#wp-submit' );
	}

	login( username, password ) {
		driverHelper.setWhenSettable( this.driver, this.userNameSelector, username );
		driverHelper.setWhenSettable( this.driver, this.passwordSelector, password, { secureValue: true } );
		return driverHelper.clickWhenClickable( this.driver, this.submitSelector );
	}
}

Although I would use the second style (properties) if I had a second method that used the same element locators.

Anonymous also asks..

When creating a new instance of a Page Object, should there be any validation that you’re a) on the correct page and/or b) the correct elements exist? From what I understand, best practice is to keep assertions in the actual tests. Should an error be thrown if I create a LoginPage object but the login page isn’t displayed?

I like to do b) as I find it keeps tests very deterministic meaning if you know at all times you’re on the correct page, then it’s easier to work out when things go wrong. This also works well for single page applications since the DOM ready state doesn’t necessarily mean the page is loaded, so passing in an element you expect you can poll for its existence.

Other ways to check you’re on the right page would be to check the browser title and/or URL but I have found these aren’t as reliable for example the title might be translated and change depending on your locale, and the URL might be dynamic or not change for each dynamic page update.

In the above examples, the base container that all pages/components extend requires an element locator which it uses to check that page exists within a given explicit wait time. It also optionally takes visit and url parameters which allows you to navigate to the page in the browser before checking that the element exists. This is only used for specific pages like login.

WebDriverJS & Mocha Part 4: More Page Objects & Config

This post continues on from my last post about writing page objects in WebDriverJS & Mocha.

At this stage our tests look pretty good…

test.describe('Ralph Says', function() {
    this.timeout(mochaTimeOut);

    test.it('shows a quote container', function() {
        var ralphSaysPage = new RalphSaysPage(driver);
        ralphSaysPage.visit();
        ralphSaysPage.quoteContainerPresent().then(function(present) {
            assert.equal(present, true, "Quote container not displayed");
        });
    });

    test.it('shows a non-empty quote', function() {
        var ralphSaysPage = new RalphSaysPage(driver);
        ralphSaysPage.visit();
        ralphSaysPage.quoteTextDisplayed().then(function(text) {
            assert.notEqual(text, '', 'Quote is empty');
        });
    });
});

…but I think they could be better.

Continue reading “WebDriverJS & Mocha Part 4: More Page Objects & Config”

A ruby testing framework, from scratch, in 15 minutes

As part of my talk last week at the Brisbane Testers Meetup, I gave a live demo (no pre-recorded or pre-written code) of writing a ruby testing framework from scratch in 15 minutes. The idea was to show that most testing frameworks contain so much functionality ‘you ain’t gonna need’, so why not try writing one from scratch and see how we go? It was also a chance to show the testers who hadn’t done automated testing that programming/automated testing is not rocket science.

Since I promised to talk about selenium, I used watir-webdriver, but I would have preferred to just show testing a simple app/class that I would have written from scratch in ruby.

Our testing problem

I wrote a beautifully simple website to welcome the testers to the first ever Brisbane Testers Meetup, and wanted to write some tests to make sure it worked. The site is accessible at data:text/html,<h1 id=”welcome”>Welcome BNE Testers!</h1> and looks something like this:

bnetesterswelcome

First I’ll give you a few moments to get over how amazing that web site is… that’s long enough, now, what we need is a couple of tests for it:

  1. Make sure the welcome message exists
  2. Make sure the welcome message is visible
  3. Make sure the welcome message content is correct

Iteration zero

Do the simplest thing that could possibly work. In our case print out the three things we want to check to the screen and we’ll manually verify them.

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
puts b.h1(id: 'welcome').exists?
puts b.h1(id: 'welcome').visible?
puts b.h1(id: 'welcome').text

which outputs:

true
true
Welcome BNE Testers!

A good start but not quite a testing framework.

Iteration One

I think it’s time to introduce a method to assert a value is true.

I like to start by writing how I want my tests to look before I write any ‘implementation’ code:

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

I usually run the my tests to give me a ‘clue’ to what I need to do next. In our case:

 undefined method `assert' for main:Object (NoMethodError)

In our case, it’s simple, we need to write an assert method. Luckily we know exactly what we need: a method that takes a description string and a block of code that should execute returning true, otherwise we have an error. We can simply write this method above our existing tests:

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}"
		else
			puts "Assertion FAILED for #{message}"
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'"
	end
end

which gives us this output when we run:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

This is awesome, but it makes me nervous that all of our three tests passed the first time we ran them. Perhaps we hard coded them to pass? Will they ever fail?

There’s an old saying, source unknown, which is ‘never trust a test you didn’t first see fail‘. Let’s apply this here by making all our tests fail. I usually do this by changing the source system, that way you can keep the integrity of your tests intact.

This is fairly easy to do in our case by changing the id of our welcome element.

data:text/html,<h1 id=”hello”>Welcome BNE Testers!</h1>

When we do so, all our tests fail: yipee.

Assertion FAILED for that the welcome message exists
Assertion FAILED for that the welcome message is visible with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'
Assertion FAILED for that the welcome message text is correct with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'

We change it back and they pass again: double yipee.

Iteration Two –

So far all the text output has been in same color, and everyone knows a good test framework uses color. Lucky I know a gem that does color output easily, all we do is:

require 'colorize'

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}".green
		else
			puts "Assertion FAILED for #{message}".red
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'".red
	end
end

which gives us some pretty output:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

and a fail now looks like this:

Assertion FAILED for that the welcome message exists

Sweet.

We can put the assert method in its own file which leaves our test file cleaner and easier to read:

require 'watir-webdriver'
require './assertions.rb'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

Iteration Three

Our final iteration involves making the tests even easier to read by abstracting away the browser. This is typically done using ‘page objects’ and again we’ll write how we would like it to look before implementing that functionality:

require 'watir-webdriver'
require './assertions.rb'

Homepage.visit

assert('that the welcome message exists') { Homepage.welcome.exists? }
assert('that the welcome message is visible') { Homepage.welcome.visible? }
assert('that the welcome message text is correct') { Homepage.welcome.text == 'Welcome BNE Testers!' }

HomePage.close

When we run this, it provides us a hint at what we need to do:

uninitialized constant Homepage (NameError)

We need to create a HomePage class with three methods: visit, welcome and close.

We can simply add this to our tests file to get it working:

class Homepage
	def initialize
		@browser = Watir::Browser.new
		@browser.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
	end

	def self.visit
		new
	end

	def welcome
		@browser.h1(id: 'welcome')
	end

	def close
		@browser.close
	end
end

After we’re confident it is working okay, we simply move it to a file named homepage.rb and our resulting tests look a lot neater:

require './assertions.rb'
require './homepage.rb'

homepage = Homepage.visit

assert('that the welcome message exists') { homepage.welcome.exists? }
assert('that the welcome message is visible') { homepage.welcome.visible? }
assert('that the welcome message text is correct') { homepage.welcome.text == 'Welcome BNE Testers!' }

homepage.close

and when we run them, they’re green as cucumbers:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

Summary

In a very short period of time, we’ve been able to write a fully functional (but not fully featured) testing framework and build on it as necessary. As I mentioned in my talk, so many test frameworks out there are so bloated and complex, sometimes all we need is simple, so if you’re putting test frameworks on pedestals because you find them too complex, start without one and see how you go!

Rspec, page objects and user flows

I love it when people challenge my views.

Recently, Chris McMahon from WMF challenged my view: is Cucumber best for end-to-end testing when RSpec will do? Especially in an environment where non-technical people aren’t involved in writing the specifications.

Admittedly, Cucumber does create some overhead (those pesky step definitions with pesky regular expressions (that I love)), but it does give you plain English (or LOLCAT) written, human readable, executable specifications. But the overhead gives you something more. It gives you a way to create reusable chunks of code that you can call wherever you want when want to test something (or set something up). For example: Given I am logged into Wikipedia. But what do you do if you’re using RSpec?

What is needed is something to sit in between RSpec code and page objects that provide common functionality to reduce repetition of code, such as logging in, in each RSpec specification.

I’ve been trying to come up with a good term to describe these and the best I can come up with is ‘user flows‘, as essentially they are a flow that a user follows throughout the system, spanning multiple pages.

Adding user flows to the Wikimedia custom page object project

I thought I would experiment with user flows in the Wikimedia custom page object project that I recently created. First I replicated my Cucumber features as RSpec specs, which was easy enough, but started to notice a lot of duplication of code.

For example: this was repeated at the beginning of most specs:

visit Wikipedia::LoginPage do |page|
  page.login_with USERNAME, PASSWORD
  page.should be_logged_in
end

Whilst it isn’t a huge amount of repetition (thanks to the useful login_with method on the LoginPage), it’s still not ideal. Enter ruby modules.

Using modules to store pages and flows

As far as I know, modules perform two main functions in ruby: first as a namespace for classes (such as the Wikipedia::LogonPage class), and secondly as a way to group methods that don’t belong in a class, which are often ‘mixed into’ other classes. Perfect! A spot to store our flows.

So, since I already had both a Wikipedia and Commons module, I could simply add module methods to these modules to represent our user flows.

module Wikipedia

  extend PageHelper
  extend RSpec::Expectations
  extend RSpec::Matchers

  def self.ensure_logged_in
    visit Wikipedia::LoginPage do |page|
      page.login_with USERNAME, PASSWORD
      page.should be_logged_in
    end
  end
end

Wiring these up to RSpec

I needed to do a little wiring to ensure these user flows can be easily used in RSpec. In my spec_helper.rb file (which is the equivalent to cucumber.rb in Cucumberland), I added the following to ensure that my browser object I created in RSpec is available to use in the flows:

RSpec.configure do |config|
  config.include PageHelper
  config.before(:each) do
    @browser = browser
    Commons.browser = @browser
    Wikipedia.browser = @browser
  end
  config.after(:suite) { browser.close }
end

and that was all that was needed to start using my user flows in my RSpec specifications.

A completed RSpec specification in my WMF suite looks something like this:

describe 'Editing a Wikipedia Page' do

  context 'Logged on user' do

    it 'should be able to edit own user page' do
      Wikipedia::ensure_logged_in
      content, edit_message = Wikipedia::edit_user_page_by_adding_new_content
      visit Wikipedia::UserPage do |page|
        page.page_content.should include content
        page.view_history
        page.history_content.should include edit_message
        page.history_content.should include Wikipedia::USERNAME
      end

    end

  end

end

where Wikipedia::ensure_logged_in and Wikipedia::edit_user_page_by_adding_new_content are two user flows that I defined in the Wikipedia module.

Summary

I found using page objects directly in RSpec lacking, so I created a concept of user flows in modules that can be easily used from RSpec to reduce repetition and increase readability of specs. If I were to use RSpec for end to end tests, I would find this incredibly useful as a replacement for what Cucumber steps provide.

Roll your own page objects

There seems to be a lot of focus being put into page object ruby gems at the moment. Cheezy has done a fantastic job of the aptly named page-object that supports Watir-Webdriver and Selenium-Webdriver, and then there’s the more recent site_prism (also fantastic) by Nat Ritmeyer that works with Capybara. Before these two came along, I even wrote my own; the now retired watir-page-helper gem.

The premise of these gems is they make it super easy to create page objects for your ruby automated testing projects. But today I want to discuss another crazy idea with you: do you even need a gem at all to do page objects?

Background

I recently refactored some automated tests that Chris McMahon wrote as a potential framework for Wikimedia Foundation (creators of Wikipedia). Chris’s code used Cheezy’s excellent page-object gem so I happily went about my refactoring his code using that gem. Suddenly… I found instead of helping me it started to hinder me. I kept having to refer to page-object user guide I got from Cheezy in Austin to work out how things work. Namely:

  1. How to define elements: as they are different from watir-webdriver (eg. list_item vs li, cell vs td etc.)
  2. How to identify elements: as they are limited to certain supported attributes by element type, unlike watir-webdriver which supports every attribute for all elements
  3. What methods each element provides and what each does: as different elements create different methods with different behaviours, so calling a link element clicks it, whilst a span element returns its text.

The main problem I personally found was that Page-object has essentially created its own DSL to describe elements in page objects, and this DSL is subtly and not so subtly different from the Watir-Webdriver API, so the API I know and love doesn’t work in a lot of places.

An example. There’s a common menu bar on all the Wikimedia sites that displays the logged in user as a link (to the user’s page).

The link is only recognizable by its title attribute, and whilst this is supported by watir-webdriver (it supports any attribute), it is not supported by page-object. The source html looks like:

<a class="new" accesskey="." title="Your user page [ctrl-.]" href="/wiki/User:Alister.scott">Alister.scott</a>

What I would have liked to do was:

  link :logged_in_as, :title => 'Your user page [ctrl-.]'

But, instead, I had to do this (which isn’t very DRY):

  def logged_in_as
    @browser.link(:title => 'Your user page [ctrl-.]').text
  end

I believe essentially what has happened is the page-object, in its neutrality between selenium-webdriver and watir-webdriver, has created its own DSL that is somewhat of a halfway point between the two. This is probably fine for most people starting out, it’s just an API to learn, but for someone like me who has extensive experience with the watir-webdriver API (and loves the power of it), I find it limiting. This is particularly evident when I write a majority of my code using the watir-webdriver API under IRB.

So, I had to take a re-think. Why not roll my own page objects for Wikimedia Foundation?

Roll your own page objects

I recently had a discussion with a colleague/good friend about page objects which went along the lines of “I don’t understand those page object gems because you end up writing a custom page object pattern for each project anyway, as every project/application you work on is different in its own way”. It was one of those aha moments for me.

What I needed was to roll my own Wikimedia page objects.

Taking it back to basics, essentially there are three functions I see a page object pattern provides:

  1. Ability to easily instantiate pages in a consistent manner;
  2. Ability to concisely describe elements on a page, keep it DRY by avoiding repetition of element identifiers (using the underlying driver’s API); and
  3. Ability to provide higher level methods that use the elements to perform user oriented functions.

You can probably notice the helper methods – the magic – that gems like page-object and site_prism provide are missing from my list. This is on purpose, and is because, after lots of thought, I actually don’t find these useful, as they encourage specifications/steps to be too lower level. I would rather a high level method on the page (eg. login) than exposing my username and password fields and a login button.

A Wikimedia Page Model

Taking those things into consideration: this is the page model I came up with for Wikimedia.

Generic Base Page Class

The generic base page class is what everything else extends. It contains the instantiation code common to all pages, and the class methods needed to define elements and methods (more on these later).

Wikimedia Base Page Class

This page class contains elements and methods that are common to all Wikimedia pages. The ‘logged in user’ example above is a good example of something that is the same on every Wikimedia page, whether you’re on Wikipedia, or Wikimedia Commons etc.

Commons & Wikipedia Base Page Classes

These two classes are placeholders for elements and methods are common to a particular site. At the moment with my limited examples, these don’t contain content.

Commons & Wikipedia Page Classes

These are the actual pages that are representations of pages in Wikimedia. These are in separate modules so they are in different namespaces (you can have a Wikipedia::LogonPage and a Commons::LogonPage).

Some example pages:

class Wikipedia::LogoutPage < Wikipedia::BasePage
  page_url "#{Wikipedia::BASE_URL}/w/index.php?title=Special:UserLogout"
  expected_title "Log out - #{Wikipedia::TITLE}"

  element(:main_content_div) { |b| b.div(id: 'mw-content-text' ) }
  value(:main_content) { |p| p.main_content_div.text }
end

Here we can see we define a page_url and expected_title, which are used to instantiate the page.

Next we define an element passing in a block of watir-webdriver code for it, and a value by referencing the element we defined before it. Since these element and value methods execute blocks against self, and the class delegates missing methods to our browser, we can refer to either the browser (shown as b) or the page class (shown as p) in our blocks.

class Commons::LoginPage < Commons::BasePage
  page_url "#{Commons::BASE_URL}/w/index.php?title=Special:UserLogin"
  expected_title "Log in / create account - #{Commons::TITLE}"

  login_elements
  value(:logged_in?) { |p| p.logged_in.exists? }

  def login_with username, password
    username_field.set username
    password_field.set password
    login_button.click
  end
end

In this example, we again define the page_url and expected_title, but we have stored the login_elements with the WikimediaBasePage (as they are the same across all the sites) so we include them by specifying login_elements. We have also defined a login_with method that performs actions on our elements.

There are three available methods to define page elements, values and actions, and these all follow the same format of specifying the method name, and passing in a block of watir-webdriver code.

Calling the page objects from Cucumber step definitions

I chose to use Cucumber for the Wikimedia Foundation framework over Chris’s choice of RSpec as I find it easier to specify end-to-end tests in this way. I find the Cucumber step definitions encourage reuse of steps typically used to set up a test (that are often duplicated in RSpec).

I try to stick to calling the exposed methods, values or actions instead of the elements themselves from my Cucumber steps to ensure I am writing them at a high level. An example step using the page above looks like:

Given /^I am logged into Commons$/ do
  visit Commons::LoginPage do |page|
    page.login_with Commons::USERNAME, Commons::PASSWORD
    page.should be_logged_in
  end
end

The visit and on methods are defined in a Pages module that is mixed into the Cucumber World so these available on all step definitions. As named, the visit method instantiates and visits the page, whereas the on just instantiates it.

module Pages
  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new @browser, visit
    block.call page if block
    page
  end
end

Summary

That’s all there really is the rolling your own page objects. I found this excercise useful as it gives me maximum flexibility and allows me to clearly define pages how I want to define them. I appreciate all the great work that Cheezy and Nat have done on their page object gems, if anything these contain great inspirations on how to roll your own custom page objects most suited to your environment and applications.

You can check out my full code here on Github.