Running Automated Tests with A/B Testing

Like a lot of modern, data driven sites, WordPress.com uses A/B testing extensively to introduce new features. These tests may be as simple as a label change or as complex as changing the entire sign up flow, for example by offering a free trial.

Since I have been working on a set of automated end-to-end tests for WordPress.com, I have found A/B testing to be problematic for automated testing on this very fast moving codebase, namely:

  1. Automated tests need to be deterministic: having a randomised experiment as an A/B test means the first test run may get an entirely different sign up flow than a second test run which is very hard to automate; and
  2. Automated tests need to know which experiments are running otherwise they may encounter unexpected behaviour randomly.

What we need is two methods to deal with A/B tests when running automated tests:

  1. We need to be able to see which A/B tests are active and compare this to a known list of expected A/B tests – so that we don’t suddenly encounter some unexpected/random behaviour for some of our test runs
  2. We need to be able to set the desired behaviour to the control group so that are our tests are deterministic.

Different sites conduct A/B testing using different tools and approaches, WordPress.com uses HTML5 local storage to set which A/B tests are active and which group the user belongs to.

Luckily it’s easy to read and update local storage using WebDriver and JavaScript. This means our approach is to:

  1. Each time a page object is initialised, there is a call on the base page model that checks the A/B tests that are active using something like return window.localStorage.ABTests; and then compares this to the known list of A/B tests which are checked in as a config item. This fails the test if there’s a new A/B test introduced that isn’t in the list of known tests. This is better than not knowing about the A/B test and failing based upon some non-deterministic behaviour.
  2. When a new A/B test is introduced and we wish to ensure our automated tests always use the control group, we can set this using a similar method window.localStorage.setItem('ABTests','{"flow":"default"}'); and refresh the page.

Ideally it would be good to know and plan every A/B test for our automated e2e tests, but since this isn’t possible, checking against known A/B tests and ensuring control groups are set means our automated tests are at least more consistent and deterministic, and fail a lot faster and more consistently when a new A/B test has been introduced.

How do you deal with non-determinism with A/B tests?

ES2015 Page Classes

As mentioned yesterday, I am updating my WebDriverJs and Mocha demo to use new ES2015 features.

ES2015 supports classes more elegantly than using prototype based ones in older versions of JavaScript.

For example, this old class:

var webdriver = require('selenium-webdriver');
var until = webdriver.until;
var config = require('config');

RalphSaysPage = function RalphSaysPage(driver, visit) {
	this.driver = driver;
	this.url = config.get('url');
	this.explicitWaitMS = config.get('explicitWaitMS');
	this.quoteSelector = webdriver.By.id('quote');
	if (visit === true) {
		this.driver.get(this.url);
	}
	this.driver.wait(until.elementLocated(this.quoteSelector), this.explicitWaitMS);
};

RalphSaysPage.prototype.quoteContainerPresent = function() {
	var d = webdriver.promise.defer();
	this.driver.isElementPresent(this.quoteSelector).then(function(present) {
		d.fulfill(present);
	});
	return d.promise;
};

RalphSaysPage.prototype.quoteTextDisplayed = function() {
	var d = webdriver.promise.defer();
	this.driver.findElement(this.quoteSelector).getText().then(function(text) {
		d.fulfill(text);
	});
	return d.promise;
};

module.exports = RalphSaysPage;

can be better written as a ES2015 class:

import webdriver from 'selenium-webdriver';
import config from 'config';

export default class RalphSaysPage {
	constructor( driver, visit = false ) {
		this.driver = driver;
		this.url = config.get('ralphURL');
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.quoteSelector = webdriver.By.id('quote');

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.quoteSelector), this.explicitWaitMS);
	}
	quoteContainerPresent() {
		return this.driver.isElementPresent(this.quoteSelector);
	}
	quoteTextDisplayed() {
		return this.driver.findElement(this.quoteSelector).getText();
	}
}

You can see I have also simplified the functions quoteContainerPresent() and quoteTextDisplayed() to directly return the webDriver promise instead of creating our own which is unnecessary.

When I introduce another page class:

import webdriver from 'selenium-webdriver';
import config from 'config';

const by = webdriver.By;
const until = webdriver.until;

export default class WebDriverJsDemoPage {
	constructor( driver, visit = false ) {
		this.driver = driver;
		this.url = config.get('demoURL');
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.expectedElementSelector = by.id('elementappearsparent');

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.expectedElementSelector), this.explicitWaitMS);
	}
	waitForChildElementToAppear() {
		return this.driver.wait(until.elementLocated(by.id('elementappearschild')), this.explicitWaitMS, 'Could not locate the child element within the time specified');
	}
	childElementPresent() {
		return this.driver.isElementPresent(by.id('elementappearschild'));
	}
}

you can see I have duplicated some common functionality such as the navigation and page waiting across these two classes. This is where we can an ES2015 parent (or base) page class to inherit from.

Our base class might look something like:

export default class BasePage {
	constructor( driver, expectedElementSelector, visit = false, url = null ) {
		this.explicitWaitMS = config.get('explicitWaitMS');
		this.driver = driver;
		this.expectedElementSelector = expectedElementSelector;
		this.url = url;

		if (visit) this.driver.get(this.url);

		this.driver.wait(webdriver.until.elementLocated(this.expectedElementSelector), this.explicitWaitMS);
	}
}

which means our page classes are much nicer now:

export default class RalphSaysPage extends BasePage {
	constructor( driver, visit = false ) {
		const quoteSelector = webdriver.By.id('quote');
		super(driver, quoteSelector, visit, config.get('ralphURL'));
		this.quoteSelector = quoteSelector;
	}
	quoteContainerPresent() {
		return this.driver.isElementPresent(this.quoteSelector);
	}
	quoteTextDisplayed() {
		return this.driver.findElement(this.quoteSelector).getText();
	}
}

export default class WebDriverJsDemoPage extends BasePage {
	constructor( driver, visit = false ) {
		super(driver, by.id('elementappearsparent'), visit, config.get('demoURL'));
	}
	waitForChildElementToAppear() {
		return this.driver.wait(until.elementLocated(by.id('elementappearschild')), this.explicitWaitMS, 'Could not locate the child element within the time specified');
	}
	childElementPresent() {
		return this.driver.isElementPresent(by.id('elementappearschild'));
	}
}

This gives us the ability to add any common functionality across pages (such as checking the page title) quickly and easily without duplication.

 

 

 

WebDriverJs & Mocha in ES2015

A friend of mine, Mark Ryall, recently created a fork of my WebDriverJs and Mocha example project and updated it to use ES2015. I’ve made some further changes and merged these in, and would like to share these.

Background

JavaScript is an implementation of the ECMAScript scripting language standard.

The latest version of ECMAScript, known as ES2015, ES6, ES6 Harmony, ECMAScript 2015, or ECMAScript 6, has some neat features which are handy to use for our WebDriverJs & Mocha tests I have previously written about.

It seems that there will be yearly releases of the ECMAScript standard from 2015 onwards, and the most common way to refer to these will be as ES2015, ES2016 etc.

Enabling ES2015 Support for our Example Tests

There is a node tool called Babel which is a JavaScript compiler that allows you to use new ECMAScript features and compile these into JavaScript. This requires two node packages which we add to our package.json file:

"babel-core": "^6.3.13",
"babel-preset-es2015": "^6.3.13"

This means we have a babel compiler and a babel library to transform ES2015.

The second thing we need to do is add a plugin to actually tell babel to transform ES2015.

We add a .babelrc file to our project with the following content:

{
"presets": ["es2015"]
}

Running our Specs using Babel

Once we’ve done this, we can use Mocha and WebDriverJs with ES2015. Instead of calling mocha specs we now need to use babel like:
mocha --compilers js:babel-core/register specs.

This isn’t as nice, so we can update our package.json file so our test command is set to the longer babel command, and we just need to call npm test to run our Mocha specs.

Updating our code to use ES2015

The great thing about ES2015 is it is backwards compatible, so we don’t need to update all our code at once, we can made gradual changes to use new features available to us.

Mark made changes to the spec and the page object to use some of the pretty ES2015 features:

Import Statements

This:

var assert = require('assert');
var webdriver = require('selenium-webdriver');
var test = require('selenium-webdriver/testing');
var config = require('config');
var RalphSaysPage = require('../lib/ralph-says-page.js');

Becomes:

import assert from 'assert';
import webdriver from 'selenium-webdriver';
import test from 'selenium-webdriver/testing';
import config from 'config';
import { ralphSays } from '../lib/pages.js';

Using let instead of var

let is block scoped so this is better to use.

This:

var driver;

Becomes:

let driver = null;

Arrow functions

The arrow functions make the clean up hooks simpler to read:

From this:

test.afterEach(function() {
  driver.manage().deleteAllCookies();
});

To this:

test.afterEach(() => driver.manage().deleteAllCookies());

Summary

Moving to use ES2015 wasn’t as daunting as I initially thought as once you add support for it using Babel, you can gradually start using the new features.

Comparison of JavaScript browser automation and test specification libraries

As part of my trial for my current role at Automattic, I was tasked with implementing some e2e acceptance tests using my choice of library/framework/language.

I very much recommend writing automated acceptance tests in the same language as your app, even though I have described some benefits of using a different language, and since WordPress is moving towards JavaScript from PHP, JavaScript seems the most suitable language for Automattic.

Continue reading

GTAC 2015 Day Two Highlights & Summary

Today’s conference began with some rather funny commentary shared by Yvette Nameth’s mother from yesterday’s talks. I was mentioned as the ‘flaky’ guy:

My main takeaway from the entire conference is that it seems we get way too caught up on complex solutions for our testing. We need to keep asking ourselves: “what’s the simplest thing that could possibly work?” If we have complex systems why do we need complex tests? We need to take each large complex problem we work on and break it down till we get something small and manageable and solve that problem. Rinse and repeat.

Continue reading