The current state of iOS automated functional testing

I’ve been working on an iOS project and have been looking for a suitable automated functional test library to drive the iOS GUI. I must say that automated iOS testing feels like automated web testing did about 10 years ago: lots of different tools taking different approaches, and all approaches quite flaky!

Through research I came across quite a few different tools that support iOS automated functional testing but need to work out which one is best.

Whilst I often advocate writing functional tests in the same language as your codebase, in the case of iOS and Objective C, I think using a more lightweight language like Ruby has its own advantages (I don’t really like Objective C).

The one thing I really dislike with a lot of these iOS functional test tools is how they are married to Cucumber and encourage users to write tests like “I tap button x” and “I scroll through list y”. These types of tests are much harder to read as they express implementation over intention and should be avoided.

I will also be writing tests for an Android app so being able to use the same tool-set is a great advantage to me.

I also personally prefer an approach where I don’t need to modify the core behavior of my app to run tests against it (designing for testability is completely different and vital). Some approaches embed some sort of server to receive automation commands, which the user must then remove later because it uses undocumented Apple APIs which will lead to app store rejection. Being able to run tests against the app that you submit to the app store means you can be more confident you have tested it right.

Finally, the iOS app I am working on using embedded web views to display dynamic content from the server, which can be interacted with, and therefore it is vital that these can be interacted with. This feature is actually very rare in an iOS automation framework.

Here’s the list of iOS functional automation tools I am aware of and how they stack up:

  • Tool: Frank
    • Language: Ruby
    • Test Framework: Also supports OSX apps
    • Supports other mobile plaforms: Also supports OSX apps
    • Approach: Requires you to embed a symbiote server in your app and uses undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO (only via JavaScript calls)
  • Tool: KIF
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Modifies your app to use undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Subliminal
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Embeds into your project but uses UIAutomation instead of undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Zucchini
    • Language: Custom DSL (CoffeeScript/Ruby)
    • Test Framework: Custom
    • Supports other mobile plaforms:
    • Approach: Generates UIAutomation JavaScript that is executed against your app
    • Deploy same app to store: YES
    • Supports testing web-views?: NO
  • Tool: Calabash
    • Language: Ruby
    • Test Framework: Cucumber
    • Supports other mobile plaforms: Also supports Android Apps
    • Approach: Requires you to embed a server in your app to control it
    • Deploy same app to store: NO
    • Supports testing web-views?: YES
  • Tool: Appium
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: Also supports Mac OSX apps, Android Firefox OS
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES
  • Tool: ios-driver
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: NO
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES

I’ll let you guess which tool I selected until I write my next blog post about how to get started in that tool.

Avoid using case statements in your cucumber/specflow/jbehave step definitions

I quite frequently come across a scenario that looks something like this:

Scenario: Create some animals
  Given I am a zoo keeper
  When I create a giraffe
  And I create a lion
  And I create a pony
  And I create a unicorn
  Then I should have a zoo

and step definitions that implement the When steps with a single step definition:

When /^I create a (\D+)$/ do |animal|
  case animal
    when 'lion'
      create_a_lion()
    when 'giraffe'
      create_a_giraffe()
    when 'pony'
      create_a_pony()
    else
      raise 'Unknown animal'
  end
end

I don’t like having case statements in steps for a number of reasons:

  • For readability and maintainability reasons I try to keep my step definitions as short as possible (usually a couple of lines), and using a case statement violates this principle;
  • Raising an exception to catch invalid usage of the step (in an else clause) replicates what these BDD frameworks already do, provide feedback about unimplemented steps;
  • IDEs that support step auto completion (such as RubyMine & Visual Studio) will not suggest valid steps as they don’t understand how you’ve implemented a case statement; and
  • If used inappropriately (such as our unicorn step), the test will only fail at run-time whereas most IDEs will highlight non-matching steps as you’re writing.

For example, you could change our steps to look something like this:

When /^I create a lion$/ do
  create_a_lion()
end

When /^I create a giraffe$/ do
  create_a_giraffe()
end

When /^I create a pony/ do
  create_a_pony()
end

Even though this is three times as many step definitions, it is actually less code (9 lines compared to 12).

By using this approach it is obvious we can’t currently create a unicorn as RubyMine tells us before we even run our tests. And we don’t need to raise any exceptions myself.

rubymine highlights unimplemented steps

Whilst a lot of people use case statements in steps to reduce the number of steps, it is actually counter intuitive as it means you have more code to do so, and the outcome is less usable when writing scenarios. So, please avoid putting case statements in your step definitions.

Rspec, page objects and user flows

I love it when people challenge my views.

Recently, Chris McMahon from WMF challenged my view: is Cucumber best for end-to-end testing when RSpec will do? Especially in an environment where non-technical people aren’t involved in writing the specifications.

Admittedly, Cucumber does create some overhead (those pesky step definitions with pesky regular expressions (that I love)), but it does give you plain English (or LOLCAT) written, human readable, executable specifications. But the overhead gives you something more. It gives you a way to create reusable chunks of code that you can call wherever you want when want to test something (or set something up). For example: Given I am logged into Wikipedia. But what do you do if you’re using RSpec?

What is needed is something to sit in between RSpec code and page objects that provide common functionality to reduce repetition of code, such as logging in, in each RSpec specification.

I’ve been trying to come up with a good term to describe these and the best I can come up with is ‘user flows‘, as essentially they are a flow that a user follows throughout the system, spanning multiple pages.

Adding user flows to the Wikimedia custom page object project

I thought I would experiment with user flows in the Wikimedia custom page object project that I recently created. First I replicated my Cucumber features as RSpec specs, which was easy enough, but started to notice a lot of duplication of code.

For example: this was repeated at the beginning of most specs:

visit Wikipedia::LoginPage do |page|
  page.login_with USERNAME, PASSWORD
  page.should be_logged_in
end

Whilst it isn’t a huge amount of repetition (thanks to the useful login_with method on the LoginPage), it’s still not ideal. Enter ruby modules.

Using modules to store pages and flows

As far as I know, modules perform two main functions in ruby: first as a namespace for classes (such as the Wikipedia::LogonPage class), and secondly as a way to group methods that don’t belong in a class, which are often ‘mixed into’ other classes. Perfect! A spot to store our flows.

So, since I already had both a Wikipedia and Commons module, I could simply add module methods to these modules to represent our user flows.

module Wikipedia

  extend PageHelper
  extend RSpec::Expectations
  extend RSpec::Matchers

  def self.ensure_logged_in
    visit Wikipedia::LoginPage do |page|
      page.login_with USERNAME, PASSWORD
      page.should be_logged_in
    end
  end
end

Wiring these up to RSpec

I needed to do a little wiring to ensure these user flows can be easily used in RSpec. In my spec_helper.rb file (which is the equivalent to cucumber.rb in Cucumberland), I added the following to ensure that my browser object I created in RSpec is available to use in the flows:

RSpec.configure do |config|
  config.include PageHelper
  config.before(:each) do
    @browser = browser
    Commons.browser = @browser
    Wikipedia.browser = @browser
  end
  config.after(:suite) { browser.close }
end

and that was all that was needed to start using my user flows in my RSpec specifications.

A completed RSpec specification in my WMF suite looks something like this:

describe 'Editing a Wikipedia Page' do

  context 'Logged on user' do

    it 'should be able to edit own user page' do
      Wikipedia::ensure_logged_in
      content, edit_message = Wikipedia::edit_user_page_by_adding_new_content
      visit Wikipedia::UserPage do |page|
        page.page_content.should include content
        page.view_history
        page.history_content.should include edit_message
        page.history_content.should include Wikipedia::USERNAME
      end

    end

  end

end

where Wikipedia::ensure_logged_in and Wikipedia::edit_user_page_by_adding_new_content are two user flows that I defined in the Wikipedia module.

Summary

I found using page objects directly in RSpec lacking, so I created a concept of user flows in modules that can be easily used from RSpec to reduce repetition and increase readability of specs. If I were to use RSpec for end to end tests, I would find this incredibly useful as a replacement for what Cucumber steps provide.

Shoshin: the Sudoku robot

I really enjoyed writing Einstein (my Minesweeper robot) recently. So much that I recently wrote another. Introducing Shoshin: a Sudoku robot.

Shoshin has come along fairly nicely. I pretty much followed the same implementation strategy as I did with Einstein: write failing specs and make them pass until I have a robot that can actually win, or solve in this case, Sudoku. I tend to use large pieces of blank white paper and draw lots of diagrams and logic when I am trying to figure something out (and take a Archimedes break and eat a pink lady apple if I get really stuck – see sticker top right).

Shoshin was written in ruby, so I used both RSpec and Cucumber to write the executable specifications. I tend to write lots of low level specifications in RSpec that run really quickly (40 specs in ~1 second), and then have a handful of high level end to end specifications in Cucumber that I run less frequently, but ultimately specify what I am trying to achieve at a high level. I find the combination works very nicely as I get fast feedback and ultimately know what I am trying to achieve.

To solve some of the more difficult sudoku problems, I printed some strategy diagrams from the web and wrote failing specs for them. It was then a matter of making them pass!

The outcome is Shoshin (see source on github) who can win easy, medium and hard games on websudoku.com. She doesn’t presently win evil games, as they involve guessing/brute force attacks which I haven’t implemented yet. Maybe one day when I get time..

Oh, and Shoshin (初心) is a concept in Zen Buddhism meaning “beginner’s mind”.

Five page object anti-patterns

I’ve observed some page object anti-patterns which commonly arise when starting out designing end-to-end automated tests. Chris McMahon recently asked for some feedback on his initial test spike for Wikipedia, and some of these anti-patterns were present.

Anti-pattern one: frequently opening and closing browsers

I often see both RSpec and Cucumber tests frequently opening and closing browsers. This slows down test execution times, and should be avoided unless absolutely necessary.

You can clear cookies between tests if you’re worried about state.

To open and close the browser only once in Cucumber, specify this in your env.rb file:

browser = Watir::Browser.new

Before do
  @browser = browser
end

at_exit do
  browser.close
end

To open and close the browser only once in RSpec:

browser = Watir::Browser.new

RSpec.configure do |config|
  config.before(:each) { @browser = browser }
  config.after(:suite) { browser.close }
end

Anti-pattern two: hard coding URLs on page classes

Chances are you’ll at some point run your automated tests in different environments, even if it’s just to verify that production has been updated correctly. If you’ve hard coded URLs in page classes, this can be problematic.

Fortunately it’s easy to avoid, by creating a module that contains base URLs which can be accessed by page classes. These base URLs can be stored in YAML files which can be switched for different environments.

module Wikipedia
  BASE_URL = 'https://en.wikipedia.org'
end

class BogusPage
  include PageObject
  page_url "#{Wikipedia::BASE_URL}/wiki/Bogus_page"
end

Anti-pattern three: pages stored as instance variables in steps or rspec specs

I don’t like seeing pages stored as instance variables (those starting with an @) in Cucumber steps or RSpec specs, as it introduces state and thus more room for error.

If you’re using the page-object gem, there are two methods available to access pages directly without using instance variables: visit_page and on_page (also visit or on from 0.6.4+). Both of these can be used as blocks, so you can perform multiple actions within these methods.

visit LoginPage do |page|
  page.login_with('foo', 'badpass')
  page.text.should include "Login error"
  page.text.should include "Secure your account"
end

Anti-pattern four: checking the entire page contains some text somewhere

I often see people checking that the entire web page contains some expected text. Even if the text was at the very bottom of the page hidden in the footer the test would probably pass.

You should check the text is where it should be, using a container that it should belong to. Ideally a span or a div may exist that contains the exact text, but even if it’s in a slightly larger container it is still better than asserting it exists somewhere on the page.

class BogusPage
  include PageObject
  cell :main_text, :class => 'mbox-text'
end

visit_page BogusPage do |page|
  page.main_text.should include 'Wikipedia does not have an article with this exact name'
  page.main_text.should include 'Other reasons this message may be displayed'
end

Anti-pattern five: using RSpec for end-to-end tests

This one is contentious, and I am sure I’ll get lots of opinions to the contrary, but I believe that RSpec is best suited to unit/integration tests, and Cucumber is suited to end-to-end tests.

I find I create duplication when trying to do end-to-end tests in RSpec, which is where Cucumber step definitions come in. Trying to do unit tests in Cucumber seems like too much overhead, and in my opinion is more suited to RSpec.

CukeSalad: not so yummy yummy

I’ve been playing around with CukeSalad: Cucumber specs / tests without step-definitions:

Cucumber, washed and ready to eat for Friction-free ATDD/BDD

In creating an example of my simple WatirMelonCucumber framework using CukeSalad, here’s some of the things I’ve found. Please note: these are my personal observations and are not direct criticisms of the huge effort that has gone into making this tool.

It’s not really no step definitions: they’re just called different things. Instead of writing steps, you’re writing roles and tasks, both of which are mandatory, so I found I actually wrote more code (in more files) than if I just wrote step definitions in the first place.

CukeSalad forces you to use predefined step syntax: which to me defeats the purpose of having features in business language. For example, as far as I can tell, Given statements are purely used for setting the roles, and therefore a business person can’t use it to define a precondition which I would normally use Given statements for. Another example is when you want to capture some data from the test, this must be on the end of a sentence following a semicolon (or comma)

When I search for: the phrase 'Watir'

I spent ages trying to work out why steps don’t match: as you still use Cucumber to run your features, Cucumber will only tell you that there’s no matching steps. This doesn’t help when there’s no steps whatsoever! At some point, I realized that Then steps follow a different syntax, and they can’t use placeholders like Whens, but must have an expected value on the end in single quotes.

Then I should see results greater than '30,000'

It’s very confusing what you’re actually asserting on a Then step: I couldn’t work out a way to capture the expected value from the feature (in the example above ‘30,000’), and CukeSalad just asserts the task returns the actual expected result, which doesn’t help if you’re not testing for equality!

It’s very confusing trying to convert an existing Cucumber test suite: I wouldn’t bother. I found I had to completely rewrite my feature to comply to the syntactical requirements of CukeSalad, so I can’t this working on an existing feature set.

Cucumber world is not supported: so just do a normal include instead in env.rb.

Summary

Sadly, CukeSalad doesn’t offer me anything over using Cucumber. Whilst it sells itself on having no step definitions, I found there was greater effort in writing the roles and tasks. I don’t like the way it forces me to write sentences using somewhat awkward constraints, and how Given, When, & Then statements all behave differently. It certainly gave me a lot of WTF moments, which isn’t what you want when writing automated tests.

I’d be very interested to hear whether anyone has had ongoing success with CukeSalad.

Cukepatch: rich editing of feature files on Github

I’ve been a strong advocate of using the built in Github web text editor for editing feature files for some time, as it means that non-technical business users don’t need to worry about having git clients installed and pulling/pushing changes.

The benefit of this method over a publishing system such as Relish is that you can send a subject matter expert a URL to a Github feature file, and if they recognize that the specification is incorrect, they can immediately update it, unlike Relish where they need to go to the source code and push a change.

The downside is that the editor is a pretty basic, meaning no syntax highligting, step completion etc. Until now that is…

Aslak Hellesøy and Julien Biezemans recently announced Cukepatch: rich editing of feature files on Github. There’s some detail on the Cukes Google Group about it, but essentially it provides rich editing (syntax highlighting/validation and step completion) using a Google Chrome user script that reads a cucumber file you create in your public cucumber repository.

I did this for both WatirMelonCucumber and EtsyWatirWebDriver, and the results look like this:

This looks very promising indeed. There’s a few caveats at the moment including the requirement for a backend server, only working with public repositories, having to manually install the user script and being Google Chrome only. As these are overcome, I can see this becoming the de facto way for business users to write and edit specifications. Well done guys.

WatirMelon Spinach

Hello Spinach!

“Spinach is a new awesome BDD framework that features encapsulation and modularity of your step definitions.”

Seems like a good idea, so I thought I’d give it a try, and convert my existing simple Watirmelon Cucumber tests to use Spinach instead (link to my source code). It was very easy to do, here’s some observations:

  • It’s easy to get existing Cukes running using Spinach, but I imagine if you were starting out using Spinach you’d design things a lot differently
  • Steps belong in their own class, but you can include mixins to reuse steps – the ruby way
  • Steps are in a steps directory under features – shorter than step_definitions
  • Goodbye regular expressions in step definitions, which is a bit of a shame, as you can no longer capture values from the step name
  • As you can’t have regular expressions in step names, I find myself repeating steps that are similar but slightly different, this means my example steps have gone from 41 to 57 lines of code
  • Scenario Outlines aren’t supported at the moment, but I have raised this as a feature request
  • Hooks are dramatically improved, so I found them very easy to use and understand
  • There is no cucumber world, so you do a normal include instead, and env.rb is still supported
  • It displays really cool ticks on the command line when you’re running your spins

Well done to Codegram for releasing Spinach. If anything, it creates some great innovations that I imagine may find their way into Cucumber in the future.

Running Watir-WebDriver tests on Travis CI: a distributed build system

I recently came across Travis CI, a distributed build system that has close links to Github. I’ve seen quite a few projects use it as a CI system, but none that run headless browser tests.

Leveraging off the work I had done recently setting up my own Jenkins server in the cloud to run headless Watir-WebDriver tests, I thought I would have a go at running my WatirMelonCucumber and EtsyWatirWebDriver headless browser tests using Travis CI.

What I didn’t realize is how easy it’d be. The only things I had to do was make sure my Gemfile included rake, and also make sure there was some file existance checking happening for some log files, and it pretty much ran straight away. Wow!

Caveat Emptor

This is pretty new territory, so there’s a few things to watch out for:

  • Every now and, Travis complains about not having Firefox installed. I am not sure why this happens, maybe something to do with the different agents in use;
  • The locale of the build agents seems to be German, so when running my Google tests, the page content is in German, so it fails because it can’t find my expected (English) results; and
  • I can’t seem to capture my test results html file nor screenshot pngs, so it’s a red/green only affair at the moment.

But still, neat distributed free headless CI!

Running your watir-webdriver tests in the cloud, for free!

  • What if you could run unlimited Watir WebDriver tests in the cloud? Check.
  • What if the Watir WebDriver tests would run automatically as soon as you pushed a change to github? Check.
  • What if you would have a full visual history of results with embedded screenshots on failure? Check.
  • What if all of this was free?* Checkmate.

I’ve spent a bit of time over the last week working out how to do this. Here are the basics of what you need to do:

And here’s the detailed instructions.

Set up an Amazon EC2 micro instance running Ubuntu.

  1. First you need to sign up for an Amazon AWS account. This means you’re eligible for a free-tier micro instance for a year.
  2. Once you have an account set up, you need to launch a new instance. I found a free tier eligible Ubuntu image (11.04 Natty 64 bit desktop) and launched that.
  3. You will also want to create an elastic IP and associate it to your instance so that if you reboot your machine, you will have the same IP address. This is done through the AWS console under Elastic IPs.
  4. While you’re here, you’ll want to edit your machine’s security group and open up port 22 for SSH, and 80 for HTTP.
  5. This gives you secure shell (SSH) access to this machine using the provided key, and user ‘ubuntu':
    ssh -i your-key-name.pem ubuntu@your-ip-address
  6. Everything you will do to configure this machine will be through this SSH session, so polish up your unix command line skills!

Set up Jenkins on your machine

There is a useful page for installing Jenkins on Ubuntu.

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo aptitude update
sudo aptitude install jenkins

Make Jenkins available on port 80 so that you don’t need to specify port

Jenkins installs by default on port 80. Ubuntu won’t let applications run on port 80 unless they’re running as root, so it’s best to set up an Apache 2 proxy to port 80 to 8080.

sudo aptitude install apache2
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod vhost_alias
sudo a2dissite default

Then create a file called jenkins in /etc/apache2/sites-available


	ServerAdmin webmaster@localhost
	ServerName ci.company.com
	ServerAlias ci
	ProxyRequests Off
	
		Order deny,allow
		Allow from all
	
	ProxyPreserveHost on
	ProxyPass / http://localhost:8080/

Then run the following commands:

sudo a2ensite jenkins
sudo apache2ctl restart

Password Protect Jenkins

You should go to your Jenkins site (accessible directly at your instance’s IP address through a web browser), and create an account, and then configure the security of Jenkins.

Install Jenkins Plugins

You will need to install the following Jenkins plugins

  • Github: to integrate to Github SCM
  • Rake: to run ruby rake tasks that run Watir-WebDriver tests
  • Green balls: because blue balls are just plain wrong

Install RVM for the Jenkins user

First we’ll need to install git

sudo apt-get install git

Jenkins will need to be able to run ruby, so we’ll install RVM as the Jenkins user.

To run as the jenkins user, we’ll use the sudo command, with the -Hiu arguments to load the home directory and bash profile:

sudo -Hiu jenkins

Once we are user Jenkins, we’ll install RVM using Git.

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Now we need to work out what Ubuntu packages Ruby needs, which is easily done via RVM.

rvm notes

which gives me something like

For Ruby (MRI, Rubinius, & REE)  you should install the following OS dependencies:
/usr/bin/apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake

So, we can log-out as the Jenkins user (control-D) and install the following as ubuntu

sudo apt-get install build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake

Once we’ve done this, we’ll want to run the following as the Jenkins user (
sudo -Hiu jenkins) to install Ruby 1.9.2.

rvm pkg install zlib
rvm install 1.9.2 --with-zlib-dir=$rvm_path/usr

Running headless Watir-WebDriver tests

I choose a desktop version of Ubuntu, so it’ll already have Firefox installed, but if you don’t, you can install it by:

sudo apt-get install firefox

To run our Watir WebDriver tests headlessly using the headless gem, we’ll need xvfb

sudo apt-get install xvfb

Configuring Jenkins to run tests via Rake

You add a new build in Jenkins where you can specify the github repository location.

As we’ve installed the rake plugin, we can configure a new Jenkins project to use an RVM ruby install (in my case ruby-1.9.2-p290@watirmelon-cucumber).

I simply set up a default task in rake, which runs all my cucumber tests. This generates a results.html file which is captured as an artifact, and also creates and captures junit xml results, which are used to show test summary information.I also capture any file created under the ‘screenshots’ directory.

Summary and Outcome

I have set up both my WatirMelonCucumber and EtsyWatirWebDriver projects on jenkins.watirmelon.com.

My Jenkins Dashboard looks something like this:

Please feel to leave a comment below and let me know what you think.

* Free for one year using an free tier EC2 micro instance