Category Archives: Automated Testing

A ruby testing framework, from scratch, in 15 minutes

As part of my talk last week at the Brisbane Testers Meetup, I gave a live demo (no pre-recorded or pre-written code) of writing a ruby testing framework from scratch in 15 minutes. The idea was to show that most testing frameworks contain so much functionality ‘you ain’t gonna need’, so why not try writing one from scratch and see how we go? It was also a chance to show the testers who hadn’t done automated testing that programming/automated testing is not rocket science.

Since I promised to talk about selenium, I used watir-webdriver, but I would have preferred to just show testing a simple app/class that I would have written from scratch in ruby.

Our testing problem

I wrote a beautifully simple website to welcome the testers to the first ever Brisbane Testers Meetup, and wanted to write some tests to make sure it worked. The site is accessible at data:text/html,<h1 id=”welcome”>Welcome BNE Testers!</h1> and looks something like this:

bnetesterswelcome

First I’ll give you a few moments to get over how amazing that web site is… that’s long enough, now, what we need is a couple of tests for it:

  1. Make sure the welcome message exists
  2. Make sure the welcome message is visible
  3. Make sure the welcome message content is correct

Iteration zero

Do the simplest thing that could possibly work. In our case print out the three things we want to check to the screen and we’ll manually verify them.

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
puts b.h1(id: 'welcome').exists?
puts b.h1(id: 'welcome').visible?
puts b.h1(id: 'welcome').text

which outputs:

true
true
Welcome BNE Testers!

A good start but not quite a testing framework.

Iteration One

I think it’s time to introduce a method to assert a value is true.

I like to start by writing how I want my tests to look before I write any ‘implementation’ code:

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

I usually run the my tests to give me a ‘clue’ to what I need to do next. In our case:

 undefined method `assert' for main:Object (NoMethodError)

In our case, it’s simple, we need to write an assert method. Luckily we know exactly what we need: a method that takes a description string and a block of code that should execute returning true, otherwise we have an error. We can simply write this method above our existing tests:

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}"
		else
			puts "Assertion FAILED for #{message}"
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'"
	end
end

which gives us this output when we run:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

This is awesome, but it makes me nervous that all of our three tests passed the first time we ran them. Perhaps we hard coded them to pass? Will they ever fail?

There’s an old saying, source unknown, which is ‘never trust a test you didn’t first see fail‘. Let’s apply this here by making all our tests fail. I usually do this by changing the source system, that way you can keep the integrity of your tests intact.

This is fairly easy to do in our case by changing the id of our welcome element.

data:text/html,<h1 id=”hello”>Welcome BNE Testers!</h1>

When we do so, all our tests fail: yipee.

Assertion FAILED for that the welcome message exists
Assertion FAILED for that the welcome message is visible with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'
Assertion FAILED for that the welcome message text is correct with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'

We change it back and they pass again: double yipee.

Iteration Two –

So far all the text output has been in same color, and everyone knows a good test framework uses color. Lucky I know a gem that does color output easily, all we do is:

require 'colorize'

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}".green
		else
			puts "Assertion FAILED for #{message}".red
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'".red
	end
end

which gives us some pretty output:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

and a fail now looks like this:

Assertion FAILED for that the welcome message exists

Sweet.

We can put the assert method in its own file which leaves our test file cleaner and easier to read:

require 'watir-webdriver'
require './assertions.rb'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

Iteration Three

Our final iteration involves making the tests even easier to read by abstracting away the browser. This is typically done using ‘page objects’ and again we’ll write how we would like it to look before implementing that functionality:

require 'watir-webdriver'
require './assertions.rb'

Homepage.visit

assert('that the welcome message exists') { Homepage.welcome.exists? }
assert('that the welcome message is visible') { Homepage.welcome.visible? }
assert('that the welcome message text is correct') { Homepage.welcome.text == 'Welcome BNE Testers!' }

HomePage.close

When we run this, it provides us a hint at what we need to do:

uninitialized constant Homepage (NameError)

We need to create a HomePage class with three methods: visit, welcome and close.

We can simply add this to our tests file to get it working:

class Homepage
	def initialize
		@browser = Watir::Browser.new
		@browser.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
	end

	def self.visit
		new
	end

	def welcome
		@browser.h1(id: 'welcome')
	end

	def close
		@browser.close
	end
end

After we’re confident it is working okay, we simply move it to a file named homepage.rb and our resulting tests look a lot neater:

require './assertions.rb'
require './homepage.rb'

homepage = Homepage.visit

assert('that the welcome message exists') { homepage.welcome.exists? }
assert('that the welcome message is visible') { homepage.welcome.visible? }
assert('that the welcome message text is correct') { homepage.welcome.text == 'Welcome BNE Testers!' }

homepage.close

and when we run them, they’re green as cucumbers:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

Summary

In a very short period of time, we’ve been able to write a fully functional (but not fully featured) testing framework and build on it as necessary. As I mentioned in my talk, so many test frameworks out there are so bloated and complex, sometimes all we need is simple, so if you’re putting test frameworks on pedestals because you find them too complex, start without one and see how you go!

Answer ‘Will it work?’ over ‘Does it work?’

Software teams must continually answer two key questions to ensure they deliver a quality product:

  1. Are we building the correct thing?
  2. Are we building the thing correctly?

In recent times, I’ve noticed a seismic shift of a tester’s role on an agile software team from testing that the team is building the thing correctly to helping the team build the correct thing. That thing can be a user story, a product or even an entire company.

As Trish Khoo recently wrote:

“The more effort I put into testing the product conceptually at the start of the process, the less I effort I had to put into manually testing the product at the end”

It’s more valuable for a tester to answer ‘will it work?‘ at the start than ‘does it work?‘ at the end. This is because if you can determine something isn’t the correct something before development is started, you’ll save the development, testing and rework needed to actually build what is needed (or not needed).

But how do we know it actually does work if we’re focused on will it work? How do we know that we’re building the thing correctly? The answer is automated tests.

Automated tests, written by programmers, alongside testers, during the engineering process validate the software does what it’s meant to do correctly. Behavior driven approaches assist to translate acceptance criteria directly into automated tests.

So, how can a tester be involved to make sure a team is building the correct thing?

  • get involved in writing the acceptance criteria for every story;
  • ensure a kick off for each story happens so the programmer(s) understand(s) what is expected and any edge cases or queries are discussed;
  • work with the programmer(s) to automate tests based upon the acceptance criteria;
  • ensure a handover/walk-through happens as soon as a story is finished in development to ensure that all the acceptance criteria are met and tests have been written;
  • showcase the finished product every iteration to the business.

You’ll soon find you can provide much greater value as a tester determining whether something will work and then working alongside the development team to ensure it works as it is developed.

You probably don’t need a specification framework

I think plain language specification frameworks like SpecFlow and Cucumber are great, but have a lot of overhead and are way overused.

If you don’t have non-technical folk collaborating with you on your specifications, try writing plain automated tests instead. This means using plain NUnit/MSTest over SpecFlow in C# or minitest over Cucumber in Ruby. You’ll avoid the overhead of maintaining a plain language specification framework and be able to focus on developing a great set of tests instead.

It’s easier than you think to add a plain language specification layer to a set of well structured plain tests. So only add the specification layer when you need it, because chances are you ain’t gonna.

Take control of your own career

During my career, I’ve come across numerous testing colleagues with no experience in automated testing who say things like “I’d love to do automated testing”. They expect to be put into an automated testing role so they can learn automated testing.

I don’t think it should work like that. Your employer shouldn’t be solely responsible for you enhancing your skills and progressing your career.

And, the thing is, it’s never been easier to pick up some new technical skills.

If you want to learn programming start by learning something like Ruby. If you want to learn about automated web testing learn Watir. If you want to learn about behavior driven development tools learn Cucumber.

I taught myself Ruby. I taught myself Watir. I taught myself C#, Python, Selenium, Cucumber and Jenkins. The list goes on.

The barrier to entry has never been lower. Try codeacademy, try ruby koans, download the free watir book, buy Cheezy’s cheap eBook about Watir & Cucumber.

So, instead of watching television or going out for drinks, spend your nights and weekends learning some new skills and taking control of your career instead of expecting your employer to hand it to you on a plate.

You’ll then be able to say “I’m learning all about Watir at the moment and I would love to apply that on a project” instead of “I’d love to do automated testing”.

UI automation of vendor delivered products always leads to trouble

wise_old_elfI’ve got three darling boys, and they love this show on ABC4Kids called ‘Ben & Holly’s Little Kingdom’. It’s a cartoon from the makers of Peppa Pig about tiny elves and fairies and there’s a character named the wise old elf (pictured) who doesn’t like the fairy magic and whose catchphrase is ‘magic always leads to trouble‘ which has become a little bit of a meme in our household where we replace the word ‘magic’ with something else that’s perilous. This leads me to the point of this article, something I have a strong opinion about:

“UI automation of vendor delivered products always leads to trouble”

Why do I believe that? To be successful in UI automation involves some critical elements which are missing when writing automated tests against a black box vendor delivered products (such as a customized CRM solution). To be successful in UI automation:

  • you need collaboration between testers and developers to write the code needed to write robust and efficient user interface tests;
  • you need opportunities to include testability into the user interface, whether this be test specific navigation controllers, or ensuring that page elements are appropriately identified and structured; and
  • you need to be able to identify areas which can be tested below the UI, whether this be through APIs and web services, or hitting the database directly. Vendors seldom provide services and almost never allow direct access to the database – particularly if it’s a SaaS product.

I strive to advise anyone that it’s a bad idea to write automated tests against the UI of a vendor delivered product. Either that vendor should be doing their own automated testing, or be providing a more robust way to automatically test that changes have been correctly applied to their product.

I also try to avoid any career opportunities that put me in a situation where this is required of me because I don’t believe in it and haven’t seen it been successfully done.

As the wise old elf says: “UI automation of vendor delivered products always leads to trouble”.

Using appium in Ruby for iOS automated functional testing

As previously explained, I recently started on an iOS project and have spent a bit of time comparing iOS automation tools and chose Appium as the superior tool.

The things I really like about Appium is that it is language/framework agnostic as it uses the WebDriver standard WIRE protocol, it doesn’t require any modifications to your app, supports testing web views (also known as hybrid apps) and it supports Android since we are concurrently developing an Android application (it also supports OSX and Firefox OS but we aren’t developing for those, yet). There isn’t another iOS automated testing tool, that I know of, that ticks that many boxes for me.

Getting Started

The first thing to do is download the appium.app package from the appium website. I had an issue with the latest version (0.11.2) launching the server which can be resolved by opening the preferences and checking “Override existing sessions”.

You run the server from inside the appium.app which takes your commands and relays them to the iOS simulator. There’s also a very neat ‘inspector’ tool which shows you all the information you need to know about your app and how to identify elements.

Note: there’s currently a problem with XCode 5.0.1 (the latest version as I write) which means Instruments/UIAutomation won’t work at all. You’ll need to downgrade (uninstall/reinstall) to XCode 5.0 to get appium to work at all.

Two Ruby Approaches

This confused me a little to start, but there’s actually two vastly different ways to use appium in ruby.

1) Use the standard selenium-webdriver gem

If you’re used to using WebDriver, like me, this will be the most straightforward approach (this is the approach I have taken). Appium extends the API to add different gestures by calling execute_script from the driver, so all other commands stay the same (for example, find_element).

2) Use the appium_lib library

There is a Ruby gem appium_lib that has a different API to the selenium-webdriver gem to control appium. I don’t see any massive benefits to this approach besides having an API that is more specific to app testing.

Using Selenium-WebDriver to start appium in ruby

Launching an appium app is as simple as defining some capabilities with a path to your .app file you have generated using XCode (this gets put into a deep folder so you can write the location to a file and read it from that file).

capabilities = {
'browserName' => 'iOS',
'platform' => 'Mac',
'version' => '6.1',
'app' => appPath
}
driver = Selenium::WebDriver.for :remote,
desired_capabilities: capabilities,
url: "http://127.0.0.1:4723/wd/hub"

Locating elements

Once you’ve launched your app, you’ll be able to use the appium inspector to see element attributes you can use in appium. Name is a common attribute, and if you find that it’s not being shown, you can add a property AccessibilityIdentifier in your Objective C view code which will flow throw to appium. This makes for much more robust tests than relying on labels or xpath expressions.

driver.find_element(:name, "ourMap").displayed?

Enabling location services for appium testing

This got me stuck for a while as there’s quite a bit of conflicting information about appium on how to handle the location services dialog. Whilst you should be able to interact with it as a normal dialog in the latest version of appium, I would rather not see it at all, so I wrote a method to copy a plist file with location services enabled in it to the simulator at the beginning of the test run. It’s quite simple (you can manually copy the clients.plist after manually enabling location services):

def copy_location_services_authentication_to_sim
source = "#{File.expand_path(File.dirname(__FILE__))}/clients.plist"
destination = "#{File.expand_path('~')}/Library/Application Support/iPhone Simulator/7.0/Library/Caches/locationd"
FileUtils.cp_r(source, destination, :remove_destination => true)
end

Waiting during appium tests

This is exactly the same as selenium-webdriver. There’s an implicit wait, or you can explicitly wait like such:

driver.manage.timeouts.implicit_wait = 10
wait = Selenium::WebDriver::Wait.new :timeout => 30
wait.until {driver.find_element(:name, 'monkeys').displayed? }

Mobile gestures

The obvious difference between a desktop web browser and a mobile app is gestures. Appium adds gestures to WebDriver using execute_script. I recommend using the percentage method (0.5 etc) instead of pixel method as it is more resilient to UI change.

For example:

driver.execute_script 'mobile: tap', :x => 0.5, :y => 0.5

or

b = driver.find_element :name, 'Sign In'
driver.execute_script 'mobile: tap', :element => b.ref

Testing Embedded Web Views

The native and web views seamlessly combine so you can use the same find_element method to find either. The appium.app inspector displays the appropriate attributes.

Note: I can’t seem to be able to execute a gesture (eg. swipe) over a Web View. I don’t know whether this is a bug or a limitation of Appium.

Summary

I have found that using the familiar selenium-webdriver gem with appium has been very powerful and efficient. Being able to open an interactive prompt (pry or irb) and explore your app using the selenium-webdriver library and the appium.app inspector is very powerful as you can script on the fly. Whilst appium still seems relatively immature, it seems a very promising approach to iOS automation.

Now to get watir-webdriver to work with appium.

The current state of iOS automated functional testing

I’ve been working on an iOS project and have been looking for a suitable automated functional test library to drive the iOS GUI. I must say that automated iOS testing feels like automated web testing did about 10 years ago: lots of different tools taking different approaches, and all approaches quite flaky!

Through research I came across quite a few different tools that support iOS automated functional testing but need to work out which one is best.

Whilst I often advocate writing functional tests in the same language as your codebase, in the case of iOS and Objective C, I think using a more lightweight language like Ruby has its own advantages (I don’t really like Objective C).

The one thing I really dislike with a lot of these iOS functional test tools is how they are married to Cucumber and encourage users to write tests like “I tap button x” and “I scroll through list y”. These types of tests are much harder to read as they express implementation over intention and should be avoided.

I will also be writing tests for an Android app so being able to use the same tool-set is a great advantage to me.

I also personally prefer an approach where I don’t need to modify the core behavior of my app to run tests against it (designing for testability is completely different and vital). Some approaches embed some sort of server to receive automation commands, which the user must then remove later because it uses undocumented Apple APIs which will lead to app store rejection. Being able to run tests against the app that you submit to the app store means you can be more confident you have tested it right.

Finally, the iOS app I am working on using embedded web views to display dynamic content from the server, which can be interacted with, and therefore it is vital that these can be interacted with. This feature is actually very rare in an iOS automation framework.

Here’s the list of iOS functional automation tools I am aware of and how they stack up:

  • Tool: Frank
    • Language: Ruby
    • Test Framework: Also supports OSX apps
    • Supports other mobile plaforms: Also supports OSX apps
    • Approach: Requires you to embed a symbiote server in your app and uses undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO (only via JavaScript calls)
  • Tool: KIF
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Modifies your app to use undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Subliminal
    • Language: Objective C
    • Test Framework: OCUnit/SenTest
    • Supports other mobile plaforms: NO
    • Approach: Embeds into your project but uses UIAutomation instead of undocumented APIs
    • Deploy same app to store: NO
    • Supports testing web-views?: NO
  • Tool: Zucchini
    • Language: Custom DSL (CoffeeScript/Ruby)
    • Test Framework: Custom
    • Supports other mobile plaforms:
    • Approach: Generates UIAutomation JavaScript that is executed against your app
    • Deploy same app to store: YES
    • Supports testing web-views?: NO
  • Tool: Calabash
    • Language: Ruby
    • Test Framework: Cucumber
    • Supports other mobile plaforms: Also supports Android Apps
    • Approach: Requires you to embed a server in your app to control it
    • Deploy same app to store: NO
    • Supports testing web-views?: YES
  • Tool: Appium
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: Also supports Mac OSX apps, Android Firefox OS
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES
  • Tool: ios-driver
    • Language: Ruby, C#, Java, JavaScript, Objective C, PHP, Python, Perl, Clojure
    • Test Framework: Agnostic
    • Supports other mobile plaforms: NO
    • Approach: Uses Instruments to control app using the standard WebDriver WIRE protocol
    • Deploy same app to store: YES
    • Supports testing web-views?: YES

I’ll let you guess which tool I selected until I write my next blog post about how to get started in that tool.

Minesweeper in Melbourne

I did a presentation on Einstein the Minesweeper robot at an ANZTB SIGIST in Melbourne this morning. The facilitator of the session was ill so I was asked to host the event as well, so I felt very much under pressure.

I came up with an idea for a quick activity to “awaken our senses” as it was early in the morning where I asked everyone to count how many human senses they are aware of. Most people gave the traditional response of five (or six), and I challenged their views by presenting some additional senses we humans possess. Check out the Wikipedia article if you’re interested to know more.

Unfortunately the first presentation on Context Driven Testing by Mark Richards went well over the allocated time, so I had to rush my presentation to ensure I could catch my 11am flight home: I made it, just.

Being for a group for testers, my intention was to not focus on the detail of the actual Minesweeper robot but what solving a complex problem such as automating Minesweeper can teach use about testing and how to be better testers.

Click the slide below to view the slideshow.

Melbourne SIGIST Einstein_ The Minesweeper Robot

Avoid using case statements in your cucumber/specflow/jbehave step definitions

I quite frequently come across a scenario that looks something like this:

Scenario: Create some animals
  Given I am a zoo keeper
  When I create a giraffe
  And I create a lion
  And I create a pony
  And I create a unicorn
  Then I should have a zoo

and step definitions that implement the When steps with a single step definition:

When /^I create a (\D+)$/ do |animal|
  case animal
    when 'lion'
      create_a_lion()
    when 'giraffe'
      create_a_giraffe()
    when 'pony'
      create_a_pony()
    else
      raise 'Unknown animal'
  end
end

I don’t like having case statements in steps for a number of reasons:

  • For readability and maintainability reasons I try to keep my step definitions as short as possible (usually a couple of lines), and using a case statement violates this principle;
  • Raising an exception to catch invalid usage of the step (in an else clause) replicates what these BDD frameworks already do, provide feedback about unimplemented steps;
  • IDEs that support step auto completion (such as RubyMine & Visual Studio) will not suggest valid steps as they don’t understand how you’ve implemented a case statement; and
  • If used inappropriately (such as our unicorn step), the test will only fail at run-time whereas most IDEs will highlight non-matching steps as you’re writing.

For example, you could change our steps to look something like this:

When /^I create a lion$/ do
  create_a_lion()
end

When /^I create a giraffe$/ do
  create_a_giraffe()
end

When /^I create a pony/ do
  create_a_pony()
end

Even though this is three times as many step definitions, it is actually less code (9 lines compared to 12).

By using this approach it is obvious we can’t currently create a unicorn as RubyMine tells us before we even run our tests. And we don’t need to raise any exceptions myself.

rubymine highlights unimplemented steps

Whilst a lot of people use case statements in steps to reduce the number of steps, it is actually counter intuitive as it means you have more code to do so, and the outcome is less usable when writing scenarios. So, please avoid putting case statements in your step definitions.

Internationalization and Localization Testing

Internationalization vs Localization

Internationalization is the process of making a software application easily adaptable to be used by international audiences. Localization is the process of adding a new locale to a software application.

Internationalization is most often implemented by ensuring that labels and values on screens are not ‘hard-coded’, they are read from a common source to ensure they can be easily switched when running in a different locale. Localization is adding a new source, or locale, so that a new audience can use your application.

I have found the most effective way to test that an application is internationalized is to create a new locale specifically for testing purposes. Your new locale has a specific known formula for translation which you use to check that you application is fully internationalized.

You should then also run all your automated acceptance tests in your new locale to ensure that all new functionality is internationalized as it developed.

I’d recommend either of the following approaches to define a new locale for testing:

Approach Description Welcome’ Translated Pros Cons
Lorem Ipsum A defined, fixed length string Lorem ipsum dolor sit amet’
Can see how short elements overflow
Easy to test
Screens don’t look as realistic
Screens can be hard to understand
Reversal Reversal of the same string emocleW Screens look realistic
Easy to test
Can’t see overflow effects

I prefer the reversal approach as it’s the most realistic representation, and it’s easy to do some exploratory testing on a different ‘real’ locale to spot overflow/formatting issues.

Say you have a screen that looks something like this in English:

English Screen

This same screen in Lorem Ipsum would look something like:

Lorem Screen

And it would look like this in Reverse:

Reverse English Screen

Manually testing internationalization

This is easy. Run your app in whatever test locale you have decided to create. Then ensure that each screen has localized data for each label.

Automatically testing internationalization

You can fairly easy ensure your automated acceptance tests can run against your test locale. You just need to ensure that all labels which are asserted can be translated in your acceptance tests.

For example:

Given I am an anonymous user
When I visit the personal details screen
Then I should see the heading “Please enter your details”
And I should see the label “Name:”
And I should see the label “Email:”
And I should see the label “Phone:”
And I should see the button with label “Continue”

Each of these steps has a specific label that needs to be translated. Instead of asserting the actual value, in your automated test code you first translate the string to your test locale, and then assert that value. Your automated tests will then fail if the actual string appears on the screen untranslated.