Do you need an automated acceptance testing framework?

Posted on by
Reply

This post is part of the Pride & Paradev series.

Do you need an automated acceptance testing framework?

Yes, you need an automated acceptance testing framework

If you’re starting off with automated acceptance testing and you don’t have some kind of framework, eg, page object models, in place then you can quickly develop a mess. If your automated acceptance tests are being written by various people on your project, then having a framework in place that people can follow will make for the most consistent approach.

There are certain operations that you can abstract to a base page class to ensure consistency across pages, and you can write helper methods for automated test drivers so that the same functionality is being repeated across your code base.

Without some kind of framework in place you’re likely to have various approaches implemented which will eventually cause a maintenance overhead as your automated test suite expands.

No, you don’t need an automated acceptance testing framework

There’s an old saying in extreme programming: YAGNI: you ain’t gonna need it, which means a programmer shouldn’t add functionality until absolutely necessary.

An automated acceptance testing framework violates this principle, there is a strong risk of developing functionality in your framework which you ain’t gonna need.

Over-engineered automated acceptance test frameworks are harmful for a team as they dictate certain ways of doing things which means the team can be less efficient in developing what they need to deliver.

Developing a framework before any functionality is delivered is particularly inefficient, as it is not until you start using a framework you will understand what you require it to do and what it shouldn’t do.

Pair programming on the automated acceptance tests can ensure a consistent approach is taken to development and knowledge across functional areas is shared.

Don’t bury your hooks

Posted on by
Reply

A slightly technical post here.

If you’re using a BDD framework such as Specflow or JBehave, can I please ask that you don’t bury your hooks. These frameworks provide hooks, or events, you can use to repeat certain things. An example in Specflow is:

[BeforeScenario]
public static void GoToHomePage()
{
Driver.Navigate().GoToUrl(GoogleUrl);
}

You can put these hooks in any steps class, but please, put them in one place, all together, preferably in a class/file named hooks or the like.

I recently came across a code base where these hooks where spread across lots of different steps files, which makes it very confusing and hard to debug as you don’t know where all this code is being called from when you’re running your tests.

Should you involve real users in testing?

Posted on by
Reply

This post is part of the Pride & Paradev series.

Should you involve real users in testing?

Yes, involve real users in testing

Unless you involve real users you risk releasing something into production that is not user friendly. User testing doesn’t need to be expensive; you can conduct it in house with a focus on simplicity by reading Steve Krugg’s excellent how-to guide Rocket Surgery Made Easy.

The caveat is that you need to conduct user testing as earlier as possible: there is no point doing user testing if it doesn’t result in useful change, and the earlier you do it the more likely you are able to change.

You don’t even need to have high-fidelity screens to conduct user testing: low fidelity screens are often enough to get critical usability and concept feedback from real users.

No, don’t involve real users in testing

The problem with involving real users in testing is the risk that your organization will use this as an opportunity to listen to what users want.

Listening to what users want is dangerous. This is because users think of things in incremental rather than revolutionary terms. Users don’t know how to ask for something they’ve never conceived of. Listening to feedback from users is going to yield incremental improvements (‘make that button green’), but this does in no way correlate to releasing something that users actually want.

In my pre-MP3 university days, I had a very large collection of audio CDs. I worked hard in a casual job and saved up and bought a Pioneer 25 CD stacker: it was amazing, I could listen to 25 CDs on shuffle! If I was asked what could have made listening to music better I would have said a 100 CD stacker: I could store four times as many CDs! A few years later Apple released a new thing called an iPod, a small pocket sized device capable of holding 1000 albums. If Apple had listened to users like me they would have built a faster, larger, better CD player. Instead they designed something I had no capability of thinking could even exist, I was thinking incrementally, Apple were thinking revolutionary.

You are better off understanding what users motivations are and then building something to satisfy that than involving them in user testing.

Fixing bugs in production: is it that expensive any more?

Posted on by
6

You’ve most likely seen a variant of this chart before:

bug fix costs

I hadn’t seen it for a while, until yesterday, but it’s an old favourite of test managers/test consultants to justify a lot of testing before releasing to production.

But I question whether it’s that accurate anymore.

Sure, in the good old days of having a production release once or twice a year it cost a large order of magnitude more to fix a bug in production, but does it really cost that much more in the present age of continuous delivery/continuous deployment where we release into production every fortnight/week/day?

If the timeline on the chart above is a year then of course bugs will cost more to fix, because presumably, if the project took a year to start with, you don’t have a very rapid software development process. And there’s more likely to be requirements ‘bugs’ in production because an awful lot happened in the year that the requirement was being developed. Hence along came agile with its smaller iterations and frequent releases.

Mission critical systems aside, most web or other software applications we build today can be easily repaired.

Big waterfall projects, like building a plane, are bound to fail. The Boeing 787 Dreamliner was an epic fail. Not only was it five delays and many years late, it had two major lithium ion battery faults in its first 52,000 hours of flying which caused months of grounding and has no doubt affected future sales, causing millions of dollars in damages. But it seems to have been well tested:

“To evaluate the effect of cell venting resulting from an internal short circuit, Boeing performed testing that involved puncturing a cell with a nail to induce an internal short circuit. This test resulted in cell venting with smoke but no fire. In addition, to assess the likelihood of occurrence of cell venting, Boeing acquired information from other companies about their experience using similar lithium-ion battery cells. On the basis of this information, Boeing assessed that the likelihood of occurrence of cell venting would be about one in 10 million flight hours.”

NTSB Interim Report DCA13IA037 pp.32-33

After months of grounding, retesting, and completely redesigning the battery system, the cause of the original battery failures are still unknown. If they can’t work out what the problem is after it has occured twice in production, it’s not likely it could have been found or resolved in initial testing.

But most of us don’t work on such mission critical systems anyway.

And production fixes can be very easy.

Take this very different example; I provide support for a production script that uploads a bunch of files to a server. There was a recent issue where a file-name had an apostrophe in it which meant this file was skipped when it should have been uploaded.

Upon finding out about the problem I immediately looked at my unit tests. Did I have a unit test with a file name with an apostrophe? No I didn’t. I wrote a quick unit test – it failed: as expected. I made a quick change to the regular expression constant that matches file names to include an apostrophe, I reran the unit test which passed. Yippee. I quickly reran all the other unit and integration tests and all passed, meaning I could confidently package and release the script. All of this was done in a few minutes.

I could have possibly prevented this happening by doing more thorough testing to begin with, but I am pretty sure that would have taken more effort than it did for me to fix the production bug, by writing a test for it and repackaging it. So for me it wasn’t an increase in cost whatsoever to find that bug ‘late’.

Unless you’re working on mission critical software, shipping some bugs into production is almost always better than shipping no software at all. If you work on very small, frequent deployments into production, the cost of fixing bugs once they have gone live will only be marginally greater than trying to find every bug before you ship. The longer your spend making sure your requirements are 100% correct and everything is 100% tested, ironically, your software is more likely to be out of date, and hence incorrect, once you finally go live.

Stop Firefox auto-updating (and breaking your CI build)

Posted on by
3

If you run regular automated WebDriver tests using Firefox then chances are you’ve encountered the problem where Firefox updates itself and your WebDriver bindings don’t support that version which causes all your tests to fail. It’s really annoying, so I suggest you set your Firefox install to never automatically update as below to avoid this happening.

Stop Firefox auto updating

Update:

As Alex points out below, you can do this programmatically for a server you don’t actually launch Firefox on, ie. a headless machine.

profile = Selenium::WebDriver::Firefox::Profile.new
 # disable autoupdate
 profile['app.update.auto'] = false
 profile['app.update.enabled'] = false

Leanpub pricing analysis

Posted on by
1

I am nearly at the point of publishing my eBook on Leanpub which has raised the question about how much I will price it at (it will be available free online in HTML form). There are two price points on Leanpub (minimum and suggested) and since I don’t go into these decisions lightly I thought I would do some analysis around existing titles to see what they set the price-points at.

One caveat is that I completely understand that it’s just as important to have a good product as worrying about its pricing as pricing alone doesn’t make a book good.

I wrote a quick watir-webdriver script (at the end of this post and online) to scrape the top 100 all time grossing and best-selling (volume) books off Leanpub and some data about each. One data point I would have loved to have seen on each book page was page count so I could correlate page count to price, but sadly it’s missing.

I put this into a google spreadsheet and did some analysis

Leanpub pricing analysis

What I can read about Leanpub pricing is:

  • Obviously, higher selling books are more likely to be available for free
  • The number of books with the same minimum and suggested prices are the roughly the same across the lists at about one quarter
  • The average non-free minimum price is slightly less for higher selling books, but roughly $12-$14
  • The average non-free suggested price is much higher for higher grossing books at roughly $19-$22, compared to $12-$14 for higher selling books

I did some further analysis of the raw data and a particular title stood out to me: ‘How to Do What You Love & Earn What You’re Worth as a Programmer‘. What made this stand out was it’s ranking at number 2 on number of copies sold (8191 copies), but just number 28 on gross revenue. Based upon the minimum price of $9.99, this translates into minimum revenue of $81,828.09. But this book is at number 28 on the earnings list, meaning that possibly 27 other books each earned more than $81,828.09. But there’s no a single book that has earned more than this book that has a minimum price times copies (or suggested price times copies for that matter), that is higher than this book. Which is very odd.

I know of two possible explanations, but I don’t know if either is true.

One is that this book once was free, or a lot cheaper than it’s current minimum price, which explains why it has sold many copies but (comparatively) grossed so little.

The second is that people who buy books on Leanpub pay a lot more than minimum or even suggested prices.

I think the first explanation is true but hope that it’s the second.

Here’s my script:

$: << File.dirname(__FILE__)+'/lib'

require 'book_list'
require 'watir-webdriver'

profile = Selenium::WebDriver::Firefox::Profile.new
profile['permissions.default.image'] = 2 #no images
browser = Watir::Browser.new :firefox, :profile => profile

book_list = BookList.new 'bestsellers', 'https://leanpub.com/most_copies_lifetime'

browser.goto book_list.url
browser.links(:class => 'book-link').each_with_index do |link, index|
  book_list.add_book link.text, link.href, index + 1
end

book_list.books.each do |book|
  browser.goto book.url
  book.min_price = browser.element(:css => 'span[itemprop="lowPrice"]').text
  book.suggested_price =  browser.element(:css => 'span[itemprop="highPrice"]').text
  if browser.strong(:text => /^This book has (\d+) readers!$/).exists?
    book.copies_sold = browser.strong(:text => /^This book has (\d+) readers!$/).text.match(/^This book has (\d+) readers!$/).captures.first
  end
end

book_list.to_csv 'copies.csv'

browser.close

require 'book'

class BookList

  attr_reader :name, :books, :url

  def initialize name, url
    @name = name
    @url = url
    @books = []
  end

  def add_book name, url, rank
    @books << Book.new(name, url, rank)
  end

  def to_csv filename
    File.open(filename, 'w') do |file|
      @books.each do |book|
        file.puts book.to_csv_string
      end
    end
  end
end

class Book

  attr_reader :name, :url, :rank
  attr_accessor :min_price, :suggested_price, :copies_sold

  def initialize name, url, rank
    @name = name
    @url = url
    @rank = rank
  end

  def to_csv_string
    "#{name.gsub(",","")},#{url},#{rank},#{min_price},#{suggested_price},#{copies_sold},"
  end
end

Make your employees productive

Posted on by
Reply

I’ve stated this many times before, but I can’t emphasize enough how important it is to do anything you can to make your employees productive.

This recent article explains how important this is at Facebook, to the extent that they have two vending machines on every floor where employees can self-serve themselves to common equipments such as keyboards, mice, cables and headphones using their employee badge (items are free although prices are displayed to encourage staff to question whether they really need $200 headphones). This was because they found 20% of help desk calls were for simple equipment requests.Facebook also makes sure staff don’t spend unnecessary time fixing issues with their laptops:

Mr. Campos said that some companies try to squeeze every last penny out of laptops that aren’t working properly and employees bear the burden of that approach. The cost of a laptop, $1,500 to $2,000, is minor in comparison to hampering the productivity of an employee who makes $100,000 per year, he said.

~ Rachael King – Wall Street Journal – on Facebook

 

So if you have staff who work in software development, make it as easy as possible to be as efficient as possible by providing everyone with:

  • local admin rights to computers so employees don’t waste time contacting the help desk to install necessary development tools;
  • unfiltered high speed Internet, free of restrictive proxies, so staff can access the material they need to do their job quickly;
  • reliable and efficient hardware including solid state drives for every staff member;
  • choice of browser: locking machines down to use an old version of IE is like torture for most staff who use a modern browser like Chrome at home;
  • up to date software: employees resent having to use Windows XP at work when they’re using Windows 8 or Mac OS X at home;
  • a clean, consistent place to work eg. don’t make employees hotdesk (and if you do – provide enough clean workstations to avoid employees spending time ‘hunting’ for workstations); and
  • like-for-like remote access: to make it easy to work whenever you want to.

Some of these seem almost ridiculous to mention, as without them your staff can’t be productive at all, but the number of workplaces I have seen that fail to meet any of the above provisions is crazy. I can’t count how long I have spent on the phone to a corporate help desk to get trivial things done, spent copying software from personal USB keys to workstations because the proxy blocked it, hunting for a free workstation or dealing with reliability issues of hardware all because employers have failed to take any of these things into account.