Specification by Example: a love story

I attended a three day Advanced Agile Acceptance Testing/Specification by Example workshop in Melbourne last week run by Gojko Adzic, which I enjoyed immensely.

I took copious amounts of notes so I could regurgitate them here, but instead I decided to do something a bit different and write a story. It’s a story about how a team can apply what he taught us to improve their software development practices. It’s entirely fictitious (although I’d love to live in the Byron Bay hinterland!) and any resemblance to any real person or project is purely coincidental.

So here it is: Specification by Example: a Love Story (PDF). Enjoy.

Your automated acceptance tests needn’t be written in the same language as your system being tested

When selecting which tool to implement for business facing automated acceptance tests, I’ve often heard that it must use the same programming language as the system under test. Whilst this can sometimes work well, often it is better to choose a tool in another language that will ulimately deliver you better success as it can be adopted more passionately. Here are some of the reasons on why I think so.

The simplicity of some other languages mean it’s easier for less technical testers to understand them

Let me show you an example that compares the same test written in C# as in Ruby.

C# WebDriver Example – Inspired by David Burns

using OpenQA.Selenium;
using OpenQA.Selenium.IE;
 
using NUnit.Framework;
 
namespace Selenium.Two.DotNetExample
{
    [TestFixture]
    public class Test_Google
    {
	    IWebDriver driver;
	     
	    [SetUp]
	    public void Setup()
	    {
		    driver = new InternetExplorerDriver();
	    }
	    
	    [TearDown]
	    public void Teardown()
	    {
		    driver.Quit();
	    }
    		
        [Test]
        public void TestSearchGoogleForTheAutomatedTester()
	    {
	        //Navigate to the site
            driver.Navigate().GoToUrl("http://www.google.com");
            //Find the Element and create an object so we can use it
            IWebElement queryBox = driver.FindElement(By.Name("q"));
            //Work with the Element that's on the page
            queryBox.SendKeys("Watir");
			queryBox.SendKeys(Keys.ArrowDown);
            queryBox.Submit();
            //Check that the Title is what we are expecting
            Assert.True(driver.Title.IndexOf("Watir - Google Search") > -1);
        }
    }
}

Ruby Watir Example – written by me

require 'watir-webdriver' #watir, celerity, firewatir
require "test/unit"
class GoogleTest < Test::Unit::TestCase
  def test_google
    b =Watir::Browser.start "www.google.com"
    b.text_field(:name => "q").set "watir"
    b.button(:name, "btnG").click
    assert_equal("Watir - Google Search", b.title)
    b.close
  end
end

I am slightly biased, but I think the ruby example is neater and easier to read, especially to someone with a less technical background. Sure, a C# developer might baulk at this premise, but for most people who don’t know C#, I imagine it’s the case.

Often times developers like learning another programming language to add to their experience
When I’ve been working on automated tests in ruby, I’ve often seen a strong interest in developers wanting to learn ruby and the automated testing framework, as it’s common for developers to specialize in a language (Java or .NET) and then focus solely on that. Don’t get me wrong, this isn’t always the case, but it happens more than you think.

Testers can embrace simpler programming languages such as ruby, and can become the “experts” in that domain

I’ve seen testers pick up a programming language like ruby and polish their skills in it, which helps their morale by becoming experts in it. If the automated acceptance tests were written in the same language as the developers, yes this brings benefits, but the testers might not become experts as quickly, and may lack ownership of the tests.

Automated acceptance test tools provide different mileage across languages
Some languages have particularly good tools for defining business facing automated tests, for example, Cucumber’s support for ruby is superb, whereas support for C# equivalents such as Cuke4Nuke and SpecFlow isn’t quite as good. RSpec is another example of a tool that is awesome in its domain.

Summary

I know this post has ended up sounding like a pro-ruby rant, but it is :) I am trying to point out that there’s simpler, easier languages for testers to pick up than your Javas and C-Sharps. The ruby and python programming languages are two great examples, and as I mentioned, it seems that ruby is quickly becoming the testing language of choice, because of its simplicity, and the wide variety of testing related gems available. Ruby also makes it extremely easy to bootstrap configurations across different environments, and its lack of licensing requirements or need for an IDE make it an excellent choice for testers.

One of the reasons that the Selenium project is so successful is that it supports various programming languages (and browsers), which appeals to the majority of developers who can use their established skills, but it’s common to meet testers who are passionate about Watir, because of its simplicity and focus: a tool for testers by testers. It is for this reason I believe Ruby & Watir & Cucumber is a winning combination, and will bring about many success stories for agile testing teams in the future.

Rubular & RubyMine: makes Cucumber easier

I’ve spent the last couple of months establishing Acceptance Test Driven Development on a medium sized software delivery project using Watir & Cucumber.

Cucumber is premised on reading feature files and matching strings to determine what to do, which is done by using regular expressions in step files. This means constant use of ruby regular expressions, and in polishing up on my regular expression skills, I have found rubular incredidly useful. I love the simple layout and the ‘Regex Quick Reference’ at the bottom, just where you need it. It’s very well done.

Rubular: A Ruby regular expression editor and tester

Rubular: A Ruby regular expression editor and tester

The other tool I have found incredibly useful is RubyMine. Up until now, I’ve never really bothered with an IDE for Watir stuff. I’ve mainly stuck to text editors before, but since I have started using Cucumber, I have found the RubyMine tool almost critical, as its support for Cucumber feature files and step definitions is superb. It features click through linking for feature files so you know exactly what step you’re calling, and the debugger is awesome; no more puts statements for me :)

So, if you’re thinking about implementing Cucumber, or using Cucumber but are annoyed with the lack of efficiency in managing a large suite of step definitions, I would thoroughly recommend these two tools. Rubular is free, and I believe RubyMine is about $99, but less if you need multiple licenses. There’s also an Early Access Program where you can use RubyMine for free as long as you’re happy to test it along the way.

Agile software, continuous delivery and passionate users

Rowly Emmett, a test consultant who lives here in Queensland, Australia, recently dismissed agile software development in a blog post titled “Agile Recipe“.

“…there is no difference between Agile and Waterfall”

“The thing is, if people followed the (rigourous software methodology or waterfall) process correctly, then they wouldn’t need to try out Agile.”

Rowly says it’s not really about your software development approach, but how rigorous you are in your chosen approach. One of the things that Rowly wrote that stood out to me was:

“I believe the process should adapt to the conditions of the project and the needs of the application.”

But that’s what agile is about isn’t it? Adapting and evolving solutions through collaboration.

Rowly finishes by listing some of the conditions for a project moving to agile, which included:

“Is the domain ABSOLUTELY clear (does the customer really know what they want?)”

But I’d actually say that’s one of the reasons I have seen agile work over other methods, in that customer needs can evolve iteratively, rather than having to be specified upfront which is more likely to produce something not needed at the end.

I would say my biggest criticism of Rowly’s article, and why I disagree with it, is he’s looking at it from a pure methodoligical viewpoint, rather than what it does differently and what that can ultimately deliver. Sure, a project will fail if it’s done poorly, no matter what methodology is used. An agile project will probably fail sooner, but if done correctly can ultimately deliver better outcomes. Not just that, it’s increasingly anti-competitive to use non-agile methods to deliver software. So it’s no longer a choice, but a mandate. I’ll explain in a bit more detail using a case study.

Google Docs and how they deliver software

I have no idea what methodology Google uses to develop its Google Docs product, but I know they must use iterative agile techniques. How do I know this? Because I am a passionate Google Docs user and I subscribe to the Google Docs blog. Every couple of days Google Docs releases some new functionality into Google Docs, and they write about it on their blog. They couldn’t do this using a waterfall methodology. They’d release new functionality every few months, or years. Compared to Microsoft who bring out a new product with lots of bundled enhancements every couple of years (Office 2003, Office 2007, Office 2010), Google release some enhancements every couple of days.

How does this make me feel as a user? Passionate. I love that there’s new functionality every couple of days, as it allows me to master it and kick ass. As I recently tweeted:

 

Which is exactly how you want your users to feel. As Kathy Sierra explained during her great Business of Software presentation: you don’t just need users who think you’re company is awesome, or your product is awesome, but rather that they’re awesome, when they’re using your product. Like when I showed the expenses app I built with Google Spreadsheets and forms to my wife, and because she was so impressed I felt awesome. Like all the tweets I saw about the Gmail Priority Inbox, something that was delivered immediately, iteratively, not through an ‘upgrade’.

If Google Docs was developed using a rigorous software methodology, it may have only just been released, or it may have not even been released yet. And I certainly wouldn’t have had an opportunity to get excited about all the incremental improvements I have seen over the past years.

Moving beyond the Project into Continuous Delivery

The Google Docs case study highlights a bigger point, that Google Docs is a product, and not a project. As Evan Bottcher (a Thoughtworker from Melbourne) recently wrote on his aptly titled blog post: Projects are evil and must be destroyed, we need to move beyond looking at things as a project that are eventually handed-over to BAU, and move towards “form(ing) long-lived teams around applications/products, or sets of features”. Like the Google Docs team, who continually develop and deliver functionality to Google Docs users (and write about it). There’s a name for this concept, it’s continuous delivery, and a fellow Thoughtworker Jez Humble recently published co-published (with Dave Farley) a book about it, titled Continuous Delivery.

My bugbear with Continuous Delivery

The only downside I see to continuous delivery is when it’s used in an environment that needs to be actively upgraded by users; it’s no point pushing out new functionality daily if your users have to do an upgrade daily. One place I see this happen prevalently is iPhone apps, how many apps need updating every time you check? Too many in my opinion. In a web environment, such as Google Docs, continuously delivery rocks! In a non-web environment, such as iPhone apps, browser, firmware and OS updates, it sucks. As Michael Neale recently pointed out on Twitter.

 

Firefox are doing some work in this space around passively delivering (minor) updates, but there’s still a lot of work to do in this space if we’re going to have continous delivery of non web applications.

Wrapping Things Up

I started off by disagreeing with Rowly in his views on agile software methodology in that it doesn’t matter. I believe we no longer have a choice, we’re at a point where it is increasingly uncompetitive not to be agile, as we need to deliver soon and the way to do this is through iterative and incremental development and continuous delivery. This is even more important for environments where the domain is unclear and the customer doesn’t know what they want. Delivering software this way is also great for creating passionate users who feel they can master what you give them and kick ass.

My thoughts on Brisbane ANZTB SIGIST: September 2010

I went along to the ANZTB SIGIST last night, at the Hilton here in Brisbane. It was probably the best one I’ve been to, both in attendance, and caliber of presenters. There were five presenters all up, which is a lot to squeeze into two hours including drinks and conversation. It’s no surprise then that the last presentation by Craig Aspinall was a little bit rushed, which is a shame because I would have liked to ask more questions than time allowed (apparently the Hilton would kick us out if we stayed longer).

Craig Smith and Rene Masten from Suncorp began with an excellent presentation on Agile Testing. Craig was awesome in both presenting his extensive knowledge (he’s a coach) and his slides were also very cool (minimal text and no bullets!).

Craig’s main theme was about how to make testing cool: ensuring people say “that’s cool” when you tell them about your testing. He talked about what makes up an agile team, the techical divide between devs and testers, ATDD, ensuring you deliver, and how there is currently a great opportunity for testers (who want to be bothered). Rene followed by talking about organizational change, training and coaching, communicating what you’re doing (internally and externally) and building quality in.

Craig finished with a great motivating style of emphasizing it’s all about passion and craft, “who’s awesome!”, and not being afraid of technical challenges.

The other presenters were Ben Sullivan and Brent Acworth, both also from Suncorp, who gave a demo of BDD using JBehave and Hudson. I enjoyed the talk but there wasn’t much I hadn’t seen before or didn’t know.

The final presentation was by Craig Aspinall and was unfortunately squeezed into a small time slot. It was about what Craig dubs “Automated Black Blob Testing”, the rationale being testing is not black box as a box has a predefined shape, it’s more of a blob.

Craig’s approach looked solid, although I was slightly concerned when he mentioned the project being a SaaS solution and how much effort was being put into automated testing.  I’m not criticizing what Craig did tehnically, I am just concerned about the prevalent practice of the onus of testing SaaS solutions being put onto the customer. I believe if you buy a SaaS, you should get a working SaaS, minimal, if any at all, testing required. But that’s just me. Otherwise, a great talk, besides Craig using Java when Ruby and Watir would have done the trick. ;)

A great ANZTB SIGIST, and hopefully more good ones to come!

Craig Smith’s Slides

Some of my photos

This slideshow requires JavaScript.


Upcoming agile testing meetup in Brisbane, Australia

I just got this email from Thoughtworks, and have registered to attend. See you there if you’re in Brisbane.

You are invited to the launch meeting of the Brisbane chapter of the Agile Alliance Australia (AAA). Last year’s inaugural Agile Australia conference established the AAA and it is envisaged that local chapters will provide ongoing education and knowledge-sharing.

ThoughtWorks Global Testing Practice Lead Kristan Vingrys will be our guest speaker discussing the Changing Role of a Tester. Kristan has over 10 years experience in software testing encompassing a wide range of testing practices for both products and applications within a diverse range of industries. He has presented on agile testing and articles – including a chapter in the ThoughtWorks Anthology Book. He has been involved in solution delivery, coaching and mentoring teams with a focus on testing, agile processes and how the two compliment each other.

The Changing Role of a Tester

IT development is getting faster business now expects applications in months not years. Testing has to keep up and can not afford to be seen as a bottleneck. Traditional ways of testing no longer work with the rapid software development methodologies being used; there is no time to do big up-front test case design or multiple cycle test executions. Testers have to become more agile and embrace change while still providing quality information about the application being developed.

Testing on an agile project is different to a waterfall project; it is not just about doing waterfall in smaller iterations. Nor is it focused on finding as many defects as possible, instead the goal is to work as a team delivering quality working software that satisfies customer need. There are some new skills that testers will need to learn, but they do not need to throw away everything they already know. Changing the mindset about how, when and why of testing will help a tester adapt their existing skills to become an invaluable resource on any agile team.

When: 5:30pm, Thursday 13th May
Where: See the meetup site

After the discussion, we will go locally for drinks, food and networking for those interested, location TBA

Please register at the Brisbane Chapter meetup.com site: Brisbane Chapter – First Meeting

Look forward to seeing you there!

Update 19 July 2010:

Link to a great write up by Craig Smith is here, plus slides are here (pdf).