Category Archives: ATDD

Avoid using case statements in your cucumber/specflow/jbehave step definitions

I quite frequently come across a scenario that looks something like this:

Scenario: Create some animals
  Given I am a zoo keeper
  When I create a giraffe
  And I create a lion
  And I create a pony
  And I create a unicorn
  Then I should have a zoo

and step definitions that implement the When steps with a single step definition:

When /^I create a (\D+)$/ do |animal|
  case animal
    when 'lion'
      create_a_lion()
    when 'giraffe'
      create_a_giraffe()
    when 'pony'
      create_a_pony()
    else
      raise 'Unknown animal'
  end
end

I don’t like having case statements in steps for a number of reasons:

  • For readability and maintainability reasons I try to keep my step definitions as short as possible (usually a couple of lines), and using a case statement violates this principle;
  • Raising an exception to catch invalid usage of the step (in an else clause) replicates what these BDD frameworks already do, provide feedback about unimplemented steps;
  • IDEs that support step auto completion (such as RubyMine & Visual Studio) will not suggest valid steps as they don’t understand how you’ve implemented a case statement; and
  • If used inappropriately (such as our unicorn step), the test will only fail at run-time whereas most IDEs will highlight non-matching steps as you’re writing.

For example, you could change our steps to look something like this:

When /^I create a lion$/ do
  create_a_lion()
end

When /^I create a giraffe$/ do
  create_a_giraffe()
end

When /^I create a pony/ do
  create_a_pony()
end

Even though this is three times as many step definitions, it is actually less code (9 lines compared to 12).

By using this approach it is obvious we can’t currently create a unicorn as RubyMine tells us before we even run our tests. And we don’t need to raise any exceptions myself.

rubymine highlights unimplemented steps

Whilst a lot of people use case statements in steps to reduce the number of steps, it is actually counter intuitive as it means you have more code to do so, and the outcome is less usable when writing scenarios. So, please avoid putting case statements in your step definitions.

A tale of three ruby automated testing APIs (redux)

Redux Note: I originally wrote a similar article to this before going on parental leave about six weeks ago. Whilst I didn’t intend to offend, it seemed that a few people took my article the wrong way. I understand that a lot of effort goes into creating a web testing API, but that doesn’t mean that everyone will agree with what you’ve made.

Sadly, an anonymous coward attacked myself and the company who I work (even though I don’t mention that company on this blog), so for the first time in this blog’s history, I have had to turn comment moderation on. I am sorry to the other genuine commenters whose comments have been lost in transition, and now have to wait for their new comments to be approved.

Since then I have received numerous emails asking where my article went, and commenting that people found it interesting and worthwhile. So I have decided to repost this article, hopefully with a little less contention this time around, making it clear, this is my opinion and experience: YMMV.

Intro

As a consultant I get to see and work on a lot of automated testing solutions using different automated web testing APIs. Lately I’ve been thinking about how these APIs are different and what makes them so.

My main interest is in ruby, and fortunately ruby has three solid examples of three different kinds of web testing APIs, two of which extend the lowest level API: selenium-webdriver.

I’ll (try to) explain here what I consider to be three kinds of automated web testing APIs and where I consider the sweet spot to be and and why.

A meaty example

As a carnivore, I thought I would explain my concept in terms I can relate to. If you’re a beef eater, there are many different kinds of beef that you can use to make some tasty food to eat. I’ll use three different kinds of beef for my example. The first (rawest) kind would involve getting a beef carcass and filleting it yourself to eventually make some edible food. The second kind of beef you could use is beef that is already in a slightly usable form, but you can then use yourself to make some edible food. For example, you can buy minced beef at a butcher, and then make your own hamburger patties, taco fillings etc from it. The final type of beef you could use is beef that has already been prepared so you can directly consume it, for example, sausages which can be cooked and consumed as is.

I consider these three examples of different kinds of beef to roughly correlate to automated web testing APIs, of which I also consider to be three kinds of.

The first is a Web Driver API, which is the rawest form of an API, its job is to drive a browser by issuing it commands. It provides a high level of user control, but like filleting a beef carcass it’s more ‘work’. An example in ruby of this API is the selenium-webdriver API, which controls the browser using the webdriver drivers.

The second kind of automated web testing API is the Browser API, which is a higher level API but still provides user control. This is the minced beef of APIs, as whilst it’s in a more usable form than a carcass, you still have a lot of control (and potential to what you can do with it). An example in ruby of this API is the watir-webdriver API, which uses the underlying selenium-webdriver carcass to control the browser.

The final kind of automated web testing API is the Web Form DSL (Domain Specific Language) which is a very high level API that provides users with specific methods to automate web forms and their elements. This is the beef sausages of APIs as sometimes you feel like eating something else besides sausages, but it’s difficult to make anything else edible but sausages from sausages. An example in ruby of this Web Form DSL is the Capybara DSL.

Visually, this looks something like this:

Show me the code™

So exactly what do these APIs look like?

I knew you’d ask, that’s why I came prepared.

Say I want to accomplish a fairly basic scenario on my example Google Doc form:

  • Start a browser
  • Navigate to the watir-webdriver-demo form
  • Check whether text field with id ‘entry_0′ exists (this should exist)
  • Check whether text field with id ‘entry_99′ exists (this shouldn’t exist)
  • Set a text field with id ‘entry_0′ to ’1′
  • Set a text field with id ‘entry_0′ to ’2′
  • Select ‘Ruby’ from select list with id ‘entry_1′
  • Click the Submit button

This is how I would do it in the three different APIs:

# * Start browser
# * Navigate to watir-webdriver-demo form
# * Check whether text field with id 'entry_0' exists
# * Check whether text field with id 'entry_99' exists
# * Set text field with id 'entry_0' to '1'
# * Set text field with id 'entry_0' to '2'
# * Select 'Ruby' from select list with id 'entry_1'
# * Click the Submit button

require 'bench'

benchmark 'selenium-webdriver' do
  require 'selenium-webdriver'

  driver = Selenium::WebDriver.for :firefox
  driver.navigate.to 'http://bit.ly/watir-webdriver-demo'
  begin
    driver.find_element(:id, 'entry_0')
  rescue Selenium::WebDriver::Error::NoSuchElementError
    # doesn't exist
  end
  begin
    driver.find_element(:id, 'entry_99').displayed?
  rescue Selenium::WebDriver::Error::NoSuchElementError
    # doesn't exist
  end
  driver.find_element(:id, 'entry_0').clear
  driver.find_element(:id, 'entry_0').send_keys '1'
  driver.find_element(:id, 'entry_0').clear
  driver.find_element(:id, 'entry_0').send_keys '2'
  driver.find_element(:id, 'entry_1').find_element(:tag_name => 'option', :value => 'Ruby').click
  driver.find_element(:name, 'submit').click
  driver.quit
end

benchmark 'watir-webdriver' do
  require 'watir-webdriver'
  b = Watir::Browser.start 'bit.ly/watir-webdriver-demo', :firefox
  b.text_field(:id => 'entry_0').exists?
  b.text_field(:id => 'entry_99').exists?
  b.text_field(:id => 'entry_0').set '1'
  b.text_field(:id => 'entry_0').set '2'
  b.select_list(:id => 'entry_1').select 'Ruby'
  b.button(:name => 'submit').click
  b.close
end

benchmark 'capybara' do
  require 'capybara'
  session = Capybara::Session.new(:selenium)
  session.visit('http://bit.ly/watir-webdriver-demo')
  session.has_field?('entry_0') # => true
  session.has_no_field?('entry_99') # => true
  session.fill_in('entry_0', :with => '1')
  session.fill_in('entry_0', :with => '2')
  session.select('Ruby', :from => 'entry_1')
  session.click_button 'Submit'
  session.driver.quit
end

run 10

This is how long they took for me to run:

                        user     system      total        real
selenium-webdriver  1.810000   0.840000  22.130000 ( 73.123340)
watir-webdriver     1.940000   0.870000  24.380000 ( 79.388494)
capybara            1.950000   0.890000  24.080000 ( 79.920051)

Note: Capybara doesn’t always require a ‘session’, it’s only for non ruby rack applications, but since my example (Google) is not a rack application, as are most of the applications I test, my example must use the session.

When using ruby, why Watir-WebDriver is my sweet spot

I personally find Watir-WebDriver to be the most elegant solution, as the API is high enough for me to be highly readable/usable, but low enough to be powerful and for me to feel like I’m in control.

For example, being able to select an element by a explicit identifier (name, class name, id, anything) is a huge deal to me. I personally don’t like relying on the API to determine which selector to use: for example Capybara only supports name, id and label, but you can’t tell fill_in which specific one to choose: it appears to try each selector one by one until it finds it.

I have found that Watir-WebDriver also also provides lots of flexibility/neatness. For example: it’s the only API shown here that allows URLs to not have a ‘http://’ prefix (how many people do you know who type in http:// into a browser?).

In my opinion, the high level APIs like Capybara don’t provide enough control (for example – being able to specify the explicit selector), but the low level APIs like webdriver don’t provide enough functionality. This is evident when I am using a language other than ruby (like C#) when I find myself writing a large number of web element extension methods because webdriver doesn’t provide any of them. A .set method is a classic example, even Simon Stewart writes a clearAndType method in his examples even though he wrote webdriver which sadly misses it (you must call .clear, and .send_keys).

My biggest concern about high level field APIs

But my biggest issue with the high level APIs is that I’ve frequently seen them used to write test scripts that are step by step interactions with a web form. Instead of thinking of a business application as that, people see it as a series of forms that you ‘fill in’. This means people create scenarios like Aslak Hellesøy included in his recent post about cucumber web steps (which uses Capybara) and the problems it has created.

Scenario: Successful login
  Given a user "Aslak" with password "xyz"
  And I am on the login page
  And I fill in "User name" with "Aslak"
  And I fill in "Password" with "xyz"
  When I press "Log in"
  Then I should see "Welcome, Aslak"

I’m not saying it’s not possible to end up with something as ugly as above using other APIs, but I am saying the web form DSL style naturally relates to this: as the APIs look so similar to this style because that’s what the DSL was designed for: filling in forms. I’ve seen people frequently write generic, reusable cucumber steps to match the web form DSL like:

When /^I fill in "(.+)" with "(.+)"$/ do |value, field|
  fill_in field, :with => value
end

But this means you end up with less readable, less maintainable test scripts rather than business readable executable specifications.

Summary

Ultimately what I am looking for in an automated web testing API is simplicity and full control. I personally find browser APIs like Watir-WebDriver and Watir give me this, and this is why I love them so. Your mileage may vary, you may like different styles of APIs better, but I’ve seen other APIs so badly abused by people not even thinking about it, so it makes sense to think about what you’re trying to achieve and whether what you’re doing is the right way.

Use cucumber feature folders for functional organization, tags for non-functional classification

Originally posted on testingwithvision, moving content here for completeness.

Cucumber allows you to organize your features in a directory structure, as well as label features with tags (keywords). A common question I am asked on projects is what would you use tags for as opposed to a using directory organization.

Use directories for Cucumber feature functional organization

I suggest creating a feature directory structure to organize Cucumber features by functionality, as opposed to using the suggested convention of applying tags for organization. The reason I use directories over tags for functional organization is that I find it’s more efficient to organize features into folders than tagging them, as you don’t have to tag every feature to them to organize them functionally. You also have the benefit of running a subset of scenarios organized into a lower level directory. For example:

Given a folder structure of:

  • features
    • online_ordering
      • shopping_cart.feature
      • check_out.feature
      • check_order_status.feature
    • order_fulfillment
      • modify_orders
        • cancel_order.feature
        • edit_order.feature
      • ship_order.feature
    • admin
      • update_catalog.feature

you could run all online ordering features using:

cucumber features/online_ordering

or all modify order features etc.

cucumber features/order_fulfillment/modify_orders

This allows you maximum flexibility without having to tag each feature as to how it fits functionally with the other features. It also means you don’t end up with a large features directory full of a large number of .feature files that are hard to sort.

The Cucumber feature directory structure you create allows you to see how functionality of your application relates to other functionality through the hierarchial nature of directory organization.

Use tags for non-functional feature classification

I suggest using tags for Cucumber feature and scenario non-functional classification; storing attributes or meta-data about a feature or scenario with it.

Some useful ways you can use Cucumber tags to classify features and scenarios:

  • Tagging the status of the feature/scenario: for example, @wip is commonly used to indicate the work in progress status. These can be run in a continuous integration system so the build fails if any @wip scenarios pass (they are after all, work in progress and shouldn’t pass);
  • Tagging the size of the feature/scenario. I have recently seen some great examples on the web on how to classify scenarios/tests. Simon Stewart from Google suggests t-shirt styled sizes (Small, Medium, Large), Dave Astels from Engine Yard suggests speed of execution (Fast, Slow, Glacial or Hare, Tortoise, Sloth) which is sort of the same thing;
  • Tagging how often a feature or scenario should be run (every build, hourly, daily etc);
  • Tagging whether the scenario is destructive or invasive (or indeed non-destructive or non-invasive: good for running in production!); and
  • Tagging the feature or scenario with some form of meta-data for requirements traceability, or symbolic link to some other document or system.

Once you have started using certain tags, it is easy to run only scenarios that meet certain characteristics. For example, you could run non-invasive scenarios in production that are fast to run:

cucumber --tags @non-invasive --tags @fast

Or you can combine both functionally organized directories and non-functional tags. For example, you could run all ‘online ordering’ scenarios that are ‘non-invasive’ in production, but only those that are ‘small’ or ‘medium’, so they’re fast to run.

cucumber features/online_ordering --tags @non-invasive --tags @small,@medium

Summary

By combining the power of storing your features functionally in directories, and classifying them with non-functional tags, it gives you the power to run scenarios both by functionality, and classification, or both.

Update – 17 Jan 2011

If you’re running directories below the features directory, you should require the features directory so it knows where to look for step definitions. For example:

cucumber -r features features/online_ordering 

Previous Comments

The elements of (cucumber) style

Originally posted on testingwithvision, moving content here for completeness.

I’m rather pedantic when it comes to writing English like Cucumber steps; so here’s some of my style guidelines I’ve developed the last few months. Some of my sources of inspiration include: You’re Cuking it Wrong, (My) Cucumber best practices and tips, and You’re Almost Cuking it.

Never write code in steps

When I click button with id "NameEditForm.Save Button" within table with id "NameDetails00001"

instead write something like

When I click the "Save" button in the "Name Details" section

Use double quotes in steps to indicate it’s an imperative (literal) step

As a rule of thumb, anything that is literally specified should be in double quotes, otherwise it should be specified within the sentence.

For example – an imperative style step

I set the field labeled "First Name" to "John"

As opposed to a declarative style step that still has parameters but doesn’t use double quotes.

I enter the details for user John

Cater for a/an where needed

Given I have found an active user
Given I have found a cancelled user

Matching step definition

^Given I have found an? (\w+ \w+)$

Use multi-line step arguments when checking multiple things

For example use:

Then I should see a form with fields: | First Name | | Surname | | Date of Birth |

instead of

Then I should see a form with fields "First Name, Surname, Date of Birth"

Make tables look like real tables

  • Line up the columns;
  • Put a single space at the beginning of each cell;
  • Don’t use double quotes for contents; and
  • Put a single colon (:) on the end of the preceding line.
Then I should see a form with fields: | First Name | Joe | | Surname | Heinzenburgersteinington | | Date of Birth | 01/01/1900 | 

Instead of

 Then I should see a form with fields: |"First Name"|"Joe"| |"Surname"|"Heinzenburgersteinington"| |"Date of Birth"|"01/01/1900"|

Cater for support for 1st, 2nd, 3rd etc. values in step defintions

For example:

I click the 3rd button labelled "Save"

Matching step definition

^I click the (\d+)\w\w button labelled "([^"]*)"$

Summary

These are some elements of style that I have found to be useful. What has worked for you? Do you have your own elements of style?

CopyofAnautomatedtestingjourney(1)

An automated testing journey

I did a presentation this evening in Brisbane on an automated testing journey that one may embark on. The whole thing was centered around this tube map style diagram I came up with recently: (download in PDF)

Here’s a link to my prezi slides and it should appear below (if you have flash enabled that is).  You can also download them in very printable PDF if you so choose.

I feel the presentation was well received, but I really shouldn’t have tried to squeeze three days of thinking into 30 mins. Oh well.

As always, I welcome your feedback.

Specification by Example: a love story

I attended a three day Advanced Agile Acceptance Testing/Specification by Example workshop in Melbourne last week run by Gojko Adzic, which I enjoyed immensely.

I took copious amounts of notes so I could regurgitate them here, but instead I decided to do something a bit different and write a story. It’s a story about how a team can apply what he taught us to improve their software development practices. It’s entirely fictitious (although I’d love to live in the Byron Bay hinterland!) and any resemblance to any real person or project is purely coincidental.

So here it is: Specification by Example: a Love Story (PDF). Enjoy.

Introducing the Watir Page Helper gem

I’ve very recently released a watir-page-helper gem that provides some useful helper methods when creating page models when using Watir-WebDriver (I chose not to support Watir as it doesn’t work on Firefox 4, therefore not on non-Windows machines).

This is loosely based upon the great work that Jeff Morgan did in his series of UI testing blog posts, but takes the concept a lot further IMO.

I wanted it to be solid before I released it as a gem, so I wrote unit tests for every method, and wrote yard tagged documentation for each method also. You can see the documentation automatically generated on rdoc.info.

Installation

gem install watir-page-helper

Example

require 'watir-webdriver'
require 'watir-page-helper'

class MyPage
  include WatirPageHelper

  direct_url "http://www.google.com"
  expected_element :text_field, :name => "q"
  expected_title "Google"
  text_field :search_box, :name => "q"
  button :search, :name => "btnG"

  def initialize browser, visit = false
    @browser = browser
    goto if visit

    expected_element if respond_to? :expected_element
    has_expected_title? if respond_to? :has_expected_title?
  end
end

browser = Watir::Browser.new :chrome
page = MyPage.new browser, true
page.search_box = "Watirmelon" #This method is created by WatirPageHelper
page.search #This method is created by WatirPageHelper also
browser.close

The above example is very basic, but shows the concept well. You’re much better off putting the initialize method into a base page class so you don’t need to call the same methods for every page.

What you get

Page Methods

  • direct_url: allows you to navigate to a page upon initialization, if visit is set to true
  • expected_title: allows you to automatically assert the expected title of the page when it is initialized
  • expected_element: allows you to initialize the page by looking for a certain element. This is useful for pages that load dynamic content.

Element Methods

Element methods for a majority of the Watir-WebDriver supported elements which generate useful helper methods.
For example: text_field, select_list, radio_button, form, div, span, h1..h6 etc.

Summary

I’ve been using a watir page helper on a few different projects now and I’m sick of copy-and-pasting the code each time, so I thought it’d be nice to package it as a gem.

I also think it’s important to have unit tests for this kind of stuff, so it was a good opportunity to write unit tests for every method in these classes. The documentation was an added bonus too.

I hope you find this useful, please let me know if anything doesn’t make any sense.

Links/Further Info

SpecDriver now includes Etsy.com examples

A fellow ThoughtWorker, Paul Hammant, recently announced his intent for QuickTest Professional. And his method? Provide some new technology tutorials for newbies, in the form of a series of open source automated tests for craft site Etsy.com.

He asked me to assist by providing some SpecDriver examples, and since I enjoy doing this stuff I am more than happy to help out.

So, I’ve updated the SpecDriver repository on Github to include some Etsy.com feature files, and working tests. It was a lot of fun to write these tests, because Etsy.com is awesomely testable. Its nicely formatted code with proper ids and class names makes automating against it a real treat.

The benefit of doing this exercise is that it’s meaty enough for me to get some insight into how other people code real automated tests. For example, I’ve had a look into a couple of the code samples and I see a lot of XPath selectors. I personally really dislike XPath stuff, just like I dislike XML as I find it hard to read, and less intuitive than using straight identifiers. I am proud that there isn’t a single XPath used in my Etsy.com examples.

Another thing that stood out to me was how embedded some of the stories are. For example, in the JBehave example the features are located under //etsy-stories/src/main/resources/stories, which took me some time to locate at first, and every time I forget where they are. I much prefer having a features folder in root that makes it pretty obvious what’s in there.

One thing I am very interested in is seeing how people specify tests. I found two of Paul’s scenarios repetitious so I moved them into a single scenario outline. Moving them into a scenario outline made me think about some different ways to test, so I added a new scenario, which didn’t require much more code, but strengthened my feature.

For example, Paul’s two scenarios:


Scenario: Advanced Search for a hat
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for hat
  Then there are search results

Scenario: Advanced Search for a ring
  Given I am searching on Etsy.com
  When I specify the Jewelry sub category
  And I search for ring
  Then there are search results

became my:

Scenario Outline: Advanced Search for items in various categories that exist and don't exist
  Given I am searching on Etsy.com
    When I specify the <Sub Category> sub category
    And I search for <Search Term>
    Then I should see <Search Results> search results for <Search Term> 

Examples:
  | Sub Category | Search Term  | Search Results |
  | Knitting     | 'hat'        | some           |
  | Jewelry      | 'necklace'   | some           |
  | Jewelry      | 'specdriver' | no             | 

While I was at it, I noticed there was a spelling correction feature of the advanced search. I also thought it was a good idea to test that this was working correctly.


Scenario: Misspelling a word corrects search automatically
  Given I am searching on Etsy.com
  When I specify the Knitting sub category
  And I search for 'scalf'
  Then I should see some search results for 'scarf'
  And I should see that the search was for 'scarf' instead of 'scalf'

Now I’ve done this exercise using SpecFlow and WebDriver in C#, I’m keen to do it again using Cucumber & Watir (WebDriver) in Ruby, to compare the implementation. When I am done I will post it on here for you all to see. Enjoy.

SpecDriver: A simple, open-source, page object model framework for C# automated web testing

There doesn’t seem to be a lot of material available in the C# .NET automated testing space, so I thought I would create and share my own page object model centered framework: SpecDriver.

It uses SpecFlow to define features, scenarios and steps, and then WebDriver to actually drive the browser to automate these steps, with a page object model in between to ensure maximum maintainability of the solution.

I have previously documented the steps to getting SpecFlow up and running with Visual Studio C# Express which is free to use for both commercial and non-commercial reasons. You should follow these if you would like to play around with SpecDriver.

You can access all the source code on the github repository, and please feel free to fork/improve it as you see fit.

I will explain the various elements here and how they fit together.

Feature Files for Test Specifications

SpecFlow uses .feature files exactly the same as Cucumber, so it’s pretty easy to create these. I actually used my feature files from my Cucumber framework for this example; the benefits of writing these in a non-technical style!

Feature: Google Search
  As a casual internet user
  I want to find some information about watir, and do a quick conversion
  So that I can be knowledgeable being

Scenario: Search for Watir
  Given I am on the Google Home Page
  When I search for "Watir"
  Then I should see at least 100,000 results

Scenario: Do a unit conversion
  Given I am on the Google Home Page
  When I convert 10 cm to inches
  Then I should see the conversion result "10 centimetres = 3.93700787 inches"

Scenario: Do a search using data specified externally
  Given I am on the Google Home Page
  When I search for a ridiculously small number of results
  Then I should see at most 100 results

Step Definitions that call page objects

The step definitions are small, granular methods that call methods on page objects and do assertions against expected results.

namespace Project1.StepDefinitions
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using TechTalk.SpecFlow;
    using NUnit.Framework;
    using Project1.Pages;

    [Binding]
    public class GoogleSearchStepDefinitions : BaseStepDefinitions
    {

        [Given(@"I am on the Google Home Page")]
        public void GivenIAmOnTheGoogleHomePage()
        {
            GoogleHomePage = new GoogleHomePageModel(Driver);
            Assert.AreEqual("Google", GoogleHomePage.Title);
        }

        [When(@"I search for ""(.+)""")]
        public void WhenISearchForSomething(string searchTerm)
        {
            GoogleSearchResultsPage = GoogleHomePage.Search(searchTerm);
        }

        [When(@"I search for a ridiculously small number of results")]
        public void WhenISearchForARidiculouslySmallNumberOfResults()
        {
            GoogleSearchResultsPage = GoogleHomePage.Search("macrocryoglobulinemia marvel");
        }

        [When(@"I convert (.+)")]
        public void WhenIConvertSomething(string ConversionString)
        {
            GoogleSearchResultsPage = GoogleHomePage.Search("convert " + ConversionString);
        }

        [Then(@"I should see at most ([\d,]+) results")]
        public void ThenIShouldSeeAtMostNumberOfResults(string expMaxNumberResults)
        {
            expMaxNumberResults = expMaxNumberResults.Replace(",", "");
            Assert.LessOrEqual(Convert.ToInt32(GoogleSearchResultsPage.NumberOfResults), Convert.ToInt32(expMaxNumberResults));
        }

        [Then(@"I should see at least ([\d,]+) results")]
        public void ThenIShouldSeeAtLeastNumberOfResults(string expMinNumberResults)
        {
            expMinNumberResults = expMinNumberResults.Replace(",", "");
            Assert.GreaterOrEqual(Convert.ToInt32(GoogleSearchResultsPage.NumberOfResults), Convert.ToInt32(expMinNumberResults));
        }

        [Then(@"I should see the conversion result ""(.+)""")]
        public void ThenIShouldSeeTheConversionResult(string expectedConversionResult)
        {
            Assert.AreEqual(expectedConversionResult, GoogleSearchResultsPage.ConversionResult);
        }
    }
}

Page Object Model

Each page in the application under test is represented by a page class (that inherits from a base page class), and this page class has elements and methods associated with it. The pages are the things that actually use WebDriver to interact with browsers. You can see, like my ruby page object pattern, methods that change pages return an instance of that new page. The other thing to note is that there is a base page class that has a constructor, that requires an known element to instantiate the page. This is a way of knowing where you are in your application and constantly checking it is in the right place. This also ensures consistent syncronization, especially when pages contain dynamic content such as AJAX calls.

namespace Project1.Pages
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using OpenQA.Selenium;

    public class GoogleHomePageModel : BasePageModel
    {
        private static readonly By SearchEditSelector = By.Name("q");
        private static readonly By SearchButtonSelector = By.Name("btnG");

        public GoogleHomePageModel(IWebDriver driver)
            : base(driver, SearchEditSelector)
        {
        }

        private IWebElement SearchEditElement
        {
             get { return Driver.FindElement(SearchEditSelector); }
        }

        private IWebElement SearchButtonElement
        {
            get { return Driver.FindElement(SearchButtonSelector); }
        }

        public GoogleSearchResultsPageModel Search(string term)
        {
            this.SearchEditElement.Set(term);
            this.SearchEditElement.SendKeys(Keys.Escape);
            this.SearchButtonElement.Click();
            return new GoogleSearchResultsPageModel(Driver);
        }
    }
}

Putting it all together

I have two batch files that I use in Visual Studio 2010 C# Express, one to generate the SpecFlow NUnit tests, the other to execute them and provide a visual report. I add these as external tools in VS2010 and run them from the menu. Neat.

C# ATDD on a shoestring (or the complete guide to SpecFlow & NUnit in Visual Studio 2010 Express)

Introduction

When I’m working on something new at work, I like to be able to work on it in my own time at nights and on the weekend to polish my skills through experimentation, and share what I learn. I’ve recently been working on setting up Acceptance Test Driven Development in C# at work, but unfortunately Microsoft makes it really, really hard to polish your skills in C# development at home, unless you’re willing to part with large sums of cash (say from $20,775, and I’d prefer a new car). This is probably one of the reasons why I find the Microsoft .NET testing community (and the dev community too) quite insular, Microsoft just doesn’t encourage their users to share their craft.

Sure, Microsoft provides Visual Studio 2010 (VS2010) Express Edition – which can be used free of charge, even for commercial purposes, but the limitations on its functionality are borderline ridiculous. For example, it doesn’t support Microsoft’s own unit testing framework (Microsoft obviously don’t encourage TDD), and all add-ins are banned (no ReSharper for you). It also doesn’t work with Microsoft’s own version control system (TFS).

So, Microsoft it basically makes it really hard to do anything you want to do, but there are ways to get stuff done, even if they’re convoluted.

ATDD Frameworks in C#

I know of three main open source ATDD/BDD frameworks that work in C#: SpecFlow, Cuke4Nuke and StoryQ. Only two of these use Gherkin (StoryQ uses its own slightly modified DSL), and I like Gherkin, so I’ll ignore StoryQ for now. Cuke4Nuke still requires some Ruby glue, and to keep things pure, I like SpecFlow as it’s a pure .NET implementation of ATDD with full Gherkin syntax support. For this reason, SpecFlow seems to be the most prevalent ATDD framework in the .NET community and this is what I will use for this example.

SpecFlow as an Acceptance Test Driven Development Framework

SpecFlow is an open source framework that is delivered as a Windows installer, and this installs support for feature and step definition files in Visual Studio through an Add-In that automatically generates unit tests for you from the feature files. Unfortunately, as VS2010 Express doesn’t allow add-ins, SpecFlow won’t work without some tinkering.

Install SpecFlow & NUnit

Before we get started, we need to install VS2010 Express, SpecFlow and NUnit.

  1. Install VS2010 C# Express if you don’t already have it
  2. Install the latest SpecFlow version from the SpecFlow GitHub downloads page.
  3. Install the latest NUnit version from the NUnit download page.
  4. Add the Specflow directory (C:\Program Files\TechTalk\SpecFlow or similar) and the NUnit directory (C:\Program Files\NUnit 2.5.9\bin\net-2.0 or similar) to your Windows path.

Adding SpecFlow templates to Visual Studio 2010 Express

When installing SpecFlow to VS2010 (non-express) it creates templates that can be accessed through the “New Item” menu. Fortunately, it is fairly trivial to add these to VS2010 Express. Just download the templates I have uploaded here (2010, or here (2012) unzip this directory and place the .zip files (SpecFlowFeature.zip, SpecFlowStepDefinition.zip etc.) into your Visual Studio 2010 Express C# templates directory (usually: C:\Users\username\Documents\Visual Studio 2010\Templates\ItemTemplates\Visual C# or similar).

This means you now have access to the SpecFlow templates when adding a new item to your C# project.



Generating SpecFlow NUnit tests from Feature Files

When you add a SpecFlow feature to the full version of VS2010, it automatically generates a unit test file behind the feature file, so that it’s always up to date. This doesn’t work in Express edition, so you have to use the specflow.exe file to do this for you. Fortunately, this can be done in a batch file, and then added as an “external tool” to VS2010 Express, so you get the same functionality, but not quite as neat.

  1. Enable Expert Settings in VS2010 Express (if not already enabled), by choosing Tools->Settings->Expert Settings.
  2. Create a batch file in your project directory called something like “generatespecflowtests.bat” with the following content:

echo Starting to generate SpecFlow unit tests
SpecFlow generateAll %1 /force /verbose
echo Finished generating SpecFlow unit tests
  1. Add an external tool command in VS2010 Express, using Tools->External Tools->Add. Make sure you set the starting directory, and “show output”

  1. You can then generate your tests from the Tools menu or, if you like, you can add this command to a new Toolbar by using Tools -> Customize -> Toolbars -> New, then Tools -> Customize -> Commands -> Toolbar -> Add Command.

Running SpecFlow NUnit tests from Feature Files

Simon Whittemore from London wrote an excellent post about how to automatically run Specflow NUnit tests, and capture a pretty html result, on his blog. I’ve included his batch file here, as it’s what we’ll use to run our tests from within VS2010 Express.

  1. Create a batch file in your project directory named runspecflowtests.bat or similar.
  2. Include Simon’s content (below) in your batch file.
 
@echo off
nunit-console %1
specflow.exe nunitexecutionreport %2 /xmlTestResult:%3
if NOT %errorlevel% == 0 (
echo "Error generating report - %errorlevel%"
GOTO :exit
)
if %errorlevel% ==0 TestResult.html
:exit
  1. Create an external tool setting the same as we did for the generate unit tests, but with the parameters defined on this dialog:
    1. Command: Link to your batch file
    2. Arguments: $(TargetName)$(TargetExt)$(ProjectDir)$(ProjectFileName)$(BinDir)TestResult.xml
    3. Initial Directory: $(BinDir)
    4. Use Output Window: checked

  1. Optionally add this to your VS2010 Express Toolbar as above

When you run this command in VS2010, all SpecFlow features are run, and the resulting report is launched in your default web browser.

Summary

What I have shown is that although Microsoft makes it very hard, it is possible to use VS2010 Express and open source tools like SpecFlow so that you can polish your skills in your own time.

In this theme of ATDD using C# and .NET, over the coming days I will demonstrate how you can use SpecFlow to write and run automated web tests against a web browser using the open source tool WebDriver. Stay tuned!

Update: 20 Feb 2011

The original link to the templates was wrong, I have now updated it to the correct link.

Update 29 Sep 2013

Yannick de Kercadio has kindly provided VS2012 templates here.