Deciding to have lots of children and lots of tests is still fun later on

I recently saw a paraphrased quote by James Bach from a testing meetup in Sydney.

Deciding to have lots of (automated) checks [sic: tests] is like deciding to have lots of children. It’s fun at first, but later…

I read it a number of times and each time I read it I disagreed with it a little more.

As a proud father of three beautiful boys, I truly believe having lots of children is fun at first AND fun later on. Sure, having lots of kids is hardest thing you’ll ever do and continues to be hard as each day goes by, but hard and fun aren’t opposites or mutually exclusive whatsoever1; I’ve actually found them to be strongly correlated (think of your funnest job: was it easy?). So don’t let anybody put you off having lots of kids ever, because they are still loads of fun later on (assuming you’re not scared of hard work). I love my boys: they’re the funnest people I know and they get funner every day.

As a developer of software, I also believe having lots of automated tests is fun later on, on the proviso that you’ve put thought into them upfront. I truly believe the only way to make sustainable software that you can change and refactor with confidence is to develop it using self-testing code. Sure, having too many automated e2e tests can be a PITA2 but I’d choose lots of automated tests over no or very few automated tests any day of the week3. Again, don’t let someone put you off having lots of automated tests: just do them right!


Addendum

I asked James Bach on Twitter about his quote (and how many children he has, the answer is one), and in the typical self-righteous context driven testing ‘community’ style I was called ‘reckless’ for choosing to have three beautiful boys with my lovely wife.

It didn’t end there with other members of the ‘community’ doing what they do4 and taking the opportunity to jump in uninvited, attack me for even wondering how someone with only one child can comment on having lots of children, and try to intimidate me by accusing me of using ‘ad-hominem’ falacies/attacks against James Bach (they like big words).

This entire episode reaffirms my choice to have nothing whatsoever to do with the context driven testing ‘community’ and anyone who associates themselves with it (which started by me deleting my twitter account so they can’t attack me or have anything to do with me).

My final word of warning to those of you who still consider yourself part of that ‘community’, a comment about ‘context-driven testing':

“I chose not to engage in those dogmatic discussions. I once had a job interview where the term context-driven led one of the devs to do some googling. I had to defend myself for affiliating as he’d found some right contentious and dogmatic stuff and wondered if I were some kind of extremist for including that term in my resume. It’s no longer in my resume, FWIW.”

[source]

Footnotes

[1] I recently read that happiness and unhappiness aren’t actually the opposite of one another: you can be both happy and unhappy at the same time.

[2] In case you didn’t know: PITA means ‘pain in the ass’, and lots of end to end tests are a pain in the ass. There’s lots of articles on here about why, the most recent one being about Salesforce.com and its 100,000 e2e tests.

[3] FWIW most codebases I have worked on have had zero to little automated tests, so I don’t think having too many automated tests is our common industry problem.

[4] It’s not hard to find examples of where members of this ‘community’ rally against and intimidate a particular person they disagree with on twitter, for examples: here, here, here, here, here, etc. I personally know a fellow tester who had a very similar negative experience to me a couple of years ago and has since distanced herself also.

5 books / 5 slides / 5 minutes

At the last Brisbane Software Testers meetup I volunteered to do a 5 minute lightning talk. Since I’ve read a lot of books lately I thought I would share what I had read and some of the key snippets and set my set a challenge of talking about 5 books using 5 slides in 5 minutes.

Unfortunately some of the other volunteers for lightning talks withdrew so I had a longer window and ended up talking way longer (including some bonus slides about Think Like A Freak).

I am keen to try this again using 5 books I have since read to see if it’s actually possible to communicate this amount of information. My slides are below and are also available in PDF format (accessible).

5 books - 5 slides - 5 minutes (1) 5 books - 5 slides - 5 minutes (2) 5 books - 5 slides - 5 minutes (3) 5 books - 5 slides - 5 minutes (4) 5 books - 5 slides - 5 minutes

 

100,000 e2e selenium tests? Sounds like a nightmare!

This story begins with a promo email I received from Sauce Labs…

“Ever wondered how an Enterprise company like Salesforce runs their QA tests? Learn about Salesforce’s inventory of 100,000 Selenium tests, how they run them at scale, and how to architect your test harness for success”

saucelabs email

100,000 end-to-end selenium tests and success in the same sentence? WTF? Sounds like a nightmare to me!

I dug further and got burnt by the molten lava: the slides confirmed my nightmare was indeed real:

Salesforce Selenium Slide

“We test end to end on almost every action.”

Ouch! (and yes, that is an uncredited image from my blog used in the completely wrong context)

But it gets worse. Salesforce have 7500 unique end-to-end WebDriver tests which are run on 10 browsers (IE6, IE7, IE8, IE9, IE10, IE11, Chrome, Firefox, Safari & PhantomJS) on 50,000 client VMs that cost multiple millions of dollars, totaling 1 million browser tests executed per day (which equals 20 selenium tests per day, per machine, or over 1 hour to execute each test).

Salesforce UI Testing Portfolio

My head explodes! (and yes, another uncredited image from this blog used out of context and with my title removed).

But surely that’s only one place right? Not everyone does this?

A few weeks later I watched David Heinemeier Hansson say this:

“We recently had a really bad bug in Basecamp where we actually lost some data for real customers and it was incredibly well tested at the unit level, and all the tests passed, and we still lost data. How the f*#% did this happen? It happened because we were so focused on driving our design from the unit test level we didn’t have any system tests for this particular thing.
…And after that, we sort of thought, wait a minute, all these unit tests are just focusing on these core objects in the system, these individual unit pieces, it doesn’t say anything about whether the whole system works.”

~ David Heinemeier Hansson – Ruby on Rails creator

and read that he had written this:

“…layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests (Which btw do not need to be so slow any more, thanks to advances in parallelization and cloud runner infrastructure).”

~ David Heinemeier Hansson – Ruby on Rails creator

I started to get very worried. David is the creator of Ruby on Rails and very well respected within the ruby community (despite being known to be very provocative and anti-intellectual: the ‘Fox News’ of the ruby world).

But here is dhh telling us to replace lower level tests with higher level ‘system’ (end to end) tests that use something like Capybara to drive a browser because unit tests didn’t find a bug and because it’s now possible to parallelize these ‘slow’ tests? Seriously?

Speed has always seen as the Achille’s heel of end to end tests because everyone knows that fast feedback is good. But parallelization solves this right? We just need 50,000 VMs like Salesforce?

No.

Firstly, parallelization of end to end tests actually introduces its own problems, such as what to do with tests that you can’t run in parallel (for example, ones that change global state of a system such as a system message that appears to all users), and it definitely makes test data management trickier. You’ll be surprised the first time you run an existing suite of sequential e2e tests in parallel, as a lot will fail for unknown reasons.

Secondly, the test feedback to someone who’s made a change still isn’t fast enough to enable confidence in making a change (by the time your app has been deployed and the parallel end-to-end tests have run; the person who made the change has most likely moved onto something else).

But the real problem with end to end tests isn’t actually speed. The real problem with end to end tests is that when end to end tests fail, most of the time you have no idea what went wrong so you spend a lot of time trying to find out why. Was it the server? Was it the deployment? Was it the data? Was it the actual test? Maybe a browser update that broke Selenium? Was the test flaky (non-deterministic or non-hermetic)?

Rachel Laycock and Chirag Doshi from ThoughtWorks explain this really well in their recent post on broken UI tests:

“…unlike unit tests, the functional tests don’t tell you what is broken or where to locate the failure in the code base. They just tell you something is broken. That something could be the test, the browser, or a race condition. There is no way to tell because functional tests, by definition of being end-to-end, test everything.”

So what’s the answer? You have David’s FUD about unit testing not catching a major bug in BaseCamp. On the other hand you need to face the issue of having a large suite of end to end tests will most likely result in you spending all your time investigating test failures instead of delivering new features quickly.

If I had to choose just one, I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week.

Why? Because it’s much easier to supplement comprehensive unit testing with human exploratory end-to-end system testing (and you should anyway!) than trying to manually verify units function from the higher system level, and it’s much easier to know why a unit test is broken as explained above. And it’s also much easier to add automated end-to-end tests later than trying to retrofit unit tests later (because your code probably won’t be testable and making it testable after-the-fact can introduce bugs).

To answer our question, let’s imagine for a minute that you were responsible for designing and building a new plane. You obviously need to test that your new plane works. You build a plane by creating parts (units), putting these together into components, and then putting all the components together to build the (hopefully) working plane (system).

If you only focused on unit tests, like David mentioned in his Basecamp example, you could be pretty confident that each piece of the plane would be have been tested well and works correctly, but wouldn’t be confident it would fly!

If you only focussed on end to end tests, you’d need to fly the plane to check the individual units and components actually work (which is expensive and slow), and even then, if/when it crashed, you’d need to examine the black-box to hopefully understand which unit or component didn’t work, as we currently do when end-to-end tests fail.

But, obviously we don’t need to choose just one. And that’s exactly what Airbus does when it’s designing and building the new Airbus A350:

As with any new plane, the early design phases were riddled with uncertainty. Would the materials be light enough and strong enough? Would the components perform as Airbus desired? Would parts fit together? Would it fly the way simulations predicted? To produce a working aircraft, Airbus had to systematically eliminate those risks using a process it calls a “testing pyramid.” The fat end of the pyramid represents the beginning, when everything is unknown. By testing materials, then components, then systems, then the aircraft as a whole, ever-greater levels of complexity can be tamed. “The idea is to answer the big questions early and the little questions later,” says Stefan Schaffrath, Airbus’s vice president for media relations.

The answer, which has been the answer all along, is to have a balanced set of automated tests across all levels, with a disciplined approach to having a larger number of smaller specific automated unit/component tests and a smaller number of larger general end-to-end automated tests to ensure all the units and components work together. (My diagram below with attribution)

Automated Testing Pyramid

Having just one level of tests, as shown by the stories above, doesn’t work (but if it did I would rather automated unit tests). Just like having a diet of just chocolate doesn’t work, nor does a diet that deprives you of anything sweet or enjoyable (but if I had to choose I would rather a diet of healthy food only than a diet of just chocolate).

Now if we could just convince Salesforce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change and stop David from continuing on his anti-unit pro-system testing anti-intellectual rampage which will result in more damage to our industry than it’s worth.

My thoughts on tddGate

If you’ve somehow managed to miss the keynote, blog post and subsequent shitstorm about it, David Heinemeier Hansson (dhh), creator of ruby on rails, has recently come out and declared test-driven development (TDD) dead. I’ve dubbed it ‘tddGate‘.

I find it rather ironic that David advocates the importance of clarity of code in his keynote, yet his objections to TDD through his keynote and posts are anything but clear (to me at least).

For example:

  • I don’t fully comprehend his science/pseudoscience/diet analogy in his keynote: he claims TDD is science-based because it uses metrics and coverage, but it’s also like a diet in that most people can’t make it work so it’s pseudoscience, but he also believes information system development isn’t science because it’s actually more like writing French poetry? Very confusing.
  • He interchangeably uses TDD to mean Test Driven Development and Test Driven Design.
  • He seems to imply you can only do TDD if you’re writing unit tests and you can only write unit tests if they are isolated by using dependency injection (DI) and mocks. He also seems fairly negative on unit testing, DI and mocks, therefore negative on TDD, and wants it dead so he can write (slower) system tests without using TDD, mocks or DI.
  • David gives an example of why unit tests aren’t valuable because they didn’t catch a BaseCamp bug to do with attachments (hint: the issue isn’t to do with unit testing per se, but having only one style of tests).
  • Because David thinks TDD is about unit testing, he sees driving system design from units is bad because people don’t care about units, they care about the whole thing, and doesn’t see the importance of testability.
  • Most importantly, he seems to not fully understand TDD (or at least doesn’t communicate his understanding very well):

 “TDD was what I was supposed to do. With TDD I was supposed to write all my tests first and then I would be allowed to write my code. It just didn’t work.” 25:29

The one subject that I wholeheartedly agree with David on is the importance of reading other people’s code. Writers read much more than they write, so should programmers.

So, here’s some of my current thoughts on TDD:

  • I have met few programmers who write unit tests, let alone who practice TDD.
  • Self testing code (eg. automated testing) is critically important to the health of a codebase as it allows someone to confidently make changes and/or perform refactoring without worrying they may have inadvertently broken something.
  • One way to achieve self testing code is via TDD, but it’s by no means the only way. You can easily achieve a self testing codebase by writing tests after code (or even having someone else write tests).
  • There are circumstances where it doesn’t make sense to write tests first (see some examples here).
  • It’s common to practice TDD by writing unit tests but it’s not the only way to practice TDD (for example: you could write an integration test first or an acceptance test first).
  • It’s common to write ‘isolated’ unit tests using DI and test doubles (so they’re fast and decoupled)  but it’s not the only way to write unit tests (you can interact with your database and you can test real dependencies, they’re not isolated unit tests, but are still unit tests nonetheless).
  • I personally find practicing TDD and writing unit tests first does result in a clearer, more well designed API as you’re calling your own API and you can design it how you like, but it isn’t the only way to achieve a clear API.
  • I also find practicing TDD is very effective for bug fixes as it’s easy to write a failing test and have confidence you’ve fixed the problem (and not created any others) when the test finally passes.
  • I don’t trust a test I haven’t seen fail: and this is much easier to do with TDD. You can also achieve this after the fact by (temporarily) changing your code to not work.
  • Unlike David, I strongly believe in the value of testability.
  • I believe it’s important to have the right mix of different types of automated tests for your context. Most often this means more unit tests and less end to end tests, but there are some cases where this is skewed. A diet of just one, like eating only chocolate, or completely banning sweet foods, is unhealthy and unsustainable.
  • Do what works for you personally and in your context. If you love the flow you achieve doing TDD that’s great, if you can get self testing code another way, that’s equally good.
  • If you don’t enjoy it and it doesn’t work for you, don’t make yourself do something like TDD just because someone else says to do it. But don’t stop something like TDD if you like it just because someone else declares it ‘dead’.

A ruby testing framework, from scratch, in 15 minutes

As part of my talk last week at the Brisbane Testers Meetup, I gave a live demo (no pre-recorded or pre-written code) of writing a ruby testing framework from scratch in 15 minutes. The idea was to show that most testing frameworks contain so much functionality ‘you ain’t gonna need’, so why not try writing one from scratch and see how we go? It was also a chance to show the testers who hadn’t done automated testing that programming/automated testing is not rocket science.

Since I promised to talk about selenium, I used watir-webdriver, but I would have preferred to just show testing a simple app/class that I would have written from scratch in ruby.

Our testing problem

I wrote a beautifully simple website to welcome the testers to the first ever Brisbane Testers Meetup, and wanted to write some tests to make sure it worked. The site is accessible at data:text/html,<h1 id=”welcome”>Welcome BNE Testers!</h1> and looks something like this:

bnetesterswelcome

First I’ll give you a few moments to get over how amazing that web site is… that’s long enough, now, what we need is a couple of tests for it:

  1. Make sure the welcome message exists
  2. Make sure the welcome message is visible
  3. Make sure the welcome message content is correct

Iteration zero

Do the simplest thing that could possibly work. In our case print out the three things we want to check to the screen and we’ll manually verify them.

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
puts b.h1(id: 'welcome').exists?
puts b.h1(id: 'welcome').visible?
puts b.h1(id: 'welcome').text

which outputs:

true
true
Welcome BNE Testers!

A good start but not quite a testing framework.

Iteration One

I think it’s time to introduce a method to assert a value is true.

I like to start by writing how I want my tests to look before I write any ‘implementation’ code:

require 'watir-webdriver'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

I usually run the my tests to give me a ‘clue’ to what I need to do next. In our case:

 undefined method `assert' for main:Object (NoMethodError)

In our case, it’s simple, we need to write an assert method. Luckily we know exactly what we need: a method that takes a description string and a block of code that should execute returning true, otherwise we have an error. We can simply write this method above our existing tests:

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}"
		else
			puts "Assertion FAILED for #{message}"
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'"
	end
end

which gives us this output when we run:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

This is awesome, but it makes me nervous that all of our three tests passed the first time we ran them. Perhaps we hard coded them to pass? Will they ever fail?

There’s an old saying, source unknown, which is ‘never trust a test you didn’t first see fail‘. Let’s apply this here by making all our tests fail. I usually do this by changing the source system, that way you can keep the integrity of your tests intact.

This is fairly easy to do in our case by changing the id of our welcome element.

data:text/html,<h1 id=”hello”>Welcome BNE Testers!</h1>

When we do so, all our tests fail: yipee.

Assertion FAILED for that the welcome message exists
Assertion FAILED for that the welcome message is visible with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'
Assertion FAILED for that the welcome message text is correct with exception 'unable to locate element, using {:id=>"welcome", :tag_name=>"h1"}'

We change it back and they pass again: double yipee.

Iteration Two –

So far all the text output has been in same color, and everyone knows a good test framework uses color. Lucky I know a gem that does color output easily, all we do is:

require 'colorize'

def assert message, &block
	begin
		if (block.call)
			puts "Assertion PASSED for #{message}".green
		else
			puts "Assertion FAILED for #{message}".red
		end
	rescue => e
		puts "Assertion FAILED for #{message} with exception '#{e}'".red
	end
end

which gives us some pretty output:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

and a fail now looks like this:

Assertion FAILED for that the welcome message exists

Sweet.

We can put the assert method in its own file which leaves our test file cleaner and easier to read:

require 'watir-webdriver'
require './assertions.rb'

b = Watir::Browser.new
b.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'

assert('that the welcome message exists') { b.h1(id: 'welcome').exists? }
assert('that the welcome message is visible') { b.h1(id: 'welcome').visible? }
assert('that the welcome message text is correct') { b.h1(id: 'welcome').text == 'Welcome BNE Testers!' }

b.close

Iteration Three

Our final iteration involves making the tests even easier to read by abstracting away the browser. This is typically done using ‘page objects’ and again we’ll write how we would like it to look before implementing that functionality:

require 'watir-webdriver'
require './assertions.rb'

Homepage.visit

assert('that the welcome message exists') { Homepage.welcome.exists? }
assert('that the welcome message is visible') { Homepage.welcome.visible? }
assert('that the welcome message text is correct') { Homepage.welcome.text == 'Welcome BNE Testers!' }

HomePage.close

When we run this, it provides us a hint at what we need to do:

uninitialized constant Homepage (NameError)

We need to create a HomePage class with three methods: visit, welcome and close.

We can simply add this to our tests file to get it working:

class Homepage
	def initialize
		@browser = Watir::Browser.new
		@browser.goto 'data:text/html,<h1 id="welcome">Welcome BNE Testers!</h1>'
	end

	def self.visit
		new
	end

	def welcome
		@browser.h1(id: 'welcome')
	end

	def close
		@browser.close
	end
end

After we’re confident it is working okay, we simply move it to a file named homepage.rb and our resulting tests look a lot neater:

require './assertions.rb'
require './homepage.rb'

homepage = Homepage.visit

assert('that the welcome message exists') { homepage.welcome.exists? }
assert('that the welcome message is visible') { homepage.welcome.visible? }
assert('that the welcome message text is correct') { homepage.welcome.text == 'Welcome BNE Testers!' }

homepage.close

and when we run them, they’re green as cucumbers:

Assertion PASSED for that the welcome message exists
Assertion PASSED for that the welcome message is visible
Assertion PASSED for that the welcome message text is correct

Summary

In a very short period of time, we’ve been able to write a fully functional (but not fully featured) testing framework and build on it as necessary. As I mentioned in my talk, so many test frameworks out there are so bloated and complex, sometimes all we need is simple, so if you’re putting test frameworks on pedestals because you find them too complex, start without one and see how you go!

Pedestals

I gave a talk at the first ever Brisbane Testers Meetup last night. It was a fairly good turn out despite some very climatic conditions (very wet and windy).

Every time I give a talk I try to provide a key message, a key takeaway question and ultimately aim to make sure everyone learns something that will make them capable of kicking ass in some way when they get back to work (hat tip to Kathy Sierra).

pedestals

My key message last night is that we, as testers, put things on pedestals, and we need to stop doing it; we need to push over those pedestals. We put automated testing on pedestals because it’s about programming. We put programming on pedestals often because it’s about frameworks. And we put frameworks on pedestals as they are overly complicated and complex and offer far more than we ever need.

So I tried to knock over those pedestals by showing how you can write a ruby testing framework from scratch in 15 minutes. Crash. Bang.

yourpedestal

My key takeaway last night was “what can you take down from your pedestal?” I personally think we all put things on pedestals, we greatly or uncritically admire things. We need to stop it.

The aim of my coding exercise was to show the 20 or so testers in the room who hadn’t done automated testing but wanted to do automated testing that it’s not that hard. Ignore the frameworks, focus on programming and build the simplest thing that could possibly work. Ignore the complex frameworks, the bar to learning programming and automated testing has never been lower.

My slides are available here if you’re interested in taking a look.

Waterfall, Agile Development & Hyperbole

Hyperbole. Love it or hate it, it’s been around for centuries and is here to stay. And, as someone pointed out this week, I’m guilty as charged of using (abusing?) it on this blog. You just need to quickly flick through my recent posts to find such melodramatic titles such as ‘Do you REALLY need to run your WebDriver tests in IE?‘, ‘UI automation of vendor delivered products always leads to trouble‘, and  ‘Five signs you’re not agile; you’re actually mini-waterfall‘. Hyperbole supports my motto for this blog and my life: strong opinions, weakly held.

But it’s not just me who likes hyperbole mixed into their blog posts. Only this morning did I read the catchy titled ‘Waterfall Is Never the Right Approach‘ followed quickly with a similarly catchy titled rebuttal: ‘Why waterfall kicks ass‘ (I personally would have capitalized ‘NEVER’ and ‘ASS’).

While I found both of articles interesting, I think they both missed the key difference between waterfall and agile software development (and why waterfall rarely works in these fickle times): waterfall is sequential whereas agile development is (at least meant to be) iterative.

I personally don’t care whether you do SCRUM or XP, whether you write your requirements in Word™ or on the back of an index card, or even if you stand around in a circle talking about what card you’re working on.

What I do care about is whether you’re delivering business value frequently and adjusting to the feedback you get.

Sequential ‘big bang’ development such as waterfall, by its nature, delivers business value less frequently, and chances are when that value is realized the original problem has changed (depending on how long ago that was), because as I stated and believe, we live in fickle times.

Iterative development addresses this by developing/releasing small fully functional pieces of business value iteratively and adjusting to feedback/circumstance.

Just because an organization practices what they call ‘agile’, doesn’t mean they’re delivering business value iteratively. I’ve seen plenty of ‘agile’ projects deliver business value very non-frequently, they’re putting a sequential process into agile ‘sprints’ followed by a large period of end to end, business and user acceptance testing, with a ‘big bang’ go live.

Whilst I believe iterative development is the best way to work; I’m not dogmatic (enough) to believe it’s the only way to work. Whilst I believe you could build and tests parts of say an aeroplane iteratively, I still hope there’s it’s a sequential process with a whole heap of testing at the end on a fully complete aeroplane before I take my next flight in it.

Do you REALLY need to run your WebDriver tests in IE?

I recently read that Microsoft are now on board to officially support Selenium WebDriver from Internet Explorer (IE) 11+

Whilst I welcome the news, I try to avoid running WebDriver tests in Internet Explorer completely for the following reasons:

  • Internet Explorer is a very non-testable browser. Whilst everyone agrees testability of your app is paramount, testability of its run-time container, the browser, is equally important. Settings such as security zones, proxies and auto-complete in IE must be manually configured on each machine instead of being programmatically specified by profiles in Firefox and Chrome; and
  • Because IE has historically been so hard to test, WebDriver’s support for IE is much less mature and much less stable and efficient than Firefox and Chrome

The only way automated UI tests can succeed (and the chances of success aren’t high to begin with), is if they are fast and consistent. WebDriver against IE is neither (I see it more of a problem with IE than WebDriver). So if you want to use WebDriver, don’t test against IE, test against Firefox or Chrome.

But, In my role as a consultant, I continually hear managers say that we must run our WebDriver automated tests in Internet Explorer. There’s usually one or two reasons given:

  1. Our web app is for internal staff only and our only supported browser is IE (which is usually IE8); and/or
  2. Our web app (or the one we pay for) has been specifically coded to work only in IE and therefore it’s not possible to test in another browser.

You need to explain that your WebDriver automated tests aren’t the only tests you’ll run against your app. In a corporate environment (such as those who only support IE8), chances are you’ll have a period of business acceptance testing or user acceptance testing. This will be conducted by users in the browser they use, so this straight away mitigates the risk of only running your automated tests against a non-IE browser.

From my experience testing many applications against older versions of IE, the one thing that doesn’t work well (and causes web apps to break) is not the HTML but JavaScript support. If your app contains a decent amount of JavaScript you could write some JavaScript tests in a tool like js-test-driver and run these automatically against older versions of IE automatically. That way you can be assured your JavaScript is working without having to deal with IE/WebDriver issues (and slow running tests).

As for applications specifically coded to work in IE. Web standards exist for a reason and in my opinion it’s crazy to develop a web app that is tied to the implementation of a browser by a single vendor. Microsoft made IE11 purposely report itself to a web server as not being IE so Microsoft can avoid this exact situation happening in the future.

Chances are if your app is hard-coded to only work in IE then it won’t work in IE11 anyway. If it works in IE11, then it’ll work in Chrome and Firefox as they all follow web standards, and you can run your WebDriver tests reliably now.

I believe you’re better off not having any automated UI tests if you there’s a mandate in place that you must run them against IE. If you can’t automatically test your app in Firefox or Chrome, I believe you’re better off spending your time manually testing your app in IE than trying to maintain a test suite that will never be efficient or reliable.

Tips for great brown bag lunches

I’m a big fan of brown bag seminars also called brown bag lunches or just brown bags. I’ve seen them used very successfully to share knowledge and increase team bonding. Here’s some tips to make them successful for you.

Commit to a date and lock in a topic and presenter

Since a brown bag lunch is just as much about discussion as content, I find it’s good to commit to a date and lock in a topic and presenter. This puts pressure on the presenter to make time to get their content ready, and also not worry about having it ‘perfect’.

Give everyone an opportunity to present: try to avoid having the same person presenting over and over again. A good way to harvest ideas is to have spot near your team wall (or a trello board) where people can suggest topics they would like to hear or present.

Don’t limit the audience

Resist the temptation to make a brown bag lunch only for programmers, or only for business analysts etc. Even if the topic is aimed at programmers or testers, it’s good to have a goal to make your content interesting enough that it’ll appeal to the programmer or tester in anybody.

Don’t limit yourself to content that is directly aligned with your current work

Whilst content that is directly aligned to work is good as it’s a good way to get buy in, it’s also good to present content loosely related to what people are working on. For example, you could present a brown bag on distributed version control systems (such as git) to a team purely used to working with centralized version control (such as Subversion or TFS).

If you have a couple of short presentations during a single brown bag lunch you could possibly even have one that isn’t related to work. This is a little risky of course, but it can also be fun (I’m sure that everyone would love to hear about arid plants!). It’s also a good way to break any information filters we have.

Provide lunch

When I first started organizing brown bags, I couldn’t work out whether the term brown bag seminars came from people bringing along their own lunch in a brown bag or being provided lunch in a brown bag. But through experience I have found providing a good lunch is a key contributor to a successful brown bag seminar: ‘chimpanzees who share are chimpanzees who care‘. It also provides a good motivator for people to give up their lunch break and come along because who can resist a free lunch, right?

Make sure everyone knows each other

If you’ve got a new team, or people from different areas who don’t know each other, start with a quick icebreaker where you go around the room and get everyone to introduce themselves. I usually follow the format of ‘name’, ‘role’, ‘a fun fact’ and another random tidbit such as ‘my biggest fear’ or ‘what I’m looking forward to’.

Make sure everyone takes something away

I follow the icebreaker with a question to the audience: ‘what do you expect to get out of today’s session?’ I bring a bunch of Post-it notes and sharpies along and get each person to write a few things they want to get out of the session and stick them to the wall. Ten minutes before the end of the session the presenter reads out each objective and confirms each one has been met with whomever wrote it. If there’s something that wasn’t covered, it can be discussed, or it could even become the topic of a future brown bag.

I’ve seen lots of great objectives written from things like “learn more about automated mobile testing” to “have a nice lunch with my colleagues”.

Always leave plenty of time for discussion

The discussion generated by a brown bag seminar is as important as the content. Make sure you leave plenty of time to discuss what is being presented.

Summary

I thoroughly recommend brown bag lunches as an effective information sharing and team bonding technique, and if you get them right people can really enjoy them and look forward to them.

What’s your experience been with brown bag lunches? Good? Bad? Do you have any tips yourself?