Don’t quit your day job

February 4, 2010

The term “don’t quit your day job” is often used as a funny way to say someone’s crap at something. This post isn’t about being crap at something, that’s pretty easy, it’s about what to do when you’re in a day job that’s stable and pays well but you don’t like it.

Everyone wants a perfect job. The problem is the perfect job often doesn’t come around that often, might not be that stable, and might not pay that great. Most people have bills to pay, families to feed, so that’s where a day job fits in, and probably why a lot of people have day jobs. I have often found myself in a day job that’s been fairly stable, and pays the bills, but isn’t that great. What do you do? Quit your day job? Or worse, do nothing and become moldy.

My perfect job would be working on software development projects, where I could use my ruby/watir skills, have a fair degree of autonomy with challenging but intellectually rewarding work. Oh, and it would have to pay enough and be stable to support a young family. The problem is I havent’ ever found such a job. I’ve worked on some great projects, sure, but they’ve been so up and down I’ve felt seasick.

My current job is stable and pays fairly well. The downside is it’s not (that) intellectually rewarding. I would have to give up the stability and pay it offers to take on a more intellectually rewarding job.

It’s not the best situation to be in, I would rather a perfect job, but what do you do in this situation?

My answer involves open source, communities, blogs and user groups. You need to find you own interests, and do what you love, outside of your day job. And the Internet makes it so easy. It’s the modern-day equivalent of the novelist who works a day job in a bookstore.

One of the reasons I got involved in the Watir project, and started this blog, is to find my own interests, and challenges, outside of my day job. To do what I love, so the day job doesn’t feel that bad.

I’m not saying never stop looking for your perfect job, I’ve even written a list of my perfect employers, I’m just saying that sometimes your day job can only be that, a day job, you need to do what you love someplace else. To quote Richard Dean Anderson, on his day job acting as MacGyver: “That job was just a paycheck to me“.


Rocket Surgery Made Easy™ by Steve Krug

January 8, 2010

I am a really big fan of no-frills usability testing. Steve Krug is a pioneer in this space and I have a lot of respect for him.

I’ve had his first book “Don’t make me think” for a number of years, and have read it numerous times and I have enjoyed it immensely. I was very happy to find yesterday his second book titled: “Rocket Surgery Made Easy” has just been released. I ordered it straight away from The Book Depository (free worldwide shipping!) which is $30 cheaper than buying it locally. I can’t wait until it arrives so I can read it.

In the meantime I found a presentation Steve did in late 2008, which, whilst an hour long, is a great introduction into no-frills usability testing. I love how he doesn’t try to sell anything (actually the opposite, he gives away free PDFs), and  shares everything he knows. Amazing stuff.


Software Quality Metrics

January 3, 2010

Software quality metrics are a very interesting topic, and from my experience, there doesn’t seem to be a widely used or accepted list of metrics that can be used to measure software quality. After many years of thought on the topic, and many years of trialing different metrics, I believe the number one metric that accurately measures software quality is defects in production. Quality software won’t include defects in production, so I believe that’s the metric we should use to measure whether testing is done successfully.

Various organizations I have worked in have used this metric in different ways. One organization called each production defect a ‘quality spill’. Another used a mean time to failure metric which is often used to measure the reliability of a production system, or machine. This could be used for example with your car and how long it takes to break down.

The issue I have with some other software quality metrics is that they motivate people the wrong way. For example, having a metric about bug count encourages testers to report bugs. But it can also encourage them to report bugs that aren’t bugs, or split one major bug into multiple bug tickets, so the metrics look good. Also, is a high bug count (in test) a bad thing? Doesn’t it mean you got all the bugs? Or does a low bug count mean the developers are doing a good job? Or perhaps you didn’t catch all the bugs? That’s why production defects is a true measure of software quality. No one wants bugs in production, they cause all sorts of headaches. In the last few days there have been numerous, embarrassing, public computer glitches, some related to the beginning of the year 2010. Have we become complacent after Y2K?

  • 3 Jan 2010: “Businesses stung by BOQ computer bug” (link)
  • 3 Jan 2010: “Bank of Queensland’s (BOQ) Eftpos terminals go down costing retailers thousands” (link)
  • 3 Jan 2010: “Chaos as check-in problems affect Qantas” (link)
  • 3 Jan 2010: “Flights delayed after check-in system malfunction” (link)
  • 10 Dec 2009: “Computer glitch brings Brisbane trains to a standstill” (link)
  • 16 Dec 2009: “Check-in failure sparks Brisbane Airport delays” (link)
  • 16 Nov 2009: “Computer glitch delays Qantas flights” (link)

What’s interesting is the Amadeus system Qantas uses failed in November and failed again today. The lesson here is if you do discover bugs in production, make sure you fix them.


Lessons Learned in Automated Testing Presentation

December 11, 2009

Yesterday I took time out from my paternity leave to do a quick presentation at the Brisbane ACS Testing SIG on some of my lessons I have learned in automated testing.

You can download the full presentation here, or view it online here.


Happy 5th Birthday to two great open source projects

November 20, 2009

November 2009 marks the fifth birthday of two great open source projects, both of which I couldn’t live without.

  1. Firefox was first released on November 9th, 2004. I remember first installing it in 2004, loving it and I still use it to this day. I’m actually using it right now to write this post.
  2. Watir made its first public appearance on November 15th, 2004, in a tutorial by Bret and Paul at StarWest 2004.

The two projects became connected in late 2006 when FireWatir was developed by Angrez Singh. Since then Watir has incorporated native Firefox support.

So here’s cheers to these two great open source projects! Hope there are many more great birthdays to come!


James Whittaker on Exploratory Testing

October 17, 2009

I had an amicable hallway conversation with James Bach. His blogger angst at my use of the title ‘Exploratory Testing’ didn’t spill over to a face-to-face discussion. Frankly, I am not surprised. I’ve never claimed the term as my own, I simply took it and made it work in the hands of real testers on real software under real ship pressure. Consultants can coin all the terms they want, but when us practitioners add meat to their pie, why cry foul? Is it not a better reaction to feel happy that there are people actually doing something with the idea?

~ James A Whittaker from the Google Testing Blog


Not Ruining your Test Automation Strategy

October 1, 2009

I was very curious yesterday when I read a quote on twitter taken from a blog post by Uncle Bob Martin.

“The net result is that GUI driven automated tests are fragile & the process of maintaining them is expensive & unreliable” @unclebobmartin

I clicked through to the blog post but I wasn’t impressed with what else I saw.

“The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.”

I immediately thought, I have made automated GUI testing successful, so who’s this guy to state all automated GUI testing is wrong.

Whilst I understand there are better ways of automated testing without using a GUI, I also understand that sometimes it’s not possible to test anything but the GUI, for example: poorly written, old, legacy systems.

But having to use the GUI doesn’t mean you can’t make a maintainable solution, I have this at many places, it just takes skill. I see what Bob thinks of this:

“Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?”

Whether outsourced or not, the people creating automated tests do need to care about this. Somewhat still amazed at what I was reading, I had an aha moment when I read the following (emphasis added):

“What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.”

Aha! This guy has only ever worked on a project where GUI automated testing has been done through record/playback. No wonder he thinks poorly of it.

So, to conclude, I would like to make the following points clear:

  • Automated testing through non-GUI means is smart, but sometimes you have no choice.
  • I have made automated testing through the GUI reliable and maintainable, but it required skill on my part.
  • Automated GUI tests can be used to deliberately show discrepancies in the GUI, often highlighting unintended GUI changes.
  • It’s generally not a good idea to completely write something off because you may have seen it done poorly yourself. It’s like saying Agile is wrong because you worked somewhere where Agile was done poorly.


Software Piracy

September 22, 2009

I consciously choose not to use any pirated software. There are a few reasons why I made this decision:

  • I work in software development, so using pirated software is like ‘biting the hand that feeds you‘;
  • When you install pirated or ‘cracked’ software on your system, you are potentially opening your system up to all sorts of nasties included in the pirating process; and
  • I have found viable free and/or open source alternatives that remove the need to use pirated software in the first place, so I am not really missing out at all.

pirate

So what do I use instead? Here’s some commercial software and the free/open source alternatives that I use:

A lot of these alternatives use open standards and formats, which is a good thing to encourage. The more people who use open standards and formats, the more commercial software vendors will feel the need to support them.

The other benefit of using these alternatives is that if you decide to switch operating systems then most of these are cross platform, so you don’t need to change your software that you use.

Which brings me to my final point. The more you start to use these cross platform alternatives, and the more things are done on the web (email, IM etc.), the more you start to question why you need a commercial operating system.

Most of the packages above work on Ubuntu or other flavours of Linux, so there’s no reason not to switch to a non-commercial operating system as well.


Why I do automated testing

September 19, 2009

I often get asked the question: “so why do you do automated testing?” both in job interviews and from people I meet. My answer is always pretty much the same; being that it provides me with exposure to a variety of business domains and technologies, and allows me to write code.

Let me expand a bit further on this. I like writing code but I’ve chosen not to be a software engineer. The reason is that I have met lots of different software engineers that have responsibility for one area of a system. They become the expert in this area of the system, hence when you find a problem in that area, you know who to speak to. Often time these people don’t really know much outside their area of expertise in the system, so when the problem is outside their own area, you need to speak with someone else. There’s nothing wrong with this, but it just isn’t me.

I like to know, or attempt to ‘work out’, how the system functions as a whole. I usually achieve this on a project I join by writing an automated build verification test, or ’smoke’ test, that verifies the system functions as a whole. I love writing these kinds of tests because you need to know how the entire thing works, not just the isolated components. Software testing is one of the few areas where you get this broad level of exposure, and this keeps things interesting.

Another added bonus of doing software testing, is that your skills are pretty transferable, both across business domains and different technologies. For example, so far in my career I have tested systems that:

  • manage job advertisements, job seeker matches and interviews;
  • process immigration and visa applications;
  • manage disability support and funding;
  • produce electronic school report cards;
  • optimise large scale open cut mines;
  • monitor health of heavy machinery; and
  • manage large funds for people’s retirements.

These systems were developed in various technologies including:

  • Java;
  • Microsoft .NET;
  • Cobol;
  • Powerbuilder;
  • Web;
  • Active X;
  • Terminal Emulators;
  • SOAP web services and;
  • CICS.

The point I am trying to make is that if I decided to be a Java Software Engineer when I finished university, because that’s why I studied, I wouldn’t have had exposure to so many technologies and business areas that I have had as an automated software tester.

Sure, I could become a business analyst, because they work across business domains and technologies, but business analysts don’t code, and I do like writing code, that’s why automated testing is perfect.

The final benefit I see in automated testing is that you can apply your skills in particular testing tools to other tools fairly easily, as the concepts are the same. The biggest issue I see with people doing automated testing isn’t picking up a new tool, but rather developing a maintainable and easy to use test framework. Once you have developed a few frameworks for various business domains and technologies, it is easy to apply the concepts to other testing tools.

So that’s why I do automated testing, and I proud of it.


Dynamically calling ruby methods in modules

September 17, 2009

When I am creating Watir tests, I write ruby methods to define user tasks, for example, adding a book to a cart becomes def add_book. I then group these ruby methods into ruby modules divided logically by the area of the application I am writing tests for. For example, I would have a ‘Customer’ module and an ‘Admin’ module for the Depot app. The benefit of using modules is you can avoid namespace conflicts as essentially each method is defined by its module’s prefix. This means that you can happily have def Customer.log_on and def Admin.log_on without any conflict or confusion.

As I have mentioned before, I like defining tests outside my code. These tests ultimately need to execute an associatted ruby method (stored in a module) by passing some data in (and getting an outcome and some output back). One way of calling these tests defined external to our code is to have a massive case statement that determines what calls what. This isn’t ideal as it is a maintenance burden, and really it isn’t needed.

In ruby it’s straightforward to dynamically load ruby modules, and then dynamically call individual methods.

require 'temp'

module_name = "Temp"
method_name = "hello_world"

required_module = Kernel.const_get(module_name)
required_method = required_module.method(method_name)
required_method.call('Alister')

This is all well and good if Temp.helloworld() exists, but if it doesn’t, our code throws exceptions:


`const_get': uninitialized constant Kernel::Temp (NameError)

or

`method': undefined method `hello_world' for class `Module' (NameError)

One way to avoid these exceptions is to wrap the code with a rescue clause, but I realised there are some easy ways to check if both modules and methods exist before loading them.

require 'temp'

module_name = "Temp"
method_name = "hello_world"

if Object.const_defined?(module_name)
  required_module = Kernel.const_get(module_name)
  if required_module.respond_to?(method_name) then
    required_method = required_module.method(method_name)
    required_method.call('Alister')
  else
    puts "Invalid method '#{method_name}' for module '#{module_name}'"
  end
else
 puts "Invalid module '#{module_name}'"
end

This ensures that the code continues to execute if the module or method name is specified incorrectly, which is sometimes the case if its specified in a spreadsheet, and especially if someone else has designed the spreadsheet.

Once we are happy about dynamically finding methods in modules, the next step is to make sure that each method is called with the correct number of parameters. This property of a method is called the arity.

The great thing about ruby and arity is that you simply determine the number of parameters and then pass in a correct sized array, using a *, and the receiving method will automatically unpack the array into the parameters specified.

puts required_method.arity()
required_method.call(*parameters)

The flexibility that ruby offers is amazing. I have tried to accomplish this same concept in VBScript but I couldn’t work out how. That’s why I am glad Watir uses ruby, it ultimately means my automated test framework is more efficient and maintainable.