One of my least favourite things in my career was being handed a SAFS framework implementation to revive, expand and maintain. At first I was positive & optimistic, but after delving deeper I continually came back to the same thought: SAFS sucks!

Here’s why:

SAFS tries to be a tool-neutral framework optimised for maintainability, but in reality I have found it to actually be very unmaintainable. There are many reasons for this, but I can narrow it down to one main reason: SAFS uses a data presentation language* for both the test case design, and the test script design. This makes it both difficult and frustrating to design test scripts.

The reason that data presentation languages are bad in this context is that it removes control from the user. Instead of having the entire functionality and power of a scripting language at your peril, you have a limited set of “action words”. You never feel in control. An analogy is a DJ who plays vinyl is often better than one who plays CDs. This is because the vinyl DJ has more control, they can touch the vinyl, feel the vinyl, know the vinyl. The CD DJ simply presses buttons that sends instructions to the laser to perform actions on the CD. It’s the same when driving a car; I prefer ‘stick’ as I have more control. Using SAFS data tables for test script design would be like using Watir but saying you can’t use Ruby!

I believe that this is the main reason, but I have some others:

The forced three-tiered decomposition approach makes it less maintainable. I like to deliver the best solution for the problem, so when you have a framework in place that says “no, you can’t call a step table (where the work is done) from a cycle table (where the test is called)” I end up delivering a poor solution just to conform. A poor solution is going to be less maintainable as it has ultimately been a sacrifice.

SAFS is ridden with nasty, horrible, global variables. Data can only be passed to and from other tables using global variables. It is like anti-poka-yoking your framewok, how easy can I make it for sometime to introduce an error?

Execution time is shocking. I implemented the same test case in three different tools to collect some execution time data. Testing a Java Swing client using vanilla QuickTest Professional took 20 seconds to execute. vanilla Rational Functional Tester took 50, RRAFS took over 4 and a half minutes! Thats a huge 1700% slower than QuickTest Professional.

SAFS makes it difficult to attract and retain skilled software testing engineers: Skilled staff don’t want to write SAFS data tables. They want to write some code and make an elegant efficient solution where they are in control. They need intellectual challenges of which SAFS excel sheets can’t provide. If I am ever in a job interview again and I hear the word “SAFS” I will make a quick exit for the fire escape.

Test Automation is context driven; but SAFS tries to be a ‘one size fits all’ framework: One of the reasons I love Watir so much is that it isn’t a framework or a tool, but a library. As Bret Pettichord says:

To some degree this is simply a matter of seeing Watir as a library rather than a tool. Although we want to grow Watir and make it into something that is reasonably able to be the basis of a testing framework, we also want it to be flexible enough to allow people to adapt it to their own needs.

Using binary excel files, such as SAFS ‘Test Generator’, makes it difficult to collaborate on test scripts. When you have tests defined in an excel file which is being simultaneously updated by two different people, it is difficult to compare and merge those changes. It is so much easier to do this if you use straight ASCII files (such as ruby scripts!) using a standard SVN client such as tortoise.

You can’t compile or syntax check a SAFS excel file; the feedback loop is too long. To work out if your SAFS script is right you have to generate the SAFS low level files and run the test, only then can you can realise that you made a small typo on one of the steps. Start all over again.

The ‘Process Container’ screen mapper doesn’t support frequent screen changes. Each time a screen changes, even if it’s only a single field, you need to ‘remap’ the whole screen. What makes this worse is that all previous customisation of screen element names (from the defaults) is lost, so potentially all your scripts will no longer run.

So there it is. This is why I don’t like SAFS. I will be really interested to know if anyone out there actually loves likes SAFS and if so, how they get around what I have mentioned above.

* See Bret Pettichord’s excellent “HomeBrew Test Automation” presentation (PDF – Page 26)
** Picture by r80o (Creative Commons)