Microservices: a real world story

Everywhere I turn I hear people talking about microservice architectures: it definitely feels like the latest, over-hyped, fad in software development. According to Martin Fowler:

“…the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies.”


But what does this mean for software testing? And how does it work in the real world?

Well, my small team is responsible for maintaining/supporting a system that was developed from scratch using a microservices architecture. I must highlight I wasn’t involved in the initial development of system but I am responsible for maintaining/expanding/keeping the system running.

The system consists of 30-40 REST microservices each with it’s own code-base, git repository, database schema and deployment mechanism. A single page web application (build in AngularJS) provides a user interface to these microservices.

Whilst there are already many microservices evangelists on board the monolith hate-train; my personal experience with this architectural style has less than pleasant for a number of reasons:

  • There is a much, much greater overhead (efficiency tax) involved in automating the integration, versioning and dependency management of so many moving parts.
  • Since each microservice has its own codebase, each microservice needs appropriate infrastructure to automatically build, version, test, deploy, run and monitor it.
  • Whilst its easy to write tests that test a particular microservice, these individual tests don’t find problems between the services or from a user experience point of view, particularly as they will often use fake service endpoints.
  • Microservices are meant to be fault tolerant as they are essentially distributed systems that are naturally erratic however since they are micro, there’s lots of them which means the overhead of testing various combinations of volatility of each microservice is too high (n factorial)
  • Monolithic applications, especially written in strongly typed/static programming languages, generally have a higher level of application/database integrity at compile time. Since microservices are independent units, this integrity can’t be verified until run time. This means more testing in later development/test environments, which I am not keen on.
  • Since a lot of problems can’t be found in testing, microservices put a huge amount of emphasis on monitoring over testing. I’d personally much rather have confidence in testing something rather than relying on constant monitoring/fixing in production. Firefighting in production by development teams isn’t sustainable and leads to impacted efficiency on future enhancements.

I can understand some of the reasoning behind breaking applications down into smaller, manageable chunks but I personally believe that microservices, like any evangelist driven approach, has taken this way too far.

I’ll finish by giving a real world metric that shows just how much overhead and maintenance is involved in maintaining our microservices architected system.

A change that would typically take us 2 hours to patch/test/deploy on our ‘monolithic’ strongly typed/static programming language system typically takes 2 days to patch/test/deploy on our microservices built system. And even then I am much less confident that the change will actually work when it gets to production.

Don’t believe the hype.

Addendum: Martin Fowler seems to have had a change of heart in his recently published ‘Microservice Premium’ article about when to use microservices:

“…my primary guideline would be don’t even consider microservices unless you have a system that’s too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application. Do pay attention to good modularity within that monolith, but don’t try to separate it into separate services.”


Extensive post release testing is sign of an unhealthy testing process

Does your organization conduct extensive post-release testing in production environments?

If you do, then it shows you probably have an unhealthy testing process, and you’ve fallen into the “let’s just test it in production” trap.

If testing in non-production environments was reflective of production behaviour, there would be no need to do production testing at all. But often testing isn’t reflective of real production behaviour, so we test in production to mitigate the risk of things going wrong.

It’s also the case that often issues are found in a QA environment don’t appear in a local development environment.

But it makes much more sense to test in an environment as close to where the code was written as possible: it’s much cheaper, easier and more efficient to find and fix bugs early.

For example, say you were testing a feature and how it behaves across numerous times of day across numerous time zones. As you progress through different test environments this becomes increasingly difficult to test:

In a local development environment: you could fake the time and timezone to see how your application behaves.
In a CI or QA environment: you could change a single server time and restart your application to see how your application behaves under various time scenarios: not as easy as ‘faking’ the time locally but still fairly easy to do.
In a pre-production environment: you’ll probably have clustered web servers so you’ll be looking at changing something like 6 or 8 server times to test this feature. Plus it will effect anyone else utilizing this system.
In a production environment: you’ll need to wait until the actual time to test the feature as you won’t be able to change the server times in production.

Clearly it’s cheaper, easier and more efficient to test changing times in an environment closer to where the code was written.

You should aim to conduct as much testing as you can in earlier test environments and taper this off so by the time you can a change into production you’ll be confident that it’s been tested comprehensively. This probably requires some change to your testing process though.

Tests Performed per Environment

How to Remedy A ‘Test in Production’ Culture

As soon as you find an issue in a later environment, ask why wasn’t this found in an earlier environment? Ultimately ask: why can’t we reproduce this in a local environment?

Some Hypothetical Examples

Example One: our tests fail in CI because of JavaScript errors that don’t reproduce on a local development environment. Looking into this we realize this is because the JavaScript is minified in CI but not in a local development environment. We make a change to enable local development environments to run tests in minified mode which reproduces these issues.

Example Two: our tests failed in pre-production that didn’t fail in QA because pre-production has a regular back up of the production database whereas QA often gets very out of date. We schedule a task to periodically restore the QA database from a production snapshot to ensure the data is reflective.

Example Three: our tests failed in production that didn’t fail in pre-production as email wasn’t being sent in production and we couldn’t test it in pre-production/QA as we didn’t want to accidentally send real emails. We configure our QA environments to send emails, but only to a white-list of specified email addresses we use for testing to stop accidental emails. We can be confident that changes to emails are tested in QA.


It’s easy to fall into a trap of just testing things in production even though it’s much more difficult and risky: things often go wrong with real data, the consequences are more severe and it’s generally more difficult to comprehensively test in production as you can’t change or fake things as easily.

Instead of just accepting “we’ll test it in production”, try instead to ask, “how can we test this much earlier whilst being confident our changes are reflective of actual behaviour?”

You’ll be much less stressed, your testing will be much more efficient and effective, and you’ll have a healthier testing process.

Testing beyond requirements? How much is enough?

At the Brisbane Software Testers Meetup last week there was a group discussion about the requirement to test beyond requirements/acceptance criteria and if you’re doing so, how much is enough? Where do you draw the line? It came from an attendee who had a manager pull him up for a production bug that wasn’t found in testing but wasn’t in the requirements. If it wasn’t in the requirements, how could he test it?

In my opinion, testing purely against requirements or acceptance criteria is never enough. Here’s why.

Imagine you have a set of perfectly formed requirements/acceptance criteria, we’ll represent as this blue blob.


Then you have a perfectly formed software system your team has built represented by this yellow blob


In a perfect, yet non-existent, world, all the requirements/acceptance criteria are covered perfectly by the system, and the system exists of only the requirements/acceptance criteria.

Requirements - System

But in the real world there’s never a perfect overlap. There’s requirements/acceptance criteria that are either missed by the system (part A), or met by the system (part B). These can both be easily verified by requirements or acceptance criteria based testing. But most importantly, there are things in your system that are not specified by any requirements or acceptance criteria (part C).

Requirements - System(1)

These things in part C often exist of requirements that have been made up (assumptions), as well as implicit and unknown requirements.

The biggest flaw about testing against requirements is that you won’t discover these things in part C as they’re not requirements! But, as shown by the example from the tester meetup, even though something may not be specified as a requirement, the business can think they’re a requirement when it effects usage.

Software development should aim to have as few assumptions, implicit and unknown requirements in a system as reasonably possible. Different businesses, systems and software have different tolerances for how much effort is spent on reducing the size of these unknowns, so there’s no one size fits all answer to how much is enough.

But there are two activities that a tester can perform and champion on a team which can drastically reduce the size of these unknown unknowns.

1 – User Story Kick-Offs: I have only worked on agile software development teams over the last number of years so all functionality that I test is developed in the form of a user story. I have found the best way to reduce the number of unknown requirements in a system is to make sure every user story is kicked-off with a BA, tester and developer (often called The Three Amigos) all present and each acceptance criterion is read aloud and understood by the three. At this point, as a tester, I like to raise items that haven’t been thought of so that these can be specified as acceptance criteria and are unlikely to either make it or not make it into the system by other means or assumptions.

2 – Exploratory Testing: As a tester on an agile team I make time to not only test the acceptance criteria and specific user stories, but to explore the system and understand how the stories fit together and to think of scenarios above and beyond what has been specified. Whilst user stories are good at capturing vertical slices of functionality, their weakness, in my opinion, is they are just a ‘slice’ of functionality and often cross-story requirements may be missed or implied. This is where exploratory testing is great for testing these assumptions and raising any issues that may arise across the system.


I don’t believe there’s a clear answer to how much testing above and beyond requirements/acceptance criteria is enough. There will always be things in a system that weren’t in the requirements and as a team we should strive to reduce the things that fall into that category as much as possible given the resources and time available. It isn’t just the testers role to either just test requirements or be solely responsible/accountable for requirements that aren’t specified, the team should own this risk.

Test your web apps in production? Stylebot can help.

I test in production way too much for my liking (more details in an upcoming blog post).


Testing in production is risky, especially because I test in a lot of different environments and they all look the same. I found the only way I could tell which environment I was in was by looking closely at the URL. This was problematic as it led to doing things in a production environment thinking I was using a pre-production or test environment – oops.

I initially thought about putting some environment specific code/CSS into our apps that made the background colour different for each environment, but the solution was complex and it still couldn’t tell me I was using production from a glance.

I recently found the Stylebot extension for Chrome that allows you to locally tweak styles on any websites you visit. I loaded this extension and added our production sites with the background colour set to bright red, so now I immediately know I am using production as it’s bright red, be extra careful.

Stylebot Example

I’ve also set some other environments to be contrasting bright colours (purple, yellow etc.) so I am know from a quick glance what environment I am using.

I like this solution as I haven’t had to change any of our apps at all and it works in all environments: which is just what I needed.

Do you do something similar? Leave a comment below.

Software testers shouldn’t write code

Software testers shouldn’t write code. There I’ve said it.

“If you put too much emphasis on those [automated test] scripts, you won’t notice misaligned text, hostile user interfaces, bad color choices, and inconsistency. Worse, you’ll have a culture of testers frantically working to get their own code working, which crowds out what you need them to do: evaluate someone else’s code.”

~ Joel Spolsky on testers

I used to think that you could/should teach testers to write code (as it will make them better testers), but I’m now at a point where I think that it’s a bad idea to teach testers to code for a number of reasons:

  1. A software tester’s primary responsibility/focus should always be to test software. By including a responsibility to also write code/software takes away from that primary focus. Testers will get into a trap of sorting out their own coding issues over doing their actual job.
  2. If a software tester wants their primary focus to be writing code, they should become a software programmer. A lot of testers want to learn coding not because they’ll be a better tester, but they want to earn more money. These testers should aim to be become programmers/developers if they want to code or think they can earn more money doing that.
  3. Developing automated tests should be done as part of developing the new/changed functionality (not separately). This has numerous benefits such as choosing the best level to test at (unit, integration etc.) at the right time. This means there isn’t a separate team lagging behind the development team for test coverage.
  4. Testers are great at providing input into automated test coverage but shouldn’t be responsible for creating that coverage. A tester working with a developer to create tests is a good way to get this done.

I think the software development industry would be a lot better if we had expectations on programmers to be responsible for self-tested code using automated tests, and testers to be responsible for testing the software and testing the the automated tests. Any tester wanting to code will move towards a programming job that allows them to do that and not try to change what is expected of them in their role.

Update 19th Jan 2015: this post seems to have triggered a lot of emotion, let me clarify some things:

  • A tester having technical skills isn’t bad: the more technical skills the tester has the better – if they can interrogate a database or run a sql trace then they’ll be more efficient/effective at their job – and a tester can be technical without knowing how to code
  • I don’t consider moving from testing into programming by any means the only form of career advancement: some testers hate coding and that’s fine, other’s love coding and I think it would be beneficial for them to become a programmer if they want to code more than they test.
  • I still believe everyone should take responsibility for their own career rather than expecting their employer/boss/industry leader/blogger to do it for them (more about this here).

What is a good ratio of software developers to testers on an agile team?

The developer:tester ratio question comes up a lot and I find most, if not all, answers are “it depends”.

I won’t say “it depends” (it’s annoying). I will tell you what works for me given my extensive experience, but will provide some caveats.

I’ve worked on different agile software development teams as a tester for a number of years and I personally find a ratio of 8:1 developers to tester(s) (me) works well (that’s 4 dev-pairs if pair programming). Any less developers and I am bored; any more and I have too must to test and cycle time is in jeopardy.

Some caveats:

  • I’m an efficient tester and the 8:1 ratio works well when there’s 8 equally efficient programmers on the team – if the devs are too slow, or the user stories are too big, I get bored;
  • Everyone in the team is responsible for quality; I have to make sure that happens;
  • A story must be kicked off with the tester (me) present so I can question any assumptions/anomalies in the acceptance criteria before any code is written;
  • A story is only ready for test if the developer has demonstrated the functionality to me at their workstation (bonus points in an integrated environment) – we call this a ‘shoulder check’ – much the same way as monkeys check each others shoulders for lice;
  • A story is also only ready for test if the developer has created sufficient and passing automated test coverage including unit tests, integration tests (if appropriate) and some acceptance tests; and
  • Bug fixes take priority over new development to ensure flow.

What ratio do you find works for you?

Deciding to have lots of children and lots of tests is still fun later on

I recently saw a paraphrased quote by James Bach from a testing meetup in Sydney.

Deciding to have lots of (automated) checks [sic: tests] is like deciding to have lots of children. It’s fun at first, but later…

I read it a number of times and each time I read it I disagreed with it a little more.

As a proud father of three beautiful boys, I truly believe having lots of children is fun at first AND fun later on. Sure, having lots of kids is hardest thing you’ll ever do and continues to be hard as each day goes by, but hard and fun aren’t opposites or mutually exclusive whatsoever1; I’ve actually found them to be strongly correlated (think of your funnest job: was it easy?). So don’t let anybody put you off having lots of kids ever, because they are still loads of fun later on (assuming you’re not scared of hard work). I love my boys: they’re the funnest people I know and they get funner every day.

As a developer of software, I also believe having lots of automated tests is fun later on, on the proviso that you’ve put thought into them upfront. I truly believe the only way to make sustainable software that you can change and refactor with confidence is to develop it using self-testing code. Sure, having too many automated e2e tests can be a PITA2 but I’d choose lots of automated tests over no or very few automated tests any day of the week3. Again, don’t let someone put you off having lots of automated tests: just do them right!


I asked James Bach on Twitter about his quote (and how many children he has, the answer is one), and in the typical self-righteous context driven testing ‘community’ style I was called ‘reckless’ for choosing to have three beautiful boys with my lovely wife.

It didn’t end there with other members of the ‘community’ doing what they do4 and taking the opportunity to jump in uninvited, attack me for even wondering how someone with only one child can comment on having lots of children, and try to intimidate me by accusing me of using ‘ad-hominem’ falacies/attacks against James Bach (they like big words).

This entire episode reaffirms my choice to have nothing whatsoever to do with the context driven testing ‘community’ and anyone who associates themselves with it (which started by me deleting my twitter account so they can’t attack me or have anything to do with me).

My final word of warning to those of you who still consider yourself part of that ‘community’, a comment about ‘context-driven testing’:

“I chose not to engage in those dogmatic discussions. I once had a job interview where the term context-driven led one of the devs to do some googling. I had to defend myself for affiliating as he’d found some right contentious and dogmatic stuff and wondered if I were some kind of extremist for including that term in my resume. It’s no longer in my resume, FWIW.”



[1] I recently read that happiness and unhappiness aren’t actually the opposite of one another: you can be both happy and unhappy at the same time.

[2] In case you didn’t know: PITA means ‘pain in the ass’, and lots of end to end tests are a pain in the ass. There’s lots of articles on here about why, the most recent one being about Salesforce.com and its 100,000 e2e tests.

[3] FWIW most codebases I have worked on have had zero to little automated tests, so I don’t think having too many automated tests is our common industry problem.

[4] It’s not hard to find examples of where members of this ‘community’ rally against and intimidate a particular person they disagree with on twitter, for examples: here, here, here, here, here, etc. I personally know a fellow tester who had a very similar negative experience to me a couple of years ago and has since distanced herself also.

100,000 e2e selenium tests? Sounds like a nightmare!

This story begins with a promo email I received from Sauce Labs…

“Ever wondered how an Enterprise company like Salesforce runs their QA tests? Learn about Salesforce’s inventory of 100,000 Selenium tests, how they run them at scale, and how to architect your test harness for success”

saucelabs email

100,000 end-to-end selenium tests and success in the same sentence? WTF? Sounds like a nightmare to me!

I dug further and got burnt by the molten lava: the slides confirmed my nightmare was indeed real:

Salesforce Selenium Slide

“We test end to end on almost every action.”

Ouch! (and yes, that is an uncredited image from my blog used in the completely wrong context)

But it gets worse. Salesforce have 7500 unique end-to-end WebDriver tests which are run on 10 browsers (IE6, IE7, IE8, IE9, IE10, IE11, Chrome, Firefox, Safari & PhantomJS) on 50,000 client VMs that cost multiple millions of dollars, totaling 1 million browser tests executed per day (which equals 20 selenium tests per day, per machine, or over 1 hour to execute each test).

Salesforce UI Testing Portfolio

My head explodes! (and yes, another uncredited image from this blog used out of context and with my title removed).

But surely that’s only one place right? Not everyone does this?

A few weeks later I watched David Heinemeier Hansson say this:

“We recently had a really bad bug in Basecamp where we actually lost some data for real customers and it was incredibly well tested at the unit level, and all the tests passed, and we still lost data. How the f*#% did this happen? It happened because we were so focused on driving our design from the unit test level we didn’t have any system tests for this particular thing.
…And after that, we sort of thought, wait a minute, all these unit tests are just focusing on these core objects in the system, these individual unit pieces, it doesn’t say anything about whether the whole system works.”

~ David Heinemeier Hansson – Ruby on Rails creator

and read that he had written this:

“…layered on top is currently a set of controller tests, but I’d much rather replace those with even higher level system tests through Capybara or similar. I think that’s the direction we’re heading. Less emphasis on unit tests, because we’re no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests (Which btw do not need to be so slow any more, thanks to advances in parallelization and cloud runner infrastructure).”

~ David Heinemeier Hansson – Ruby on Rails creator

I started to get very worried. David is the creator of Ruby on Rails and very well respected within the ruby community (despite being known to be very provocative and anti-intellectual: the ‘Fox News’ of the ruby world).

But here is dhh telling us to replace lower level tests with higher level ‘system’ (end to end) tests that use something like Capybara to drive a browser because unit tests didn’t find a bug and because it’s now possible to parallelize these ‘slow’ tests? Seriously?

Speed has always seen as the Achille’s heel of end to end tests because everyone knows that fast feedback is good. But parallelization solves this right? We just need 50,000 VMs like Salesforce?


Firstly, parallelization of end to end tests actually introduces its own problems, such as what to do with tests that you can’t run in parallel (for example, ones that change global state of a system such as a system message that appears to all users), and it definitely makes test data management trickier. You’ll be surprised the first time you run an existing suite of sequential e2e tests in parallel, as a lot will fail for unknown reasons.

Secondly, the test feedback to someone who’s made a change still isn’t fast enough to enable confidence in making a change (by the time your app has been deployed and the parallel end-to-end tests have run; the person who made the change has most likely moved onto something else).

But the real problem with end to end tests isn’t actually speed. The real problem with end to end tests is that when end to end tests fail, most of the time you have no idea what went wrong so you spend a lot of time trying to find out why. Was it the server? Was it the deployment? Was it the data? Was it the actual test? Maybe a browser update that broke Selenium? Was the test flaky (non-deterministic or non-hermetic)?

Rachel Laycock and Chirag Doshi from ThoughtWorks explain this really well in their recent post on broken UI tests:

“…unlike unit tests, the functional tests don’t tell you what is broken or where to locate the failure in the code base. They just tell you something is broken. That something could be the test, the browser, or a race condition. There is no way to tell because functional tests, by definition of being end-to-end, test everything.”

So what’s the answer? You have David’s FUD about unit testing not catching a major bug in BaseCamp. On the other hand you need to face the issue of having a large suite of end to end tests will most likely result in you spending all your time investigating test failures instead of delivering new features quickly.

If I had to choose just one, I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week.

Why? Because it’s much easier to supplement comprehensive unit testing with human exploratory end-to-end system testing (and you should anyway!) than trying to manually verify units function from the higher system level, and it’s much easier to know why a unit test is broken as explained above. And it’s also much easier to add automated end-to-end tests later than trying to retrofit unit tests later (because your code probably won’t be testable and making it testable after-the-fact can introduce bugs).

To answer our question, let’s imagine for a minute that you were responsible for designing and building a new plane. You obviously need to test that your new plane works. You build a plane by creating parts (units), putting these together into components, and then putting all the components together to build the (hopefully) working plane (system).

If you only focused on unit tests, like David mentioned in his Basecamp example, you could be pretty confident that each piece of the plane would be have been tested well and works correctly, but wouldn’t be confident it would fly!

If you only focussed on end to end tests, you’d need to fly the plane to check the individual units and components actually work (which is expensive and slow), and even then, if/when it crashed, you’d need to examine the black-box to hopefully understand which unit or component didn’t work, as we currently do when end-to-end tests fail.

But, obviously we don’t need to choose just one. And that’s exactly what Airbus does when it’s designing and building the new Airbus A350:

As with any new plane, the early design phases were riddled with uncertainty. Would the materials be light enough and strong enough? Would the components perform as Airbus desired? Would parts fit together? Would it fly the way simulations predicted? To produce a working aircraft, Airbus had to systematically eliminate those risks using a process it calls a “testing pyramid.” The fat end of the pyramid represents the beginning, when everything is unknown. By testing materials, then components, then systems, then the aircraft as a whole, ever-greater levels of complexity can be tamed. “The idea is to answer the big questions early and the little questions later,” says Stefan Schaffrath, Airbus’s vice president for media relations.

The answer, which has been the answer all along, is to have a balanced set of automated tests across all levels, with a disciplined approach to having a larger number of smaller specific automated unit/component tests and a smaller number of larger general end-to-end automated tests to ensure all the units and components work together. (My diagram below with attribution)

Automated Testing Pyramid

Having just one level of tests, as shown by the stories above, doesn’t work (but if it did I would rather automated unit tests). Just like having a diet of just chocolate doesn’t work, nor does a diet that deprives you of anything sweet or enjoyable (but if I had to choose I would rather a diet of healthy food only than a diet of just chocolate).

Now if we could just convince Salesforce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change and stop David from continuing on his anti-unit pro-system testing anti-intellectual rampage which will result in more damage to our industry than it’s worth.

Free yourself from your filters

One of the most interesting articles I have read recently was ‘It’s time to engineer some filter failure’ by Jon Udell:

“The problem isn’t information overload, Clay Shirky famously said, it’s filter failure. Lately, though, I’m more worried about filter success. Increasingly my filters are being defined for me by systems that watch my behavior and suggest More Like This. More things to read, people to follow, songs to hear. These filters do a great job of hiding things that are dissimilar and surprising. But that’s the very definition of information! Formally it’s the one thing that’s not like the others, the one that surprises you.”

Our sophisticated community based filters have created echo chambers around the software testing profession.

“An echo chamber is a situation in which information, ideas, or beliefs are amplified or reinforced by transmission and repetition inside an “enclosed” system, often drowning out different or competing views.” ~ Wikipedia

I’ve seen a couple of echo chambers have evolved:

  • The context driven testing echo chamber where the thoughts of a couple of the leaders are amplified and reinforced by the followers (eg. checking isn’t testing)
  • The broader software testing echo chamber where testers define themselves as testers and are only interesting in hearing things from other testers (eg. developers are evil and can’t test)
  • The agile echo chamber where anything agile is good and anything waterfall is bad (eg. if you’re not doing continous delivery you’re not agile)

So how do we break free of these echo chambers we’ve built using our sophisticated filters? We break those filters!

Jon has some great suggestions in his article (eg. dump all your regular news sources and view the world through a different lens for a week) and I have some specific to software testing:

  • attend a user group or meetup that isn’t about software testing – maybe a programming user group or one for business analysts: I attend programming user groups here in Brisbane;
  • learn to program, or manage a project, or write CSS.
  • attend a conference that isn’t about context driven testing: I’m attending two conferences this year, neither are context driven testing conferences (ANZTB Sydney and JSConf Melbourne);
  • follow people on twitter who you don’t agree with;
  • read blogs from people who you don’t agree with or have different approaches;
  • don’t immediately agree (or retweet, or ‘like’) something a ‘leader’ says until you validate it actually makes sense and you agree with it;
  • don’t be afraid to change your mind about something and publicize that you’ve changed your mind; and
  • avoid the ‘daily me‘ apps like the plague.

You’ll soon be able to break yourself free from your filters and start thinking for yourself. Good luck.

Checking IS testing

The ‘testing vs checking’ topic has been in discussion for many years in the software testing community. Two very vocal participants are James Bach[1] and Michael Bolton[2].

“…we distinguish between aspects of the testing process that machines can do versus those that only skilled humans can do. We have done this linguistically by adapting the ordinary English word “checking” to refer to what tools can do.”

“One common problem in our industry is that checking is confused with testing.”

~ James Bach & Michael Bolton [1]

The issue I have with the checking vs testing topic is that it is dogmatic in implying that almost everyone around the world confuses checking with testing. Apparently unit testing is actually unit checking, the test pyramid is a check pyramid, test driven development is check driven development, and there is no such thing as automated testing, only automated fact checking.

“The “testing pyramid” is a simple heuristic that has little to do with testing. It’s called the testing pyramid because whomever created it probably confuses testing with checking. That’s a very common problem and we as an industry should clean up our language.”

~ James Bach [3]

We don’t need to clean up our language: we need to adapt, invent new language and move on.

The meaning of words aren’t static. ‘Literally’ originally meant in a literal way or sense but many people now use it to stress a point[4].  ‘Awful’ used to mean inspiring wonder but now has strong negative connotations[4]. Testing now means checking. Checking now means testing.

So perhaps instead of accusing everyone of confusing  ‘testing’ and ‘checking’, we move on, accept people call checking ‘testing’, and come up with another term to describe the value added human stuff we do on projects: you know, the questioning, studying, exploring, evaluating etc.

It’ll be much easier to educate everyone on some new terminology for pure human testing  exploratory testing based on intuition, instead of trying to get them to split their current view of testing in half and admit confusion on their behalf.

[1] Testing and Checking Refined: James Bach – 26 March 2013
[2] On Testing and Checking Refined: Michael Bolton – 29 March 2013
[3] Disruptive Testing Part 1: James Bach – 6 Jan 2014
[4] From abandon to nice… Words that have literally changed meaning through the years