Checking IS testing

The ‘testing vs checking’ topic has been in discussion for many years in the software testing community. Two very vocal participants are James Bach[1] and Michael Bolton[2].

“…we distinguish between aspects of the testing process that machines can do versus those that only skilled humans can do. We have done this linguistically by adapting the ordinary English word “checking” to refer to what tools can do.”

“One common problem in our industry is that checking is confused with testing.”

~ James Bach & Michael Bolton [1]

The issue I have with the checking vs testing topic is that it is dogmatic in implying that almost everyone around the world confuses checking with testing. Apparently unit testing is actually unit checking, the test pyramid is a check pyramid, test driven development is check driven development, and there is no such thing as automated testing, only automated fact checking.

“The “testing pyramid” is a simple heuristic that has little to do with testing. It’s called the testing pyramid because whomever created it probably confuses testing with checking. That’s a very common problem and we as an industry should clean up our language.”

~ James Bach [3]

We don’t need to clean up our language: we need to adapt, invent new language and move on.

The meaning of words aren’t static. ‘Literally’ originally meant in a literal way or sense but many people now use it to stress a point[4].  ‘Awful’ used to mean inspiring wonder but now has strong negative connotations[4]. Testing now means checking. Checking now means testing.

So perhaps instead of accusing everyone of confusing  ‘testing’ and ‘checking’, we move on, accept people call checking ‘testing’, and come up with another term to describe the value added human stuff we do on projects: you know, the questioning, studying, exploring, evaluating etc.

It’ll be much easier to educate everyone on some new terminology for pure human testing  exploratory testing based on intuition, instead of trying to get them to split their current view of testing in half and admit confusion on their behalf.

[1] Testing and Checking Refined: James Bach – 26 March 2013
[2] On Testing and Checking Refined: Michael Bolton – 29 March 2013
[3] Disruptive Testing Part 1: James Bach – 6 Jan 2014
[4] From abandon to nice… Words that have literally changed meaning through the years

29 thoughts on “Checking IS testing

  1. Whilst I think I understand the points you are attempting to make within your article, I feel you have missed a crucial point that Michael and James make in the articles you reference.

    They from my interpretation see checking as a part of testing but it is definitely not testing.

    When I try to discuss this subject within the workshops I run, I make this point very clear. It is important that checking takes place and testers can help in doing this, however which has the most value (checking or testing) depends on the priority at that time for that project. This is why I like this distinction, it is then clear which activity people are involved with and therefore we can see if people are doing a lot of manual checking this could be wasteful (automated manual tools could reduce this). The distinction between checking and testing enables managers of projects to see risk quickly and easily. If there is a lot of checking and little testing it could indicate that there is a great deal of the system which we still do not know or understand. Hence my reason for liking the way that checking and testing have been separated..

    The definition of testing currently is not static and James and Michael are trying to change its meaning to be something that is clear and purposeful, to quote from your article:

    ‘The meaning of words aren’t static’

    So instead of trying to find a new word to describe testing, which IMO would confuse even more, why not redefine it to mean something that has a clear and understandable definition instead of the current state in which it has multiple definitions in the context of software testing.

    I am not saying that my views are correct but it works very well for me and helps in discussions with people who matter. I understand your frustrations that people within the testing industry get bogged down in this debate but something are worth debating otherwise change will never happen.

    • I understand they see checking as part of testing, after all, it’s within the large bubble labelled ‘Testing’ at the bottom of James Bach’s blog post that I quoted from in my post.

      But, they also say many (most?) people confuse testing and checking through terms like the testing pyramid, unit testing, and automated testing etc. They see these all as ‘checking’ activities and therefore shouldn’t be called testing (even though, strangely, they form part of a larger testing activity).

      I am happy to call these ‘checking’ tasks testing, thereby avoiding being arrogant and dogmatic by always correcting people (I’m a testing consultant).

      If we wish to differentiate what we call consider pure human testing (not checking), we should call it something else.

      • First, thanks for the post!

        I think the point that being made is that testing is what testers do. Testers also do checking (checking is part of testing), but checking does not replace testing. It’s a way to inform people that automated checks aren’t a substitute for human testing – that automated checks are part of “tool-assisted testing” with a human in the driving seat.

        Why is it strange that checking activities shouldn’t be called testing even though they form part of a larger testing activity? Applying the brakes is an activity done as part of driving, but wouldn’t insist that braking is really just driving and we should just call it that instead.

        Why is it arrogant or dogmatic to call checking tasks checking and not testing? As a testing consultant surely you’d want to inform your clients when automation may not be suitable for part of their test project, and where and why you can’t use automation as a pure replacement for skilled testing.

        Also, I’m not sure what the difference is between calling checking something else to differentiate it from its superset (testing) as they suggest, and calling testing something else to differentiate it from its subset (checking) as you suggest.

        Many thanks for your time!

  2. You haven’t addressed any of the reasons I gave in my post for why there are important differences and why we should mind them in our language. Perhaps you are against the principle of speaking precisely, but I doubt that. I think it’s more likely that you don’t care about the problems that the checking/testing distinction is meant to help solve. Perhaps you do not study the same things I study or try to solve the testing problems that I try to solve.

    And look, I get it. You want to keep doing what you’re doing and talking the way you like to talk. That’s understandable. But here’s what is going to happen: I and the people who love to think deeply about testing are going to continue our efforts to refine the theory and practice of testing. If we are successful, then people who don’t want to up their game will eventually be considered backwoods hicks. That is as it should be. I’m sure you feel that way about what you consider progress in your work, too.

    We’re going to continue our efforts, BUT, we want to get it RIGHT. In order to do that, we need our colleagues, such as you, to make coherent arguments about the potential flaws in our ways of thinking. Your blog post, above, certainly sounds miffed, but I’m not seeing an argument against the issues I have raised. (I don’t know what you mean by “move on.” I’m not moving on, I’m building an intricate and logically consistent rhetorical and practical framework for great testing.) Anyway, I’m not seeing your alternative solution. If you are saying there IS no problem, okay then make THAT argument.

    Here’s an example: Long ago I introduced the term “sapient testing” into popular usage in my community. It caught on pretty well. But after a couple of years we realized that there is a critical flaw in the concept. I then publicly retracted it. I would have appreciated a keen critique from you, way back when I first proposed it, to help me see that flaw.

    At the moment, the checking/testing distinction is allowing me to efficiently think and speak about two very different aspects of testing. I may come to believe there is a better way to deal with this issue, though, and perhaps you will be the one to show it to me.

    – james

    • Thanks for your reply James.
      The trigger for my original post is how frequently I see yourself and Michael mention the common problem you see in our industry of checking being confused with testing. Whilst I personally understand the difference between testing and checking, I believe that no matter how much you publicly talk about it, I can’t see the testing vocabulary changing any time soon, for example getting to a point where it’s accepted to call unit tests ‘unit checks’. Hence my suggestion of to move on and accept what you consider to be widespread industry confusion by defining something new as to what you consider non-checking activities are.
      In your recent interview, quoted in my post above, you went as far to say the testing pyramid has little to do with testing, which I don’t understand. Perhaps I am missing something but your blog post from last year which I heavily referenced has this diagram in it which clearly shows human/machine checking belonging INSIDE testing, so even if the testing pyramid contains predominantly what you call ‘checking’ activities, it still surely has A LOT to do with testing?
      Perhaps you were just trying to be provocative, but saying such statements, in my opinion, discredits the context driven testing community as it shows you think industry leaders like Mike Cohen (who invented the testing pyramid) and Martin Fowler (who frequently discusses the testing pyramid) don’t understand testing, which in my opinion they do.
      I don’t have any issues whatsoever with you or the context driven community not taking my suggestion on board, but I wanted to make it nonetheless.
      But I for one won’t be telling my clients they are confused and should stop using the terms unit testing or the test pyramid and to use the term ‘checking’ instead.

      • It would be perfectly okay for Mike Cohn and Martin Fowler to claim that I don’t know much about programming. I haven’t studied and practiced programming to the degree that they have. But I think that they’d be bemused if I went on about refactoring (say) as though refactoring were all there is to programming.

        I would offer that Mike and Martin know quite a lot about checking, but don’t have very much to say about the parts of testing that aren’t related to checking and to the programming of checks. Yet that part of testing is huge and significant.

        There are two issues here: one is the choice of specific labels, but as I’ve established before, that’s negotiable. The second issue is more important, and that’s the failure to see a distinction at all, in part because we use this overloaded term, “testing”, to mean many different things without questioning how much or how little of testing we’re talking about. Wonderful checking (unit tests, automated tests, Fitnesse tables, Selenium tests; call ‘em tests if you like) is wonderful. But if the client believes that wonderful checking is all there is to learning about and evaluating a product under development, the client IS confused, whether the client is using our term or not. They’re at risk if they don’t realize that leaves aren’t trees. http://www.developsense.com/blog/2013/09/very-short-blog-posts-4-leaves-and-trees/

        And I’d be cautious: the testing pyramid doesn’t have a lot to do with testing, in the same way that a diagram of where the engine is mounted on a car doesn’t have much to do with driving. The principle is a lot more significant than the diagram, and the world of checking is a lot bigger than that single principle (and the world of testing even more so).

        Elisabeth Hendrickson has suggested that we divide testing into “checking” and “exploring”. That would be great, except there’s lots of stuff within testing that is neither checking nor exploring.

        • Thanks for your reply.
          So, using your analogy, if testing is a tree, and checking are the leaves, then a pile of leaves aren’t a tree. Get it.
          But does that pile of leaves have little to do with the tree? I don’t think so, I believe there’s an strong connection there, like the testing (checking) pyramid has a lot to do with testing.

      • >>> Mike Cohen (who invented the testing pyramid) and Martin Fowler (who frequently discusses the testing pyramid) don’t understand testing, which in my opinion they do.

        Is there an option to disagree with that statement?

        I wish the agile dev. community would some day reference (leave aside understand) the work of Cem Kaner and then choose to either ignore it or refute it when discussing ‘testing’. I don’t know if there is a single blog or book which has done so. (I use Kaner because he has a mainstream book on testing; substitute with Weinberg, Bolton or Bach)

  3. Terminology has bedeviled the testing world since I started working as a tester 20 yrs ago. It drives me nuts! Janet and I tried to go with standards set by Gerard Meszaros in _XUnit Test Patterns_ in our first book. As we write our second, we find terms have changed even more, and it’s hard to pick the right one. For example, to refer to driving coding with business-facing examples (which is Brian Marick’s term, more or less, from his agile testing matrix), people use ATDD, SBE, and BDD interchangeably. They may have subtle differences but I suspect people who use the acronyms don’t know what they are.

    The testing vs. checking makes sense to me, but my reaction to the original posts about it was “Terminology is important, we need a common language. However, is it the most important thing to focus on right now?” 10 years ago, I encountered few teams other than my own who were doing TDD, example-driven development (again, Brian’s term), or test automation. The context-driven folks railed against test automation for some reason that I didn’t get. But nowadays I meet many, many teams who are doing a good job with TDD, example-driven development, and automation, and as a result, they have a lot of time for the critical exploring that Elisabeth describes here: http://www.agileconnection.com/article/two-sides-software-testing-checking-and-exploring?page=0%2C1

    It’s good to have these discussions, it gets people thinking. It’s important, but difficult, to keep egos out of the conversations. I hope we can grow towards a common language, but I suspect it will keep evolving along various paths.

    • “The context-driven folks railed against test automation for some reason that I didn’t get”

      Can you give some examples? I haven’t read or heard anything by any CDT folks that rails against test automation or TDD in principle.

      • I shouldn’t generalize, sorry. I know many people who self-identify as “context driven school” who also see the benefits of test automation. But the debates back when I engaged in them seemed to me like some people argued test automation vs. exploratory testing, and I have never seen them as mutually exclusive. You can’t get time for ET w/o automation IME.

        Anyway, that’s off topic. But thanks for keeping me honest.

  4. Interesting post. I agree mostly with the definition of checking vs testing, but I think it is not as simple as some people like to think. For example. TDD (test driven development) is definitely testing when a programmer writes his test first, but once the code is there and passing, those unit tests become part of the automated regression test suite, and then are performing a check. The same argument can be made with the API levels tests in the test pyramid. When we are asking for examples, we are testing people’s ideas and assumption…. Only when the tests all pass and become part of the regression suite, do they become checks. Automated tests that are performed at the user interface are harder to create before the code is written, but there are instances where it is being done.

    I try not to get too hung up on the differences. I prefer to automate as many of the checks as possible (or necessary) so that we can spend our time on preventing defects early.

  5. I try not to get bogged down in testing vs checking debate. For me testing is checking and checking is testing. Both activities are important to evaluate the fitness of the app for human consumption, after all software development is a social science. Checking is the first human instinct before one dives into testing. All aspects of testing are valuable whether it is visual, manual, automated or tooling to meet functional and non-function requirements. We shouldn’t be distracted from our focus how the app would change the world of our customer from current context. Testing can restore that trust

  6. Nice post, Alister,

    I generally find Misters Bach and Bolton’s ideas interesting and thought provoking. In this case, I’m okay with them using the word “checking” to describe the difference between (for example) an automated regression test and an exploratory test. Should that terminology catch on, great.

    And I wouldn’t immediately label anyone a nut, should they insist on using the term “unit checking” or “automated checking”… but we’d be headed down that road ;)

  7. It seems that the audience of this post understands that checking is not testing and that checking is part of testing so there is no reason for me to rail on those things. I think John and Lisa has hit the nail on the head with regards to the relevancy of those topics.

    I don’t really see this as a position of un-support but more like a complaint based on inconvenience. History does dictate that certain words that was predefined as a slur can be corrected. In 1987, Oxford Dictionary defined the word “Filipina” which should mean “a Female from the Philippines” as “Domestic Help” of course this caused an uproar which forced the folks from Oxford to revise that definition. The topic of this post might not be as dramatic but i hope you see my point.

    You are proposing that we should keep “testing” colloquially defined or accepted “as is” because it’s convenient. That’s hardly a good enough reason, it’s a logical fallacy as a matter of fact (Argumentum Ad Populum). As a person that can make vendor hiring decisions, the value in differentiating these definitions raises the service that I expect from companies that sell “checking” instead of, you know, testing.

    You also concluded that it is easier to teach people a new term to describe “pure human testing” than correct people. In this day and age, especially with software, there is really no such thing as “pure human testing”. There is always a form of automated assistance whether it be in the form of the js console on any browsers (even the browser itself?), or examining results of curl requests from a terminal window, ad infinitum. Teaching people who don;t like to be taught will always be difficult.

    I think I can conclude that your primary issue is really just inconvenience. I would agree that it’s difficult to change the mindset of an “uninformed” generation of testers, developers, marketing and product specialist but I think it’s worth having the discussion and it does raise the expectation of the value that testing should bring to any given organization.

    My primary complaint to this is that, I too am a tester and I don’t want my work to be confused with “just doing checking”. It’s important to me that my role is not confused with the machines that assist me.

    I think therefore I test. Let’s not confuse Descarte with the horse.

    • Thanks for your reply Perze.
      It’s not about inconvenience, it’s about being overly critical of people who use the term testing when they’re talking about what others consider checking.
      For example, saying the test pyramid has little to do with testing is not, in my opinion, constructive and doesn’t get us anywhere.
      I believe we can educate on the differences of checking and testing without firstly accusing people of being wrong or confused.
      Thanks for the pick up on “pure human testing”, I didn’t mean it as came out so I will change it.

  8. Further to my previous comments, I would like to add that I draw the distinction between checking and testing by following the rules that I have developed for my own clarification. I call it “testing” if tests that we run, manually, automated or through automation tools, execute code and all others tests such as visual observation as “checking”.

  9. Mohinder: You are not using checking and testing the same way we are. If we don’t negotiate what we mean by our respective words, that’s a problem. But the negotiation part can be as easy as we want.

    Note that James and I are explicit: the distinction is one that we make in the Rapid Testing methodology. Everyone is welcome and encouraged to adopt the terms. Those who want to continue using their terms (or inventing new words) can go ahead; no one died and bequeathed us the passwords to the Oxford Dictionary servers. We care a lot less about linguistic orthodoxy than we care about this: when we’re looking for problems in a product, there’s a difference between applying functional algorithms (by machines or by humans) on the one hand; and on the other, gaining experience of the product by experimenting with it, interacting with it, designing tests AND checks for it, recognizing and investigating risks. This is about whether the product solves a problem for someone, and whether the product causes problems for someone.

    If it turns out that Mohinder and I are working on the same project, I need to be aware that he’s coming into the project with the same words and different ideas, and he has to be aware of the same thing about me. We can use whatever words we like. We can work that out. That part is not hard. To me, it’s basically the same as deciding whether I’m going to order my Montreal dinner in French or English.

    The trouble comes when I try to convince certain Agilistas that there is crucially more to testing than a bunch of little hunks of code, but because of a single lumped-up word, we can’t differentiate between the code and wider activity. We need to have that conversation. Checking and testing, respectively, address the two categories for us. But if you’ve got different words–unit tests for what we’d call unit checks–go ahead. Say that. We’ll deal.

  10. Michael , I agree we should have this debate and freedom for people choose what they call testing or checking as long as it is communicated to others.

    At the heart of my approach to testing is empathy for the customer. I am more concerned about the current context of the customer and how AUT will change his/her world. I am interested in what business problem we are solving and why. This helps me in choosing test scenarios that would achieve the right goals rather than bogged down into details about techniques except what to know what they can be useful. Customers do not care about what types of testing have been attempted to deliver what they are expecting. We shouldn’t lose sight of the fact that customer is the expert in all this and he/she knows what will work for him/her. Gojko Adzic in his book on Impact Mapping calls customers as ‘actors’ and developers that include testers as ‘players’ because of our role in making it a reality for the actors.

    There is no clear boundary between checking and testing, we cannot say for sure where checking stops and testing begins because it is part of the testing framework. We are never going to have enough and right data to settle that claim that they are two unique activities in their own rights. There are lots of words in the English Dictionary that differ in meaning but spelt same. Therefore, people will use checking and testing in ways that fit their context, we wish they don’t. I am not saying that this is a good or a bad thing but that is how testing is perceived. Both things happen while we explore AUT and find its fitness for purpose. I prefer not to have narrower view of testing so that I can do my job properly. You said to “differentiate code and other activities” associated with testing. What would you call those sets of activities? Gojko Adzic said on a number of occasions that code is the only thing we can trust and fallback in years to come that tells us what the system does. That’s another story.

    I resent when a technique or approach has methodology associated with it. My view is that it is hard to “negotiate” with a methodologist. A methodology puts you in a straight jacket and lead to fixed mindset that only works in a stable environment. We should be aiming for growth mindset so that people can venture out and think outside the box from customer perspective. Agile framework is popular because it is driven by agile, lean, software development and Kanban principles. Because it is a framework, you can construct your own actions and practices that work for you. We should be expanding on CDT principles engine that powers our testing. There is no harm in stealing good principles around and amalgamating them in CDT.

    We should also foster empathy, big picture and pattern recognition thinking in testing on top of analytic skills.

    Kind regards,

    Mohinder

  11. To me the terms are just a useful tool to help explain testing as a whole. If I encounter a person who sees testing as being purely composed of checking activities, then I might introduce the concept of checking versus testing to them if I think it will help them understand testing as a broader concept.

    If they really need to go to the trouble of calling their unit tests “automated unit checks” in order to grasp this concept then we’ve probably got bigger issues.

  12. My thinking is that what is labelled here as ‘testing’ and checking’ are different activities and I think that these activities can be included or excluded in a role as defined by those who employ for better or worse. So I say educate the ones who define the roles and employ. If you’re your own boss then all power to you.

    I think in the quotes you gave James Bach and Michael Bolton the ‘we’ and ‘our’ are really talking about their own selves. That doesn’t exclude you agreeing with them either.

    “One common problem in our industry is that checking is confused with testing.” could translate to “We find one common problem in the industry is that checking is confused with testing.” Then they are reporting their findings.

    Only time can tell if James Bach is right and for how long. I say he argues his is not dogma because when they are not happy with the thinking they have published they retract in the face of popularity. I don’t know as i haven’t read enough if when he publishes he is inclined to lay down principles as undeniably true. I guess if he is saying “here is what i’m thinking, try it for yourself and see what results it gets you” then this isn’t so dogmatic. I’m assuming James Bach is not reading this and if you are i am not intending on being rude.

    The feeling I get from James Bach is he expressley differentiates in his own thinking between ‘learning about it through experimentation’ and ‘making evaluations by applying algorithmic decision rules to specific observations of a product.’ I am not sure what the fundamental difference is between these two activities and think they maybe the same thing described differently. Am I not making constant evaluations and applying algorithmic decision rules in experimentation too? Experiments are highly reproducible.

    In my mind there is a differentiation between ‘play’ and ‘experimentation’. ‘Play is a useful activity’ but I would like to dissociate from play any thought of without seriousness or understanding so might use words like ‘engage, participate, represent, evaluate, move lightly and quickly, discover the elusive’

    James Bach, ISEB whatever I feel these ideas published are frameworks or understanding that can allow us to hold different frames of reference and points of view and contrast these with what we find if anything.

    All these terms and references its so difficult to know what one another’s understanding is so if someone says to me James Bach I try to think ‘that frame of reference’ if someone says to me ISEB i try to think ‘that frame of reference’.

    Is this of sound mind? I don’t know.

  13. I think in the quotes you gave James Bach and Michael Bolton the ‘we’ and ‘our’ are really talking about their own selves.

    Yes, that’s explicit. “For this reason, in the Rapid Software Testing methodology, we distinguish between aspects of the testing process that machines can do versus those that only skilled humans can do. We have done this linguistically by adapting the ordinary English word “checking” to refer to what tools can do.” (my emphasis added here)

    ‘learning about it through experimentation’ and ‘making evaluations by applying algorithmic decision rules to specific observations of a product.’ I am not sure what the fundamental difference is between these two activities and think they maybe the same thing described differently. Am I not making constant evaluations and applying algorithmic decision rules in experimentation too?

    If you have not already read http://www.satisfice.com/blog/archives/856, there’s a good deal of explanatory text there.

    In experimentation, you might be applying algorithmic decision rules, but that’s not at the centre of the activity. There’s lots of other stuff going on too: modelling, conjecture, finding and applying resources on the fly, questioning, studying, hypothesizing, questioning, learning, making inferences, identifying risks… Note that none of these things can be done algorithmically. In checking, the application of algorithmic rules IS the activity, and that at least in theory can be done by a machine.

    This might help you to understand the distinction: imagine that you’re hiring a tester. One approach to evaluating that tester is to give 40-question true-or-false quiz. The answers that your prospect gives are binary, producing a bit. That’s checking the prospect, in our lingo. Another approach is to give the prospect something to test, and to watch, listen, and ask challenging questions as the activity goes on. In our parlance, that’s testing the prospect.

  14. Ugh. WordPress, by default, conflates the title of my blog and the name of my company with my name. I’ll try to fix that; I dislike even the appearance of anonymity in handles on public posts.

    —Michael B.

  15. “In experimentation, you might be applying algorithmic decision rules, but that’s not at the centre of the activity”

    Thanks Michael. This makes sense to me. I think if I ask myself “what’s at the centre of this activity” this can also help me in all sorts of other situations too.

    Good stuff.

    wickyboi – :-D

Comments are closed.