The Great Testing Myth

Understand, mind you, that it is not because it is impossible to do, only that is not economical. Quality is an economic issue, not a technical one. There are tools and methods for structured requirements definition and test traceability that can assure complete functional coverage, and there are source code analyzers and coverage measurement tools that can focus attention on complete source code coverage. These tools have been around for years.

The problem is not that the means don't exist, but that the time and effort required to properly implement and adhere to them are rarely invested. Fortunately, where loss of life and planet are involved - think medical and defense - the investment is greater, but in the vast majority of commercial applications where "only" money is at stake, it is usually much less.

So the question is two-fold. One, how can you convince management to invest more, and two, how do you deal with it when they don't? Interestingly, the answer to the second question will eventually answer the first.

Deal With It

As they say, to solve a problem you first must admit to having it. Admitting that you can't test everything is the first step to doing something about it.

Once you accept this, the next logical question is what can you test, given the constraints you are under? Or, said another way, if you can't possibly test it all, what do you leave out? Who decides? How?

In my experience, the most effective approach is to use good old risk analysis by asking the simple question of your customer(s): What is the most critical function that this application provides, the one thing that if it did not work would spell disaster? That goes at the top of the list. Then, what next? And so on. In other words, if you know you will run out of time before you run out of tests, be sure that the ones you do first are, in fact, the most important.

While this sounds simple enough, it is surprising how often testers default to testing the easier stuff first: equivalence classes, boundary conditions, edits and error handling, etc. Don't get me wrong - these things are important - but they may not be the most important from a business risk point of view. While I may want the system to reject bad data and handle the error appropriately, I need it to accept good data.

If you follow this approach, you will end up with a risk-prioritized list of tests, and when the inevitable happens and you aren't through testing when your time is up, you can have an intelligent discussion with management or customers about whether to release the software.

At this point, the question is not whether testing is complete or whether the software is defect-free. Companies often release software with known defects, and customers often choose to install it nonetheless. The question is whether the functions that haven't been tested, or the problems with the ones that have been, represent sufficient risk to prevent release.

Which leads to the answer to the second question.

Fix It

To get increased investment in quality processes, the first step is to make it clear to management that you can't test everything. I find that most managers are laboring under the dangerously false assumption that you are, in fact, testing everything. Further, many test managers let them believe it, not wanting to appear to be inadequate to the task or an alarmist.

So, suck it up and let them know. Making this case is relatively easy - just do the math on the possible permutations for even a small subset, which will quickly yield zillions. Achieving complete coverage of even a modest application is a staggering task.

The next step is to introduce them to your risk-prioritized list, so that they can understand what you are actually planning to do. Then, when the deadline arrives, simply point out how far down the list you are and what will be at stake if the software is released. With this information, they have a choice: give you more time and/or resources, or take the risk.

There is always a chance that they are willing to take the risk, and if so, that's fine. You've done what you could. But, of course, there is also the chance that the risk will materialize into an issue, at which time you can have objective discussions about the relative costs of resolving versus preventing problems.

It is when you reach this point that you now have some traction with management for requesting more investment. Once you can tie a real-world failure to a prior decision to accept a risk, the value proposition (as in the well-worn ROI) is easier to understand.

Either way, your stress level should decline. If you can't test it all, it's not your failure or the test team's disgrace - it is an economically-based business decision. And, if problems surface as a result, then you have more data for evaluating risk in the future. It's a classic win-win.