Should Testing Report To Development?

The 300+ page report released in May by the National Institute of Software and Technology (NIST) that concluded inadequate software testing costs the U.S. economy $59.5 billion per year was good news and bad news.

(To download your own copy, go to www.nist.gov/director/prog-ofc/report02-3.pdf, or for a free printed version send an email to dherbert@nist.gov and ask for Planning Report 02-3.)

The good news was that it provided more ammunition to those of us struggling to hang on to our testing staffs and budgets. The bad news was that it lumped testing in as a development activity, which added new fuel to an old debate:

Should testing report to development?

The reason this debate keeps going is because there are so many different types of testing, performed at different stages of the development life cycle, for different reasons. The answer is easy, once you clarify the question being asked.

Should Development Test?

You better believe it. There is a significant amount of testing that only developers can do. White or glass box testing, which involves exercising the code construction, can only be performed by a programmer familiar with the language. The code itself provides the conditions that must be tested -- edits, branch conditions, loops, arrays, error handling, and so forth are creatures of the source code and otherwise invisible once the code is compiled.

Component integration testing is another area that demands development attention. Again, only the developers know where the seams are in the overall design, and how information is passed among components. Given that most modern applications are virtual constellations of objects, some written and some purchased, it is critical that the interaction among them be tested by the very developers who performed the integration in the first place.

Other Quality Quest Columns

The Economics Of Quality: A Success Story The Future of Test Automation The (Questionable) Need for Speed The Great Testing Myth Testing Fault Lines

In a nutshell, development should -- and must -- be responsible for testing the way the software is built. But just because they have some test responsibility, does that mean they should manage testing?

Should Test Report to Development?

No way. The reason is that there is an entirely separate category of testing that is equally essential to success: testing the way the software will be used.

This type of testing -- often called functional or regression testing -- is also known as black box for the simple reason that it has nothing to do with the underlying code or components. Instead, it is focused on the way the end user will actually use the finished product. It requires a completely different mindset and skill set, requiring subject matter expertise in the application and a focus on business and operational risk instead of technical risk. Because it is so different, it must be independent to be effective.

But in cases where the test group is subordinate to development, a common outcome is that all the testing -- both white and black box -- has a funny way of being absorbed by the testers. After all, why should a developer furrow their brainy brow with something as tedious as testing when there are perfectly serviceable testers right next door?

The unfortunate outcome of this arrangement is that the quality of both types of testing declines overall. If testing is subordinate to them, developers will hire testers who can serve their needs -- those with development skills -- to the detriment of the skills needed from a user point of view, resulting in products that are well constructed but don't meet user needs. Or, the testers have application expertise but lack the technical skills to do the white box testing, so the product is compromised by code defects that impede the functionality.

Common Sense

Making test equal and complementary to development, instead of subordinate, makes common sense. First, students aren't allowed to grade their own papers. A student who makes an error won't catch it, especially one that stems from a lack of understanding. While you should expect them to run a spelling and grammar check to find typos and other mistakes, you can't expect them to find their own factual or reasoning errors.

It's the same with code. A developer may find an infinite loop or incorrect branch condition, but he or she won't find a missing or misunderstood requirement. It takes a tester who is evaluating the functionality, not the structure, of the application to find these types of issues.

Second, because both of these types of testing are equally essential to the success of the product, they must be organizational equals. If inequity exists, when pressure occurs -- and it will -- testing will be sacrificed. And, if testing is sacrificed, then the inevitable outcome is that development will be slowed by firefights, leading ironically to less new development and more testing.

Or is there another option? If you know of one, or think I'm wrong, I'd love to know.

Linda Hayes is CEO of WorkSoft Inc. She was one of the founders of AutoTester. She can be reached at linda@worksoft.com.