Everyone wants test automation. As applications become more complex and pose greater risks, testing costs more and takes longer. With time to market the new mantra and budget constraints a fact of life, automating the testing process is the holy grail. No surprise there.
But what is surprising is the amount of test automation that isn't happening. With almost a billion and a half spent on software quality tools each year, you would expect the majority of testing to be fully automated. But while there have been strides in automating stress tests -- which is logical, since manually creating large traffic volumes is not practical -- functional testing is automated less than 10% of the time. What gives?
The fact is that test automation technology has changed radically over the past 20 years, but most companies have failed to keep pace. To understand why your tests are probably not automated yet, it helps to understand how we got here. As someone who has been in the industry since the PC came out, I can give you a first-hand tour.
Infancy: Capture/Playback
At first test tools were small and kind of cute. They provided functionality called capture/playback, which in a nutshell recorded manual tests in the form of keystrokes and screens and then simply replayed them, looking for differences. Easy and quick.
Looking back now, what were we thinking? Trying to automate tests this way is like trying to record your drive to work so you can play it back the next day while you read the paper. Sure you can capture the steering wheel, gas and brake pedals, but when you play it back there is no guarantee the light will still be green or there won't be a car in the lane when you move over. A wreck is inevitable.
Many companies who enthusiastically bought these tools and recorded thousands of scripts were dismayed to find themselves trying to debug miles of incomprehensible keystrokes, trying to understand what went wrong, where and why. Most gave up and went back to testing manually because it took less time. Ironic.
Adolescence: Scripting
As a test-tool vendor, we realized we were causing wrecks so we scrambled to add new commands to allow testers to make decisions, branch, loop and generally control the flow of the playback. Using our car analogy, we allowed logic that specified if the light was green to step on the gas, if red to step on the brake, and if yellow to measure the distance and decide which pedal to push.
And while these new features solved the old problems, they added new ones. Most testers aren't programmers, so these new commands were foreign to them and required skills they did not have. They either hacked their way through it, creating yet more miles of poorly structured code, or they called in the militia in the form of contract programmers or additional technical resources -- who created even more miles of uniquely structured code.
In either case, companies woke up after scripting frenzies with a hangover known as maintenance. Coding every test case can yield more code than the application being tested, and every time the software changed all the tests had to be updated. Again, many gave up and went back to manual testing, because now automation not only took more time than ever, it took more expensive resources than before.
Teen Years: Frameworks
Automation groups who lived through several projects and releases came to realize that they had to develop reusable libraries that non-technical testers could use and were easier to maintain, and frameworks were born. Frameworks are essentially libraries of components to perform standard testing tasks that can be assembled into test cases by those who arent necessarily programmers.
Of course this made perfect sense in many ways -- dividing the task between automation skill and application knowledge -- but it still required a long term development and maintenance project. It is not uncommon for a company to invest half a million or more -- annually -- to develop and maintain an automation library. Ouch.
Adulthood: Commercial Applications
After several years and millions upon millions of dollars were spent on custom, internally developed frameworks, a new-class commercially supported automation frameworks emerged. These new solutions are more like applications instead of do-it-yourself languages, and they are developed and supported by vendors.
Just as many companies finally quit developing their own accounting applications and started buying them because it was cheaper than building their own, so are companies looking for test automation starting to realize that test automation is becoming a commodity. There is no need to craft totally custom frameworks and scripts: all that is unique to each application are the test cases, not the mechanics of how they are executed.
Maturity: Growing Up
What's interesting about this sojourn is that there are still, to this day, companies that are looking for capture/playback tools and believe that that is how automation is done. There are also companies who still have contractors busily writing tens of thousands of lines of test code, or who are spending hundreds of thousands on their own special framework.
Again, no surprise. The force of habit is powerful. It took decades before some companies gave up their own general ledger programs and converted to cheaper, more powerful commercial products. It is a fact of life that old habits die hard and not all decisions are made for truly economic-only reasons.
But the key is this: just because you have done something before or are still doing it, does not mean you should do it again or keep doing it. If you have been doing automation for a long time, maybe it's time to stop, step back and start over. Sometimes a fresh start is in fact a quantum leap.
IE 7
Firefox 2.0
