The Cost of Software Testing?
By Alan Page, Quality Director, Unity Technologies
Today, I received yet-another email from a testware vendor telling me how I could get more ROI from my testing efforts. Yesterday, there was another email from a testing service company promising me that their tool could reduce the cost of my testing. I expect tomorrow will bring another variation on this “save money on testing” theme.
I’ve been in software development and testing for a long time. I worked at Microsoft when we had multi-year ship cycles with months dedicated to testing and stabilization. Testing was something we did at the end of all of our coding milestones to ensure that whatever we would eventually ship wouldn’t be so infested with bugs that customer success would suffer.
At that time, testing was absolutely a heavy cost on the business. We frequently looked for ways to efficiently test our products and reduce risk. We had test teams with hundreds (or on the mega-products, thousands) of testers. At one time, Microsoft employed ten-thousand testers. In one of my only meetings with Bill Gates, a small group of the most senior testers at Microsoft presented some of our quality innovations. I will always remember Bill starting the meeting by saying, “I know I pay a lot of testers, but I really have no idea what they do”.
Despite my cost-ridden testing past (not to mention the vendors promising to help me save money every day), I firmly believe that in sufficiently advanced software development teams, that testing is not a cost—or even free. Instead, that it can be a value-add for the team project.
In other words, I believe that testing activities directly add value to the business.
The Times Are Changing
There’s been an important shift in the ownership of quality over the last 10+ years. A decade ago, many organizations relied on dedicated test teams to pound quality into a product with development models that push testing to the end of the cycle where bug fixes are expensive, and shipping delays are common.
In my experience, it seems that too many teams think testing is synonymous with quality, and equate more time (and money) or people (and money) spent on testing will result in higher quality.
But testing alone—especially when testing work is done solely by dedicated testing specialists isn’t the key to quality. Teams that share testing, collaborate, and strive to improve using testing as a means of acquiring knowledge about their products and processes are the teams that create great software.
Extensive testing may be able to give you an idea of software functionality and adherence to requirements, but In the end, it’s only the customer who can tell you whether the software you’ve built, meets their needs or not. If you are able to deliver quality software to your customers frequently, easily get feedback, and adapt as needed in order to solve their problems; you will cut weeks or months from your delivery schedule and save your business significant amounts of money.
"Teams that share testing, collaborate, and strive to improve using testing as a means of acquiring knowledge about their products and processes are the teams that create great software"
For decades now, the software industry had dedicated testing specialists, but many teams are discovering that most testing tasks can be completed sufficiently by members of the development team. The myths of “building mindsets” for developers and “breaking mindsets” for testers are crumbling as teams emphasize and embrace the need to collaborate and share their work. The “Us and Them” notion of Developers vs. Testers is over; as good teams are realizing that there is simply “Us”— and that WE are responsible for efficient delivery of quality software.
In traditional software development approaches, testing is something you do to the product. In modern software development, testing is part of developing the product. Teams where everyone takes part in testing throughout the software process consistently deliver high quality software to their customers at a rapid pace. Testing performed as a continuous process is a shared activity that enables bugs to be fixed as soon as they are found. With this minimized time between bug discovery and bug-fix, there’s far less inefficiency (and cost) compared to the back and forth bug table-tennis match between tester and developer that old-school software teams played with passion. Modern software teams focus on learning and improvement and work together to make sure that both the team and the software are better everyday.
Testing as a Value Add
Brent Jensen and I say that the role of testing is to accelerate the achievement of shippable quality. Rather than testing being a place where a software team spends money on top of development costs, testing and quality activities consistently can have a measurable improvement on software quality and business value.
In teams that embrace testing as part of software development and use information from continuous testing to improve processes, people, and software, everyone on the team is an accelerant towards quality and a value-add for the business. Because the entire team focuses on quality and testing as part of developing software, the traditional costs associated with test-last quality not only go away—they are replaced by the business value of high quality software, as well as the ability for the business to adapt to customer needs. Of course, you can’t just decide that today, testing is a cost, and tomorrow it’s a value— but every software team can make this transition from cost to value.
From Cost to Value
Unless the marketers targeting my inbox (and supposedly targeting yours) have their customer base completely wrong, most of you reading this likely consider testing a necessary cost—or a tax you pay in order to deliver usable software to your customers. I’m asking you to re-examine that thought, and not only think about whether the cost can be reduced or eliminated, but how testing and quality efforts directly add value to your business.
I’ve seen it happen, and think it can happen for your business too.