I’ve been carping on code construction quality quite a bit lately. One of the five key methods of removing bugs early is unit testing. (The others are design, design review, code review, and code analysis, like PREfast.) Doing comprehensive unit testing has drawn dubious disbelief from some devs I meet: “Isn’t testing a job for testers?” “If devs write the [unit] tests, what’s left for testers to do?”
Eric Aside PREfast is a static analysis tool for the C and C++ programming languages that identifies suspect coding patterns that might lead to buffer overruns or other serious programming errors. Though initially used only internally, it shipped as part of Microsoft Visual Studio 2005.
First of all, unit tests are usually focused on isolated components, and testers cover far more than component testing. Their tests include boundary coverage, structural coverage, path coverage, black- and white-hat testing, and a host of system-wide and scenario-based testing, just to scratch the surface.
No, first of all, what the heck do you care if there’s nothing left for testers to do anyway? Since when have you abandoned your responsibility to check your own work? Is passing on crappy code some kind of perverse form of charity to you? Do you not give a flying fork about doing your job? Have you no decency or pride?
Eric Aside When I wrote this column, I initially got stuck for a “second of all” paragraph, so I reread the first “First of all” paragraph. Suddenly, the “No, first of all” paragraph flowed out faster than I could type it. That’s always a good thing.
Unit testing drives better implementation design; more testable code; fewer regressions, build breaks, and BVT failures to debug; and better overall construction quality.
Advanced protection
In my article called Where’s the beef? Why we need quality, here’s what I said about a dev’s responsibility to write instrumentation and unit tests:
NO, this is not a tester’s job. Test’s job is to protect the customer from whatever you miss—despite your best efforts. The testing team is not a crutch on which you get to balance your two tons of lousy code. You should be embarrassed if test finds a bug. Every bug that they find is a bug that you missed, and it better not be because you were lazy, apathetic, or incompetent.
However, in fairness to the devs that actually care enough about their peers to consider the impact of unit testing on test jobs, I should discuss more about the role of testers in protecting our customers and how testing may actually evolve if and when devs finally get their acts together.
A change will do you good
In The modern odd couple? Dev and Test, I pointed out three primary ways testers protect our customers:
- Finding bugs we miss.
- Living and breathing quality metrics (mostly from Product Studio).
- Running tests in the customer’s environment as opposed to the developer’s environment.
Devs writing unit tests doesn’t change any of these roles for testers.
But testing is changing nonetheless. Almost all new test hires are expected to know how to write automation code and white-box tests. The more cynical among us may think that there is a secret plan to eliminate testers after all tests are automated. However, the real reason for enhanced automation resides in the time it takes to run all the necessary tests to pass an urgent security, privacy, or reliability update, or even to just validate a build. As teams try to push quality upstream, automated tests are essential to ensure construction quality before check-in.
That brings us to the next key turning point. As the quality of your code at check-in increases, how testers perform their role changes. Note that their role is the same—find missed bugs, analyze quality metrics, and run in the customer’s environment. It’s how they do it that’s different.
The twilight zone
Right now finding bugs in checked-in code is akin to spotting a coffee shop in Seattle—they are hard to miss. Sure, you write a test plan and test cases to exhaust all the different possibilities. But the expectation is always, “Just run the application, the bugs are sure to be there.”
As teams push quality upstream, finding bugs after check-in becomes more difficult. At first, teams simply test buddy drops as part of their check-in criteria. A number of teams are doing this now, including some in established groups such as Windows and Office.
Eric Aside A buddy drop is a private build of a product used to verify code changes before they have been checked into the main code base. That way the impact of unstable code on other teams is minimized. Developers share the private build only with their “buddies”—that is, their teammates.
But as dev teams begin to use disciplined practices and measurements to predict and control their bug counts, the number of bugs found in a buddy drop should plummet by a factor of a thousand. Thus, the typical Microsoft team of 15–20 devs that produces 3,000–5,000 test bugs per year would instead produce 3–5 test bugs per year. When buddy builds get that solid, test needs to learn some new tricks. (I talk about how some teams have achieved these low bug counts using cool software engineering principles in A software odyssey—From craft to engineering.)
Commander Data
Remember, test’s role is to protect customers by finding bugs that devs miss, analyzing quality metrics, and running in the customer’s environment. In a world where code handed to test has only a handful of total bugs, the challenge is to take test’s three responsibilities to the next level.
With such low bug rates, testers should no longer be able to easily find bugs in isolated components or common configurations. Instead, they need to focus on full customer system scenarios and realistic customer configurations, further leveraging their unique customer perspective. While test teams do this a fair amount today, it likely will become their primary focus in the future.
The other key change is in analyzing quality metrics. For devs to get low bug rates, they must collect quality data from the beginning of design, through compile and build, and then use it to know what mistakes they are making, how best to find them, and when they can confidently say they’ve found 99.9% of them. Someone outside of dev needs to be checking the dev’s assumptions, ensuring that the data is reliable and accurate, keeping devs honest and customers protected. Of course, that’s the ideal job for test.
A side effect of this rigorous dev process is that test will have far more quality data, in far greater detail, than they’ve ever had from Product Studio. The more data you have, the more there is to analyze and discover. You can almost hear testers salivating.
Eric Aside Most good testers I know are data fanatics. A perfect afternoon for them is spent up to their eyeballs in Excel spreadsheets analyzing data in different ways.
While devs will do some analysis, this is really a showcase area for testers that entails
- Conducting in-depth statistical studies.
- Noticing key weak spots and trends.
- Finding new ways to improve quality or efficacy even further.
Testers can become the quality process and analysis kings in this new world.
It’s quite cool—I assure you
When devs graduate to engineers, testers can start leveraging comprehensive data and process analysis to assure quality. In others words, test becomes quality assurance. Quality control testing doesn’t go away, but quality assurance grows into a major role. This same pattern has happened in other industries and is due to happen to software. If we don’t make it happen, we will get beat.
So sure, most of this seems like a pipe dream now. Even though some teams are starting to see this level of quality engineering, it’s a long way from being part of our largest systems (like Windows). But if your team is beginning to make headway, or if you know forward-looking testers who want to lead the next wave of software quality, suggest that they start learning about in-depth statistical data and process analysis. It could prepare them to take our game to a whole new level.
Eric Aside While we haven’t yet gone from five thousand bugs down to five for our large complex projects, the focus on quality assurance and data is now happening in a number of divisions. One particularly interesting case is Bing. Bing changes daily (and I don’t just mean the background picture). If test were trying to do pure quality control, the test matrix would be astounding. Instead, Bing test does quality control in key areas and spot checks for quality assurance in the rest. That includes intense aggregate data analysis, where a substantial portion of Bing engineering is directed.
Be First to Comment