I love Microsoft®. We’ve been together happily for many years. If you’ve been in a healthy long-term relationship, then you know what this means—there are things about Microsoft that make me curse, stomp, and spit. I’ve learned to tolerate them, but they still make me cringe.
A prime example is our disrespect for critical disciplines like testing. The test discipline is one of the two largest engineering disciplines at Microsoft and one of three key engineering triad disciplines. How can we not grant testers the same respect and opportunities we give the other two engineering triad disciplines—program managers (PMs) and developers? Perhaps our history provides the answer.
Microsoft was started by developers and run by developers for years. We’re now run by a former program manager of sorts, so PMs receive begrudging respect. Developers can’t draw and have no sense of style, so designers are becoming appreciated. Developers hate writing, so content publishing is at least considered necessary. Localization and media are magical things that just appear in the build. But test? Developers believe testing is easy, if not remedial, so developers think testers are beneath them.
Eric Aside
Please note that I’m not giving any of these disciplines proper credit for all the work they do. I’m merely listing the superficial ways they are often viewed by developers.
It’s a different kind of flying altogether
Since developers think testing is easy compared to development, they think they can do a tester’s job. After all, isn’t that the cool, agile way? Aren’t we all just software engineers? Yeah, and everyone would get along if we just gave peace a chance. Don’t be naïve.
Developers can verify that their individual components work as specified in isolation (unit testing). They have much more trouble verifying that their components work as a system, outside of isolation and spec. Why? Tunnel vision. Developers design and write their code for a certain purpose. That’s the way developers think about it—as they should.
Real-world testing must verify that the code works appropriately when used in ways that defy all logic and purpose. To test code properly, you need to completely forget how and why it was written and instead think about how it might be used in arbitrary and even insidious ways. Developers don’t think that way—they can’t think that way and still develop with purpose. Testers do think that way. That’s why we need testers.
That’s easy!
Even though the test discipline is essential for high quality software, some groups still consider converting all their testers to developers, expecting the combined development team to write all tests—it’s a great way to get more developer headcount! The teams that actually go through with this change experience the following problems:
- They lose their test leads and leaders. The former development leads tend to lead the combined engineering teams. (I talk about why in the next section.) This relegates the former test leads to individual contributor roles (ICs), who usually don’t take this reduction in responsibility well and leave. The former top test ICs also tend to leave—after all, clearly their skill set is not appreciated, no matter what sparkly purple lipstick the management team might put on this pathetic pig of a plan.
- They lose their testers. After the test leads and leaders go, the former test ICs gravitate toward a development mindset. They gravitate there because testers get no respect, they are calibrated against developers even if they continue to work differently as testers, and they can see there’s no career growth in test. After all, their role models left the team. Now the only path up is through development.
- Team morale drops, especially among the testers. Losing team members and team leaders impacts morale. The most impacted are the testers still clinging to their discipline who lose their role models, their self-identity, and their solid reviews. (Even great apples get bad reviews in an orange grove.)
- Their code quality is initially higher and then gradually drops. The quality improves initially because developers suddenly realize they’ve lost their safety net and actually start writing unit tests, doing design and code reviews, and paying attention to those build warnings. (Of course, developers should have been doing this all along.) However, after a while, the system and boundary errors start creeping in and building up. No one is looking for them, so they are discovered by the wrong kind of testers—customers.
Combining development and test makes sense at the unit level (such as Test-driven Development). This practice can also work at the component level for well-understood and well-factored components on teams that also have strong QA. However, combining development and test doesn’t make sense at the system level. I talked about this at length in Undisciplined—What’s so special about specialization?.
I just can’t get enough
The lack of respect for testers is most apparent in leveling and career development. Depending on where they work in the company, testers are ranked one to three levels below developers and PMs. In other words, the test counterpart to a PM and developer who are all working on the same project with the same scope will tend to be a full career stage below his or her peers. Unbelievable!
“Yeah, but testing isn’t as difficult as development and program management,” says the dim developer. Really? Try it sometime. Try writing automation that works every time, even as developers alter configurations and data. Try performing penetration testing that closes gaps that foil sophisticated hacks. Try producing injection testing that discovers failure modes, system testing that finds sneak conditions, or end-to-end scenario testing that validates the billion-dollar bets we make. And I’m just scratching the surface.
“Yeah, but our test team isn’t that advanced—they don’t do all those things.” Exactly! We don’t value the test discipline enough to advance our testing capability to the same levels as PM and development.
Testing is frigging hard! We need great people to do it well, and then we need to pay, develop, and expect them to be world class. Our treatment of the test discipline is astonishing and pathetic. Believe or not, it used to be even worse until senior test leaders from across the company started tracking the numbers and driving promotions.
Yes, it’s true that testers do different jobs from PMs and developers. But we can’t design and construct complex systems and think that testing those systems will be any less complex. You get what you pay for, and by spending less on testing, we create imbalance. We sacrifice quality, productivity, and efficiency as a result. The sacrifice in quality is obvious. The sacrifice in productivity and efficiency comes from incomplete and fragile testing that results in higher error rates, more rework, and higher support and sustained engineering costs.
Eric Aside
The refrain I hear when I complain about our commitment to testing is, “Yeah, but Google and Amazon have far fewer testers.” Amazon carefully tests the systems that matter, like billing and account management. Google does endless analysis of its search results.
As for Microsoft, we are in a broader business than Amazon or Google. We are a platform company and an enterprise company. Our customers expect more from us. Quality is a differentiator and a necessity. While we don’t need perfection in all things, we do need the right quality in the right products.
The world just wants us to fit in
It’s bad enough to give up quality, productivity, and efficiency by expecting less from our testers than we do from their PM and developer counterparts. What’s worse is that lowering expectations sends a clear message that to get ahead, a tester needs to become something else. This philosophy is nothing short of irresponsible and tragic.
- Lowering expectations for testers is irresponsible because some of our most challenging engineering problems involve testing—testing in production, testing many-core highly-parallel systems, testing 1000+ machine services, testing globally distributed cloud services, testing secure and private cloud systems, testing hybrid procedural and functional languages, testing natural user interfaces, and on and on.
- Lowering expectations for testers is tragic because we send a message that testing is not a legitimate career path, when in fact it is a robust career path to the highest engineering stages for both test managers and test ICs. Instead of following this path, testers abandon their discipline to do something else, often with mixed results.
By expecting less from our testers, we are encouraging them to move away from a career they love—one that is essential to Microsoft’s success and offers tremendous opportunity—and toward a career not of their choice that may inhibit their growth. It is a travesty.
Eric Aside
Microsoft recently announced changes to its compensation plans that increase the base salary for most engineers, including testers. That’s wonderful and I’m grateful. Investing in our test discipline is a separate matter. It is choosing to be just as sophisticated in testing as we are in program management and development.
Executives might reasonably ask, “Why would we increase our spending in testing? What’s the return on investment?” While I believe a strong financial return exists, I’d turn that statement around. If we want to save money, why don’t we decrease how much we pay developers by a couple of levels? Because our quality and innovation would suffer. Why would they suffer? Our products are platforms with wide and varied usage that require sophisticated engineering to orchestrate and improve—and test.
Tell me what it means to me
At a time when we should be investing in test, we continue to demean the discipline. We have great leadership at the highest levels within the test discipline, but far too few testers join these ranks each year. Even though test is far behind PM and development, they receive a smaller promotion budget. (The promotion budget is equal in proportion to the other disciplines, but that proportion is a distortion due to the higher salaries in PM and development.) Only a handful of test ICs and test managers reach the principal and partner stages to serve as role models.
How can we allow a critical and central engineering discipline to be so disrespected and damaged? Are we that vain or foolish to think testers aren’t really needed or the problems aren’t really that difficult?
It’s time we put our money where our priorities are and push testing to the next stage. We hire the best—let’s challenge them accordingly. Let’s lay out the test career path all the way through to vice president and technical fellow. Let’s start aggressively recognizing the talent we have and developing the talent we need. Testing deserves our respect—our customers, our partners, and our business depend upon it.
Eric Aside
What can you do to help if you aren’t a test or multidisciplinary leader?
- Accept and appreciate that the test mindset and skillset are different from development, yet their problems are equally complex and critical to our product quality.
- Write high quality code from the beginning using design and code reviews, code analysis (like PREfast and FxCop), and thorough unit testing, all of which allow testers to focus on their unique value of providing quality assurance and exposing system issues developers wouldn’t normally detect.
- Encourage your test team to hire great fulltime testers that focus on the truly challenging test problems we face—problems that when solved will improve the quality of our products, our testing, and our testers.
BTW, I could write a similar column on service engineers, who are even less understood than testers.
Wow, thank you for this article! I'm a developer, and I've seen that far too often the test team is spending their time finding product bugs that devs should have found in the first place. No wonder they don't get the time to do all the coll stuff your article talks about. Devs need to step and do their job well so that test can do their job well too!
Eric, do you think it is because of the missing respect for the test discipline and the reasons you mentioned above that the latest updates in the Office family tend to break things? Is Microsoft gearing to the wrong direction?
blogs.technet.com/…/issues-after-installing-powerpoint-2002-update-kb2464617.aspx
blogs.technet.com/…/issues-after-installing-powerpoint-2003-update-kb2464588.aspx
blogs.technet.com/…/issues-after-installing-outlook-kb-kb2509470.aspx
All this is true, but there's one more reason why there are very few people in senior positions in test: most engineers with computer science background want to be developers. The lack of respect the test career gets at microsoft is but one reason why you see all those SDETs interviewing for SDE positions after a couple of years in test. You know job descriptions for test positions contain the ubiquitous quote "Do you have a passion for breaking things?" For most engineers the full answer would be "Yes, but I have an even bigger passion for building things."
It's a triangle: Dev <-> PM <-> Test, and the true story is that Devs and PMsdo not need Test in day to day scenarios. It's the Customer, who needs the Test, that means only one thing: If we miss Customer alltogether in that scheme, we miss obvious necessity of Test.
I agree both with Eric and Fritz. Since SDET positions are not valued and involves a lot of manual work that most SDETs transition/interview for an SDE position after a year. A typical day of an SDET involves setting up machines, jobs, and manual tasks to perform testing. SDETs also do not get to code nowadays. They have SDEs who develop Frameworks and Testing tools/suites for SDETs, who just need to use them to do their testing.
Great post!
A few of us in our org have been really trying to push Test as a DISCIPLINE as opposed to Test as a department down the hall where code in varying degrees of completeness is pitched in order to find the bugs that should have never been checked in in the first place.
Test is highly complex. It involves detailed analysis, measurement, experimentation, etc; yet most groups at MS treat Test as the place where you can get your code sanitized before it ships. No wonder we lose so many Testers to Dev/PM. No wonder we end up shipping sup-par quality code.
I envision Test as a place where an SDET can come in, can design and implement solutions to complex TEST problems; not just bang away at a keyboard/mouse, or write brainless automation. We do have SDE in our title, after all! We should be considering how a complex system can have health metrics embedded in it, and be implementing that.
I could keep on going, but I'll let others continue. 🙂
This is spot on.
Nice job Eric. Great awareness of the Test reality, coming from a Dev lead 🙂
I've been saying much of this for years, but it seems to fall on deaf ears – there is an unhealthy inequality embedded in our triad design, in that Test is always coming from "below" in any discussions. That can be resolved somewhat by the tester presenting things confidently and demonstrating their "equal understanding" of the code and customer scenarios, however there is an inherent disadvantage to the Test org based on levels alone, as you say.
What should be investigated further, in my opinion, is the real value, in terms of product quality and customer satisfaction and company reputation, as well as relationship health between the discipline triad IF we were to take what you have said in this column seriously, and provide leadership that demands respect on the Test side. Leaders who teach those in the Test organizations to require (and command) that respect and demonstrate/educate the other disciplines on "how hard it is" to do the Test job, where you can code at the level of some of your Developer peers, and can understand the customer's perspective as well as some of your PM peers, but due to the demands of the test role (mentioned above – setting up machines, then providing those repros in debuggers to your Dev counterpart; performing many manual tasks that neither of the other disciplines are required to do (which typically take an excessive amount of time)). As a result, you cannot make the time to do the same "high priority" tasks as much as your counterparts are able to. Said in other words, the PM and Developer disciplines are much more narrowly focused, where they are able to "specialize" in something they do well, and not be plagued by their work being "diluted" with necessary tasks that go under-appreciated (unless they are not done, of course) – my opinion at least.
If those who can make a difference understand what Eric has said in this post, we could potentially realize huge gains by making effective use of talent that does exist in the Test organizations, and which could be strongly developed (my own test org is already doing some of this).
We need leaders who "see" this vision!
What would happen if test and dev worked more as "equals"? Dev would gain from understanding what makes product code more testable, and would likely have more of an appreciation for thinking in a "system" approach toward customer use of the product (as opposed to having a more narrow focus only on their "unit" tests, and thinking of the test organization as their "sanitizing" function). At thet same time, Test would better understand the product code and be more capable of creating the complex, well-designed testcode that is more flexible, capable of adapting to changing requirements typical of software "development".
From just a lowly SDET who will likely have to be leaving the company soon 🙂
All this time I was thinking it was only happening in my organization. This article has made me realize we SDETs are being treated the same everywhere.
It is sad. It shows why we don't even grow because of the other manual non-SDET related tasks we have to do. At the same time we get treated as if those tasks are useless and can be done by anyone. Coding is also not done much among tests and you'll notice how so many SDETs quench their thirsts by being part of the Garage community. Some of them have even come up with far more innovative projects.
The other trend I have seen is that some move to Google. One of the SDETs I knew from the Garage moved to Google in the beginning of this year.
It is time to wake up before we lose all the talent!
Thanks Eric for saying this as it is.
A couple of observations:
1. An SDET's job is primary one of evaluation. Guess what – its harder to rate (and therefore reward) the evaluation. Not impossible – just harder than being able to showcase delivered pieces of code or accurate schedules or pretty user experiences. It is also the easiest to cut corners for and the most likely to be done purely for the sake of being done.
2. From a strictly CS pov, verification of a piece of code is the same order of complexity of building a system which could have automatically generated it. US schools tyically don't do a good job at Formal Methods. At MS, it shows – in culture as well as in execution.
3. At MS, when you don't have Senior Test ICs who can handle hard verification problems, but still build product code with considerable complexity without taking #2 into account, then you perpetuate the cycle by relegating unimportant tasks to SDETs. If there is a hard problem and you really need to nail it for a release – then you either hire someone senior enough who is unlikely to be in Test.
4. We suck – absolutely suck- at engineering Test infrastrucure. (There are exceptions in large & mature orgs) If a typical SDET is not expected to deal with hard problems as in #3, why waste money on investing in infrastructure that would boost his productivity?
5. Its 2011. We are still talking about test automation. The state of the art is building systems that generate Tests. But hey, where are all the Senior Test ICs who would be developing these techniques?
6. For a V1 or even a V2, features are important. How well they work really are not. Thats a myth many folks in MS buy. And the public buys it too. Wait till MS ships an SP1.
7. An SDET at MS, is not equipped to do a good job at validation – i.e ensuring that we are building the right product. His day to day job does not promote it. Unless you fix this, it will be extremely difficult to grow Senior Test ICs.
Totally great Jack's opion, especially, seems like our senior tester did not do good job to make our test automation truely automated.
Well there is no need to value test like a dev or pm. Only to an extent. Joel on software compares how a Dev gets $100/hr compared to $30/hr a test should get http://www.joelonsoftware.com/…/fog0000000043.html. So yeah it is useless to value a test as a dev.
Here is another explanation on yet another article http://www.joelonsoftware.com/…/fog0000000067.html. He explains:
"This one is painful. It's very hard to hire good testers. With testers, like programmers, the best ones are an order of magnitude better than the average ones…
Unfortunately, most people who are that smart will tend to get bored with day-to-day testing, so the best testers tend to last for about 3 or 4 months and then move on…
No matter how hard it is to find testers, they are still cheaper than programmers. A lot cheaper. And if you don't hire testers, you're going to have programmers doing testing. And if you think it's bad when you have testers churning out, just wait till you see how expensive it is to replace that star programmer, at $100,000 a year, who got sick of being told to "spend a few weeks on testing before we release" and moved on to a more professional company. You could hire three testers for a year just to cover the recruiter's fee on the replacement programmer."
So yes Devs are way, way, way better than tests. It is Better to throw away THREE tests than throw away a SINGLE Dev. There can't be any better comparison.
It's important to understand why it's critical to have great talent in Test, especially because it is possible to release software with immature and ineffective Test teams (by working harder and longer and cutting few corners on quality)
Dev work is basically a "close problem". There's a set of requirements and the objective is to create code that implements it. There are only so many ways to do that. Surely you need to worry about elegance and perf and maintainability, etc. and it's hard. But…
Test deal with an open question. The deliverable of Test is the verified quality of the software. The tests are just a tool to generate this outcome. Whether you developed thousands or millions of tests to get to this result it's still the same outcome. So the tester need to figure out a way to get to the result in a creative way and minimal cost. An experienced tester can design a set of tests successfully. A brilliant tester can identify tests that can be avoided without lowering the confidence in the quality.
Dev in contrast rarely deal with the cost constraint, if the estimated cost doesn't fit the schedule the feature is tossed out of the plan
I also believe that when it comes to the quality of the engineering system, senior testers should take a lead analyzing the efficiency of the processes and methodologies, and this requires the smartest engineers.
This is not to say that test is more important than Dev. To be effective we need all disciplines to work in a concerted and synergetic way and everyone needs to work smart. Just bear in mind that with great talent in Test magic can happen.
Responding to Sk's post. there are some dumb ways to do Testing (for $30/h) which we should avoid if possible. There are well formalized test activities like Unit Testing and those can (should) be done by Dev. I don't know if you consider this boring or interesting activities for Dev, but I know it's basic engineering practices that we should follow. There are creative ways to define the test strategy and this is definitely not boring even if it doesn't involve writing code. I'm sure that many successful developers can fail miserably if they were responsible for defining the test strategy. The Arithmetic you suggest of hiring 3 testers for 1 star developer is misleading and in fact reflects badly on this star developer. Here's how I read your statement: it takes 3 testers to clean after one careless developer. Why would I want to do that and where does it get us if we really want to release more often in the future? We dealt with a similar question in the past. We figured out the cheap manual testing is simply too expensive and we need to pay more for testers that can write automated tests. It's time to push the engineering system further ahead with the best testers.
It is a great article. I am a Senior Tester, and like to share some my thought on this. I will always suggest
many Friends to choose dev career than tester career because
1) there are more dev jobs than testing jobs
2) tester career is hard to grow than dev in many companies inclduing MS
3) tester sometime spend too many times on things other than coding and writing tests which I think
it is the most important thing tester should do.
Regarding to transform testers into dev, I think it has advantages as well
1) we change our title (that is a realy great thing).
2) we work more closely with dev
3) we reduce the size of test team. I think we have too high tester v.s dev ratio than any other company. Sometime, test team and tool team started their bussiness to grow their team which will result in too many test framework, tool many sign-off. We foget the reason of a test team exists is that it help to improve quality and help us to sale our product. Any work not related to this reason should be reduced or removed.
Finally, I believe that test team and dev team should closely work together as a team.
Eric, I appreciate the fact you mentioned service engineers, even if it was at the end.
The essence of your discussion relates directly to those of us in the service discipline too. In order to develop SDETs or Service Engineers that can successfully work the toughest development projects or customer service / infrastructure issues, you need to have a meaningful career path and compensation. Oh yeah, and it's nice to get a shout out every once in awhile too.
If you want any ideas for that column on service engineers, let me know 😉
I used to work as SDET in Microsoft and left Microsoft for the reasons mentioned above. Lack of respect for tester and the role is downgraded to more of setting up machines, helping the lazy dev's to setup their machines and other mundane tasks. The review system is another killer where testers are calibrated against dev's and it's more of how well and nice you are with the dev that matters the most. Most of the so called test leads left in Microsoft are nothing but talking c*cks who talk their way into the position. The good ones definitely leave for greener pastures where their skill sets are valued the most.