Are we having fun yet? The joy of triage.

 Tell me if I don’t have this concept nailed… Program managers want an infinite number of features in zero time, testers and service operations staff want zero features over infinite time, and developers just want to be left alone to code cool stuff. Now, put the leads of each of these disciplines with their conflicting goals in the same room, shut the door, and give them something to fight over. What happens? Triage!

Eric Aside As product development issues arise (such as incomplete work items, bugs, and design changes), they are tracked in a work item database. Triage meetings are held to prioritize the issues and decide how each will be addressed. This can be a source of conflict (understatement).

It’s amazing that blood doesn’t start leaking out from under the triage room door. Of course, that’s what solvents are for. But does it have to be a bloodbath? Most triage sessions are certainly set up that way. Some of the most violent arguments I’ve seen at Microsoft have happened behind the triage door. Is this bad, or is it “by design”?

War is hell

As anyone who’s been through a brutal triage can tell you, it’s not good. Rough triages leave you battered and exhausted even if you win most of the arguments.

Basically, dysfunctional triages go hand in hand with dysfunctional teams. They generate bad blood between team members and often set a course of reprisals and unconstructive behavior.

Why should this be? We encourage passion around here. We want people to fight for what they believe and to make the right decisions for our customers. What’s wrong with a little healthy competition? Well, when it’s not little and it’s not healthy, it’s not good.

It’s nothing personal

Bugs shouldn’t be considered personal, but they are.

• To the tester who found it, the bug represents the quality of his labor: “What do you mean the bug isn’t good enough to fix?”
• To the program manager who wrote the feature, the bug represents a challenge to her design: “It breaks the whole idea of the feature!”
• To the service ops staff, the bug represents real and continuing work: “Yeah, you don’t care about the bug; you’re not the one who’s going to have to come in at 3:00 A.M. to reboot the server!”

Eric Aside Interesting note here about 3:00 A.M. reboots. Like most software service companies, Microsoft is now moving away from service operations being on call 24/7. Instead, we are designing services to automatically heal themselves (retry, restart, reboot, reimage, replace). Service operations people, working regular business hours, simply swap components on the automatically generated replacement list.

• To the developer, the bug represents a personal value judgment: “It’s not that bad.”

Triage decisions should be based on doing what is right for our customers and for Microsoft, not on personal feelings. Yet, because of the personal investment that each discipline places on bugs, triage discussions get off track in a heartbeat.

Five golden rules of triage

How can you keep triage on track and constructive? Follow my five golden rules of triage:

1. Shut the door. Triage is a negotiation process, and negotiations are best held in private. It is far easier to compromise, to bargain, and to be candid when the decision-making process is confidential. It also allows the triage team members to present their decisions as team decisions.
2. All decisions are team decisions. After a consensus is reached, it is no longer the decision of individuals, but of the group. Everyone stands behind the choices as a team—with no qualifications. A triage team member should be able to defend every decision as if it were her own.
3. Just one representative per discipline. Triage must be decisive. Unfortunately, the more people involved, the longer the process; the more personal feelings, the more difficult it is to reach a conclusion. A single individual can make a decision the fastest, but you need the viewpoints of each discipline to make an informed choice. So the best compromise between decisiveness and discipline perspective is reached through having one representative per discipline.
4. One person is designated to have the final say. If the team can’t reach consensus, you need someone to make the call—ideally, this never happens. Personally, I prefer the PM to have the final say because PMs are used to collaboration and realize the consequences dictating decisions. They tend not to abuse the privilege. However, the very threat that someone from another discipline (let alone the PM!) could impose his decision on the team is enough to drive people to consensus.
5. All decisions are by “Quaker” consensus. This is the most important rule. Regular consensus implies that everyone agrees, but that bar is too high to meet for something as difficult and personal as triage. “Quaker” consensus means that no one objects—the team must work toward solutions that everyone can live with. This presents a far more achievable and often more optimal outcome. (Note that “Quaker” simply refers to the people who came up with this notion; it has no religious significance.)

Follow these five rules and your triage will become more cordial, constructive, and efficient. However, there are some subtleties that are worth fleshing out.

The devil is in the details

Here are a few more details that can help your triage run more smoothly:

• If your arguments are about people instead of bugs, change the focus to what’s best for the customer and the long-term stock price. This perspective takes personal issues out of the discussion and puts the focus where it should be.

Eric Aside Throughout the columns, I talk about focusing on the customer and the business, instead of on personal issues. You might wonder why you shouldn’t just think about the customer and leave the long-term stock price out of it. I’m sympathetic to this point of view, but I also know that we don’t get to serve the customer if we are no longer in business. It helps to have a business plan that aligns our work to provide sustainable benefits to our customers.

• If you need extra information about a bug or a fix, it’s sometimes necessary to invite someone from outside the triage team to join you, either by phone or in person. Always complete your questioning and bid them farewell before you begin debating your decision. Otherwise, confidentiality is broken and the decision may cease to be a triage decision.
• If you’d like to teach a member of your team about the triage process, invite him to join a triage session, but instruct him to be a fly on the wall during discussions and stress the confidential nature of the negotiating process.

It’s hard to let go, isn’t it?

If one or more of the triage members can’t seem to let go of an issue, give them a small number of “silver bullets.” The rule behind silver bullets is that you can use them at any time to get your way, but when they are used, they are gone. When a person won’t give in on an issue ask, “Do you want to use one of your silver bullets?” If so, the team is bound to support the decision. Usually the person will say, “Uh, no it’s not that important,” and the team can move on.

Eric Aside This triage column has produced a significant amount of controversy over the years, particularly this paragraph about “silver bullets.” Some complain about using the term “bullet” instead of “token,” but the primary complaint is that a critical team decision could be made by an individual using his “silver bullet.” In practice, this never happens. Silver bullets help people prioritize by associating importance with a scarce resource. People who don’t need the help don’t use their supply. Thus, if someone abused a silver bullet on a critical issue, there’s always someone else with spare tokens to counter. That said, I’ve never heard of this happening.

Finally, when it comes to resolving the triaged bugs in a database:

• Always use the “Triage” label to indicate that this was a triage decision.
• Always explain the thinking behind the triage team’s decisions.
• Never resolve a bug (especially external bugs) unless that’s the last time you want to see it. Too often, teams resolve ship-blocking bugs as “external” or “postponed” when what they mean is, “We don’t want to deal with this bug now, we’ll deal with it later.” But because the bug is “resolved,” it falls off the “active” radar and the issue gets lost.

Eric Aside You can find my column on bug fields, priorities, and resolution values, called Am I bugging you? Bug Reports.

Take care of the little things

Triage is arguably one of the most important duties that you perform as a team. Triage health almost always directly corresponds to the health of the project and of the group. The real beauty of this relationship is that making triage sessions more positive, productive, and pleasant usually leads to the same change in your work and your team. But fixing triage issues is much easier and involves fewer people than fixing entire team and project issues.

The best thing about improving your team’s triage sessions is that when you get it right, it can be the most fun that you have all day. When triage focuses on bugs instead of people and consensus instead of carnage, the stress of the exercise comes out as humor instead of aggression and frustration. Teams working well together often have triages that are filled with wisecracks, inside jokes, twisted ironies, and hilarious misstatements. Make the right adjustments to your triage techniques, and the laughter may be echoing down the halls. Better keep the door shut.