The Agile bullet

 I’m having a tough time with a decision; maybe you can help. I can’t quite decide who is more nauseating: people who use “Agile” methods and wonder why Microsoft can’t adopt Agile across the company, solving every ill we face; or people who think the Agile fad amounts to retreaded foolishness preached by ignorant academics to free developers from any sense of responsibility. It’s a toss-up; I get the same squeamish feelings in my gut listening to either of them.

Eric Aside This is one of my favorite columns because of the overwhelming love-hate reaction it evoked, often from the same person. Though imperfect, it’s a fairly balanced overview of the topic.

Let’s get two things straight right now:

• If you think Agile methods fix all that is wrong with how we build products, you are, in fact, a fool. Employing thousands of people to build highly complex and deeply integrated software that hundreds of millions of customers depend on is hard. No one in the world, including those clever folks in the Agile Alliance, knows as much about the task as we do. Not everything we are doing is wrong, and not everything Agile professes is right for our needs.
• If you are an anti-Agile curmudgeon who thinks Scrum is an acronym for a System of Clueless Reckless Untested Methods, you are as much a fool and just as ignorant. Dismissing anything thoughtlessly for whatever reason is prejudicial and unprofessional. Grassroots movements, like Agile, are always grounded in some fundamental truths that can be used to benefit our teams and customers. Those notions may not always fit our business directly, but fundamental truths have a way of applying themselves to any situation when you stop to understand them.

Eric Aside Agile has really been a grassroots effort at Microsoft, led by a wide collection of individuals and small teams throughout the company.

It’s time to expel the myths around Agile methods and explain how to use the innovative thinking behind these methods to our advantage.

Enemy of the truth

First, let’s break down the Agile myths…

• Myth #1: Agile = eXtreme Programming (pair programming, Scrum, Test-Driven Development, user stories, or some other Agile method). Agile methods are actually a collection of software development practices that share a common set of defining principles but are otherwise unrelated and at times contradictory. You can learn more about what Agile really is from the Agile Alliance.
• Myth #2: Agile methods can’t work for large groups. This statement is absurd. Agile is a collection of disparate methods. Some of those methods won’t work for large groups, some will, and some can if you get creative. You have to study the specific method in question before jumping to inane conclusions.
• Myth #3: Agile methods can work for large groups. The Agile philosophy values “customer collaboration over contract negotiation” and “responding to change over following a plan.” Customer collaboration is tough with over 100 million customers. Contract negotiation is essential to manage cross-team dependencies. (See My way or the highway—Negotiation.) Following a plan is required for business commitments because partners get touchy when millions of dollars are involved. Applying Agile methods to large-scale projects requires you to be flexible and creative to deal with these issues.
• Myth #4: Agile means no documentation. The Agile philosophy values “working software over comprehensive documentation.” Many Agile zealots read this and say, “Yay, no documentation!” If you think the world ends where your hallway ends, you don’t deserve a cut of revenue generated beyond your walls. The Agile philosophy states, “While there is value in the items on the right, we value the items on the left more.” In other words, working software is valued more than documentation, but essential documentation is still valuable for customers, partners, and cross-group dependencies.
• Myth #5: Agile means no up-front design. The Agile philosophy values “responding to change over following a plan.” Many Agile zealots misinterpret this to mean, “No need to think or plan; the design will just emerge!” Emerge from what—a radioactive sewage dump? The point is to value responding to change over taking your original plans too seriously—it’s not to jump off a cliff and see what happens next.
• Myth #6: Agile means no individual accountability. The Agile philosophy values “individuals and interactions over processes and tools” and “responding to change over following a plan.” Many terrified managers think this means zero accountability. In fact, Agile has an interesting twist in this area. Agile makes the individual accountable to the team and the team accountable to management. Accountability is strongly emphasized, but the extra level of indirection allows Agile teams to be more efficient, resilient, and…well,… agile.
• Myth #7: Scrum is an acronym. This is a silly myth, but it drives me crazy. Scrum is one of the best-known and most widely practiced Agile methods, but it is not an acronym. Scrum is named after the rugby term that describes when the teams get together, arms latched in a circle, trying to obtain possession of the ball. It also is the name of the daily standup meeting used by Scrum teams. At Microsoft, we’ve been using a form of Scrum for decades—well before the term existed. It is one of the simplest Agile methods and the closest to what many Microsoft teams already practice. More on Scrum later.

Get the rules straightened out

Talking about Agile in the abstract makes for entertaining debate, but applying it is where the action is. Because we’ve established that Agile is actually a collection of software development practices, the question remains, “Which ones work well in large-scale projects?” Many people have thought and written about this question—but many people don’t write this column. Before I give my opinion, some ground rules…

• No change for change’s sake. If a team is already working well by all the measures the business cares about, there’s no need to change. Change is costly no matter how nice the result might be. You should change only to eventually improve. So if no improvement is needed, no change is needed.
• Don’t get carried away. If change is needed, don’t change everything at once. Have feature teams pick one or two improvements each and see how that goes. Not every team needs to change simultaneously, and not every team needs to change identically. Of course, if you are changing a central service, like the build system, then all teams will eventually need to adopt it. But even those kinds of changes can be either spread out or made transparent to individual teams. The idea is this: try a little, learn a little, and then try a little more.
• Differentiate between the project level and the feature level. The biggest area where people get confused—particularly with Agile methods—is differentiating between the project level and the feature level. At the project level, you need firm dates and firm agreements between teams. At the feature level, you…well actually,…whatever. That’s the bizarre idea many managers fail to understand—your team can hit whatever date you care to set; the question is only what features you end up including. As long as the project-level plan can be tracked and followed, your feature teams should choose whatever method allows them to be the most effective.

Eric Aside This is a power-packed paragraph. One caveat: groups generally work better when the small teams within them are using similar methods. The methods needn’t be identical, but teams will work best together if they have the same pacing. Otherwise, coordination and communication get muddled between teams because they have different expectations around timing.

Ready for something different?

So you’re thinking about trying Agile—or perhaps you just want to placate the Agile maniacs in your group with Scrum snacks to go with the hypnotic Kool-Aid the maniacs are drinking. What should you try, and how can you best integrate it into common practices? There are a large number of Agile methods, so I’ll address only the most popular ones: Scrum, eXtreme Programming, Test-Driven Development, pair programming, user stories, refactoring, and continuous integration.

First, there are two methods that we’ve been using for more than a decade at Microsoft: refactoring and continuous integration. Refactoring is simply reorganizing your code without changing what it does. Refactoring is used to break up complex functions (spaghetti code) or to add new functionality to existing code—like changing a class that reads CSV files into an abstract class that could read CSV or XML files. Continuous integration is the philosophy of always integrating new code into regular, ideally daily, full builds so that everyone can test it.

Let the man speak

Next are user stories, which are like a combination of scenarios and one-page specs. The idea of user stories is to provide just enough information to be able to estimate what it would take to implement and test the functionality specified.

The difficulty with user stories is that they are supposed to be written by the user. Many Agile methods assume that the user can regularly hang around with the feature team. Unfortunately, that presents a problem when you’ve got 100 million users.

Like it or not, we need proxies for users. Groups like marketing, product planning, user experience, sales, and support can play that role. Their findings can be codified in value propositions and vision documents that draw from a broad collection of user research. However, as those broad visions and end-to-end scenarios are broken down, we can still use the concept of user stories at the feature level to provide just enough documentation to estimate the implementation and verification of a feature set.

You complete me

Pair programming involves two people sharing a desk and a keyboard and coding together. The idea is that as one person is typing, the other is seeing the bigger picture and catching suboptimal design or implementation. The pair switches off from time to time. While two heads are better than one, they also cost twice as much. I’d rather see the two heads be put to better use in design and code inspections. However, pair programming is great for getting people up to speed in new code bases by pairing developers familiar and unfamiliar with the code.

Eric Aside My team has since adopted pairing for creating new content in unfamiliar areas. It works extremely well.

Aside from refactoring and continuous integration, Test-Driven Development (TDD) and Scrum have proved to be the easiest and most effective agile methods applied at Microsoft. As I described in my column on Lean engineering, in TDD you start with a class definition of functions or methods and then write unit tests for the public functions and methods before you write the code. It is an iterative procedure, in which you are writing only a few unit tests and a little code at a time. The technique is popular because it gives developers the unit test code coverage they need while producing a minimal, yet high-quality, implementation design.

It’s also more fun to write the tests first. When you write the code first, the unit tests are a pain to retrofit and will only give you bad news—not exactly reinforcing. When you write unit tests first, it’s easy to fit the code to the tests and you feel vindicated when tests pass.

Eric Aside While I agree with many practitioners that TDD’s true purpose is exceptional implementation design, the benefits of positive reinforcement for unit testing cannot be overstated.

TDD can be used in conjunction with pair programming by having one developer write a few tests and the other implement enough code to make the tests pass, switching off from time to time. Finally, TDD gives developers a clear sense of when they are done with the implementation: when all requirements have tests and those tests pass.

A bit extreme

eXtreme Programming is a whole development methodology. It combines user stories, pair programming, TDD, refactoring, continuous integration, and a bunch of other practices into a coherent set. It is ideally applied by small teams working closely with their customers.

eXtreme Programming relies a great deal on team knowledge and direct customer interaction, using almost no documentation. This is great if your team is isolated and your customers are down the hall, but that’s not exactly common at Microsoft. Our situation would be tragic if not for the billions of dollars we earn every year. However, as I’ve already mentioned, many of the individual methods used within eXtreme Programming apply nicely to our product development.

Are you ready for some rugby!

The last and perhaps most misunderstood Agile method I’ll cover is Scrum. Aside from people confusing Scrum with eXtreme Programming (which doesn’t really use Scrum) or thinking that Agile equals Scrum (huh?), the most bewildering part of Scrum is all the strange terms associated with it: Scrum Masters, backlogs, burn-downs, sprints, and even pigs and chickens. It’s enough to scare any manager away. Big mistake.

For better or worse, Scrum was invented by a person who enjoys funny names and stories. The practice itself is neither complicated nor contentious. So, aside from refactoring and continuous integration, Scrum is the closest Agile method to what we’ve been doing internally for years, with a few significant improvements.

Let’s start by mapping some of the confusing terms. Scrums are daily stand-up meetings, Scrum Masters are feature team organizers, backlogs are feature or work-item lists, burn-downs are graphs of remaining work, sprints are mini milestones, and pigs and chickens are entrepreneurial farm animals (long story, cute joke).

None of these concepts are new, but Scrum does introduce some big improvements:

• The daily stand-up meetings in Scrum are highly organized and collect useful data. The team organizer (Scrum Master) simply asks all the team members what they accomplished since yesterday (and how long it took), what they are working on until tomorrow (and how much is left to do), and what’s impeding progress.

Eric Aside Tracking how long it took is my team’s small contribution to Scrum at Microsoft. By adding this information to the burn-down data (how much is left to do), you can produce fantastic cumulative flow diagrams, measure time on task and work in progress, and better estimate team capacity. Typical time on task is around 42% for production teams; 30% for teams focused on communication—like mine—and as much as 60% for co-located feature teams.

• The data collected at Scrums is entered into a spreadsheet or database. From the spreadsheet, you can analyze time on task, completion dates, work in progress, plan changes, and a whole host of project issues. One of the most popular graphs is a burn-down chart that plots time vs. total work remaining.

Online materials Sprint Backlog (SprintBacklogExample.xls; SprintBacklogTemplate.xlt)

• The Scrum Master is an independent force on the team. It’s best if he or she isn’t even part of the group, but often that’s not realistic. The Scrum Master has permission to keep meetings short and cut through the crud.
• The feature list or schedule is called the Product Backlog, and the work-item list or schedule is called the Sprint Backlog. By keeping these two lists separate, management can focus on the work they want done (the Product Backlog) while the team focuses on the work at hand (the Sprint Backlog). Typically once a week, the Scrum Master meets with management (for example, at the weekly lead’s meeting) and updates status, ensuring everything stays on track.

Online materials Product Backlog (ProductBacklogExample.xls; ProductBacklogTemplate.xlt) and Sprint Backlog (SprintBacklogExample.xls; SprintBacklogTemplate.xlt)

• Sprints, the mini milestones, are fixed in length. They are over when the specified number of days is over—typically around 30 days.

Eric Aside In the six years since writing this column, I’ve been part of teams that used 1-week sprints, 2-week sprints, and 30-day sprints. Right now, my team uses 2-week sprints, which are my favorite. Two weeks is short enough not to require much overhead—everything can be tracked on a whiteboard. However, two weeks is long enough to get serious work completed. I’m now looking forward to trying Kanban and being on almost a continuous sprint, but that’s another story.

• After every sprint, the feature team reviews its work with management (nice change, huh?); discusses what went well and improvements for the next sprint (a wee bit better than waiting a year or decade until the post-ship postmortem); and plans and re-estimates the work items for its next sprint (the plan and estimates changed? No way!).

By using daily, weekly, and monthly feedback mechanisms, Scrum allows teams to work efficiently and resiliently in a changing environment. By collecting a little key data, Scrum allows teams and management to know how teams are operating and to spot issues before they become problems. By separating the feature list owned by management from the work-item list owned by the feature team, Scrum allows teams to direct themselves and stay focused. It drives accountability to each member within the team and to management outside the team.

The more you know

Not all Agile methods are for everyone, and many won’t work on big Microsoft projects. But Scrum, Test-Driven Development, refactoring, and continuous integration are being used by many Microsoft teams with great effects. Pair programming and user stories are being applied to a lesser degree, but they can be effective in the right situations. As long as you don’t get carried away and start forcing Agile down your team’s throat, there’s a great deal to be gained by applying these methods.

Eric Aside Managers have forced methodology changes down engineers’ throats almost everywhere I’ve been employed. It never works, even for something popular like Scrum. Managers can suggest, support, and subsidize behavioral change, but they should never coerce it.

To learn more, search for Agile methods on our internal network or on the Web. Also watch for new courses on Agile methods coming in spring 2006. If everything is going great on your team, then don’t change a thing. But if you’d like to see a little higher quality or better feature-team project management, you owe it to yourself to take some antacid and give Agile a try.