Ever walk through a public space, like an airport terminal or public park, and get accosted by crazies trying to convert you or scare you or assault your supposed ignorance? Get into a conversation with one of these people and logic and reasoning become ludicrous. Everything to them is blind faith and irrefutable truth. Even if you wholly agree with them, there is still no room for questions or analysis. You must believe, you cannot question, even in part.
This makes me sick. I mean truly physically sick. I was given a mind of my own, and I fully intend to use it. Not just at parties and social occasions but on every subject and dealing I have. Questioning why and understanding how are at the center of who I am.
You’d think my sensibilities would be the norm for software developers, who can’t debug what they don’t understand. But the same zeal that some folks devote to religion, political battles, and environmental concerns also gets directed by some developers toward new development practices like eXtreme Programming (XP), Agile, and the Team Software Process (TSP).
All things in moderation
I love many of the ideas and approaches advocated by these development paradigms. But if I question a true believer why a certain thing is done or suggest a small change in a rule or practice to better adapt it to my work, look out! It’s like showing a ring of power to an old hobbit—the fangs come out, the hair raises on end. For some developers, eXtreme Programming and the Agile Manifesto have become a cult. For some developers, TSP is a measure of allegiance—you’re either with us or against us.
Well excuse me for being practical. Excuse me for using my head. Excuse me for doing something because it’s useful instead of magic. I don’t do things because “That’s the way you must do it.” I do things because there’s a darn good reason why they work, and there are also good reasons why working some other way fails.
Eric Aside There, I feel better. Often these rants that lead columns overstate my own feelings on a subject, but not this time. There’s harmless superstition and then there’s lunacy. I’m not a big fan of lunatics.
Waste not, want not
Which brings me to Lean. Ah yes, the title of the column. While there are many wonderful things in XP, Agile, and TSP, there is at least one concept that they all have in common: reduce wasted effort. That is the focus of Lean Design and Manufacturing, a concept from Toyota that predates XP, Agile, and TSP by more than 30 years. While XP, Agile, and TSP attack the problem of waste in different ways, we can better understand what each is doing by using the Lean model.
So, at the risk of offending some zealots’ sensibilities, let’s break it down. Lean focuses on delivering as much value as possible to the customer with a minimum of wasted effort. It accomplishes this by using a pull model and adopting continuous improvement. The pull model means simply, “Don’t do work until it’s needed.” This reduces unused, unnecessary, and undesirable work. The continuous improvement is focused on reducing waste and creating a smooth-flowing stream of customer value.
Eric Aside Kudos to Corey Ladas who first introduced me to Lean, as well as Axiomatic Design, Scrum, Quality Function Deployment (QFD), Set-Based Design, Kaizen, Pugh Concept Selection, and who knows how many other great ideas. We worked together for two productive years, and he’s left my team with a hole that cannot be easily filled. He’s got a Lean Software Engineering Web site now with another great former team member, Bernie Thompson.
Lean defines seven types of waste that disrupt the flow of customer value:
- Overproduction
- Transportation
- Motion
- Waiting
- Overprocessing
- Inventory
- Defects
These are obviously manufacturing terms, right? They can’t possibly be relevant to software, right? Oh, to be young and foolish. All seven of these sources of waste are directly related to software development. I’ll treat them like the seven deadly sins and talk about how XP, Agile, TSP, and plain common sense can help.
Overproduction
The first deadly waste is producing more than you need. Like this never happens. Has a product ever shipped without cutting features that were already spec’d and coded? Has a product ever shipped without keeping features customers never use? Too complex, too general, too extensible, too fancy, too redundant, too convoluted. Overproduction is a killer. It’s an unbelievable waste.
XP solves this with short and tight iterations. It insists on constant contact with customers and constant communication between developers. This ensures that everyone knows what others are doing and the customer always thinks it’s a good idea. As a result, almost all the work that gets done is of value to the customer. Of course, the Microsoft customer is supersized, so many Microsoft teams have turned to Agile.
Agile is a collection of Lean practices, including XP. Because Agile is more of an alliance than a specific technique, it provides a number of interesting approaches to development. One of these is a project management practice called “Scrum” (named after the rugby term). Teams meet with the customer’s representative regularly, usually every 30 days, to demonstrate progress, reprioritize items, and make process improvements. As with XP, team members also meet daily to keep tabs on each other’s progress and any blocking issues.
By reprioritizing work monthly and reorganizing work daily, a Scrum team tunes itself to only what’s important to the customer. Little work is wasted. By focusing on process improvements at regular intervals, the value stream can be constantly optimized.
Go deep
Of course, you can use Scrum and XP poorly by making the customer wait for value while you work on “infrastructure.” There is a fundamental premise behind quick iterations built around regular customer feedback: develop the code depth first, not breadth first.
Breadth first in the extreme means spec every feature, then design every feature, then code every feature, and then test every feature. Depth first in the extreme means spec, design, code, and test one feature completely, and then when you are done move on to the next feature. Naturally, neither extreme is good, but depth first is far better. For most teams, you want to do a high-level breadth design and then quickly switch into depth-first, low-level design and implementation.
This is just what Microsoft Office is doing with feature crews. First, teams plan what features they need and how the features go together. Then folks break up into small multidiscipline teams that focus on a single spec at a time, from start to finish. The result is a much faster delivery of fully implemented and stable value to demonstrate for customers.
Eric Aside Naturally, the idea of feature crews isn’t new. However, finding a way to implement Lean software development within a huge live production environment like Office is a major achievement. Keep in mind, Office is now a system of 15 desktop applications, 8 server applications, and 2 major online services.
Depth first reduces overproduction by staying focused on work that can be used, rather than on “infrastructure” that may never be leveraged or a little bit of everything that may never stabilize. Another great method for depth-first development is Test-Driven Development (TDD), but I’ll save that for the overprocessing section.
Transportation
The second deadly waste is waiting for stuff to arrive. In manufacturing, this typically means the transportation of parts. For software, it’s the transportation of deliverables between teams. There are three nasty sources of transportation issues: builds, branches, and e-mail.
- Builds The longer the build, the bigger the waste of time. Like I need to tell you this. XP and Agile both insist on daily builds, a rule they may well have gotten from Microsoft. For huge teams, a daily build has become a fantasy. Luckily, we have good people working on the issue, but it’s a big problem. Enough said.
- Branches I love Source Depot. It’s been huge for the company. But the frigging thing has become a pet elephant. Sure they’re cute when they’re a baby, but in a few years you’re constantly either feeding or shoveling, and your mobility suffers. While branching is great, many large teams have taken to branching branches. So if you are on branch A2.B3.C1 and your buddy with a key feature or fix is on branch A3.B1.C2, your buddy needs to reverse integrate C2 into B1 then B1 into A3, and then you have to integrate A3 into A2 then B3 then C1. AHHHHHHHHH!!!!!!!! You might as well watch grass grow. The solution is one level of branching off your current release line, period.
Eric Aside Source Depot is the large-scale source control system Microsoft uses to manage hundreds of millions of lines of source code and tools, including version control and branching.
- E-mail The last transportation nightmare is e-mail notification: PM telling dev and test that specs are ready; dev telling test that code is ready; test telling dev it’s blocked on a bug; dev telling PM it’s blocked on a design change; and, my personal favorite, any kind of communication between a client and dependency or vendor, particularly overseas. XP and Agile solve the e-mail notification problem by removing the roles and having the team meet daily. For remote vendors and dependencies, this can’t work. For now, we must rely on automated notification where possible, Live Meeting where reasonable, and clear e-mail that answers anticipated responses to reduce roundtrips everywhere else.
Motion
The third deadly waste is spending time just finding stuff. On the manufacturing floor, it’s the wasted motions of robots and people. In the software world, it’s time spent figuring out what to do, where to go, and how to fix. Poor search technology is a great example of wasted motion. So is untestable, unmaintainable, unmanageable code.
Using asserts and validating input help find bugs faster and reduce wasted motion. So do design reviews, code reviews, code analysis, and unit testing. XP even suggests pair programming, but personally, I think that wastes resources (except for devs learning a new code base). TSP measures all your activities and defects, which allows you to study exactly how your time is spent and significantly cut down on your wasted motion.
Eric Aside My team has since adopted pairing for creating new content in unfamiliar areas. It works extremely well.
One particularly annoying and avoidable source of wasted motion is duplicating bug fix information for code comments, Source Depot, Product Studio, and check-in mail. And everyone wastes motion managing multiple copies of bugs and project schedule data. Tools that make these things easier by entering the information once and automatically populating it to all other places can go a long way toward reducing deadly motion sickness.
Waiting
The fourth deadly waste is waiting around for work. Transportation issues cover a big part of waiting for builds, branch integrations, and timely communication. But there are plenty more places to wait. The most common dead zone is caused by teams not agreeing on the priority order of features or simply not following the predetermined order. That is, PMs writing specs out of order so devs have to wait. Devs writing features out of order so testers have to wait. Testers writing tests out of order so everyone has to wait.
XP, Agile, and TSP all force teams to decide on a priority order, get buy-off from the customer or their representative, then work in that order until they decide to review the priorities again. TSP is particularly rigorous in this way, but also can be less iterative about plans without a flexible leader.
Another source of waiting is unstable code. As long as the code is unstable, the test team has to wait, as do any other mechanisms you have for customer feedback. XP and Agile put a premium on verifiably stable code, another essential element of the depth-first strategy.
Eric Aside Another form of waiting is for service environments to stabilize or unfreeze as the result of new service deployments or other special events. The best way to avoid this wait time is exposure control and continuous deployment, which I discuss in There’s no place like production.
Overprocessing
The fifth deadly waste is over-engineering. You see this all the time in the form of producing overly complex features, fine-tuning performance in areas that already perform adequately or aren’t the true bottleneck, and adding generalization or extensibility when it isn’t required. This waste is related to overproduction but focused on specific feature implementations.
The cure: Test-Driven Development (TDD). TDD is an XP and Agile technique for implementation design. As a side benefit, it provides unit tests with full code coverage. The process is fairly simple:
- Define your API or public class methods.
Eric Aside This is a point of contention between me and some members of the Agile community: do you define your API or public class methods before writing unit tests or after? Purists will say after; I say before. The difference is the amount of up-front design and the nature of your relationship with outside groups that depend on your code. I tackle up-front design in other columns, which when taken in moderation, I believe is essential to success in projects with more than 100,000 lines of code.
- Write a unit test for a requirement of the API or class.
- Compile and build your program, and then run the unit test and ensure that it fails. (If it passes, skip step 4.)
- Write just enough code to make the unit test pass. (Also ensure that all previous unit tests continue to pass.)
- Repeat steps 2 through 4 until all API or class requirements are tested.
Naturally, after you get the hang of it, you can write unit tests for more than one requirement at a time; but when you first get started, try doing just one. It builds the right habits.
When you use TDD, you don’t write any more code than is absolutely required. You also automatically get easily testable code, which usually correlates to strong cohesion, loose coupling, and less redundancy—all very good things indeed. Oh, and did I mention you also get unit tests with full code coverage? What’s not to like?
Inventory
The sixth deadly waste is undelivered work product. This is related to cut features, but it also includes the amount of work in progress. When you develop breadth first, all your work is in progress until the code is complete and stable. All the completed specs, designs, and code that are waiting to pass tests are inventory. Their value is not yet realized.
Unrealized value is wasteful because you can’t demonstrate the value to customers and partners. You can’t get their feedback. You can’t improve and optimize your customer value stream. Of course, if product plans change, this unrealized inventory often becomes a huge wasted effort.
The Lean pull model of working only on things as they are needed drives low inventory, as demonstrated in Scrum and TDD. Scrum pays special attention to work in progress, tracking it and working hard to minimize it. Scrum also leverages regular opportunities to improve and optimize the way you deliver value. TDD has you implement code only as needed to satisfy requirements, and no more.
Defects
The seventh deadly waste is rework. It’s the most obvious one and the one I’ve ranted about incessantly in the past (see Where’s the beef? Why we need quality). XP and Agile get at reducing bugs and rework by a variety of techniques, not the least of which is TDD, daily builds, continuous code reviews, and design reviews.
However, XP and Agile also reduce bugs in a more subtle way—by creating a structure where you learn as you go. Using depth-first development, you figure out parts of the project step by step before you’ve designed and coded the whole product. This prevents serious architectural issues from remaining concealed until it’s too late to adjust. Sound familiar?
Reducing defects is a specialty of TSP. Teams using TSP have dropped their bug rates by a factor of a thousand from the industry average. I wrote in detail about the TSP defect prediction, tracking, and removal approach in A software odyssey—From craft to engineering. While TSP isn’t inherently lean, it doesn’t preclude depth-first development either.
Symbiosis
This brings me to the point where I get to infuriate the XP, Agile, and TSP true-believers. There’s no reason why you can’t combine these techniques to be greater than the sum of their parts. Use Scrum to drive a lean, depth-first, flexible, and optimized development schedule. Use TDD to create a lean implementation. And use TSP to analyze your defects and your work, which will result in vastly reduced bugs and wasted effort. While that may be heresy to some, it sounds like common sense to me.
Now if I could just find some good pastrami.
Eric Aside I grew up in New York. It’s tough to find good pastrami in Redmond.
Be First to Comment