I would estimate

When I’m discussing challenges with fellow engineers, the first topic that comes up isn’t estimation—it’s career and people challenges. That’s why those issues are so rampant in these rants. However, “How do you generate task estimates?” is always among the top non-moaning-about-your-manager-or-mates topics. After all, estimation is predicting the future. There are so many unknowns and unforeseen issues that it’s impossible to provide the accurate estimates demented despots demand. Isn’t it? It must be. Right?

Wrong. Estimation is among the most trivial tasks an engineer has to perform on a regular basis. Get over yourself, it is. It’s so easy that there are dozens of seemingly different methods that all give you remarkably accurate predictions of completion time. All those methods come down to one simple concept—how long it took last time is how long it will take this time. Nothing could be easier.

Yeah, you’ve got to understand the work well enough to compare it to previous work, but that isn’t too tough either. No, the real challenge isn’t task estimation; the real challenge is accepting the estimate. Estimation is easy; acceptance is hard.

No one would accept the program

Let’s pretend for a moment that you actually keep track of how long it takes you in calendar days to perform various tasks. (You do—the information is right there in your e-mail dates.) Let’s further imagine that you provided those previous times as estimates for doing similar tasks today (you’d be quite accurate). What would the reaction be from your project leads and managers? My guess: “Oh come on, you’ve got to be kidding me!”

This is fun, so let’s take it a step further. Let’s say you told your project leads and managers that your estimates were based on hard dates collected from your previous project. What reasons would they give for not believing this hard data? Here’s the big three:

§  Last time was different.

§  You get faster the second time.

§  Weird stuff happened last time.

Let’s break down these feeble fallacies one at time.

It’s a different kind of flying altogether

The first excuse your manager or project lead will have to reject your hard schedule data from the previous project is that the previous project was different. Things have changed. Perhaps the build system and tools have changed, the design change request process changed, the requirements changed, management changed, or perhaps the moon is in a different position relative to Saturn this time.

Out of all those excuses, only two have a small chance of affecting your estimates—the tool and process changes. Every other factor is superfluous with little or no impact to cycle time.

Even the tool and process changes would have to be extreme to noticeably affect the accuracy of your estimates. Tool changes would have to cut end-to-end build times by a factor of 5. Process changes would have to reduce the time of weekly activities by days. Otherwise, the impact is just noise in the estimate.

Look, the more the world changes, the more it stays the same. Deal with it.

I’m getting better

The second excuse to reject your hard schedule data from the previous project is that you get better the second time around. The funny thing is that you do get better the second time around. The problem is that you’re not doing the same project (hopefully). The only things that are the same are the tools, process, project scope, and the general task of software engineering.

You should already be well versed in the general task of software engineering, so getting better at the details of the previous project has no impact on the estimates for the next project. Of course, if you’re fresh out of school then the second project will take less time than the first.

If you did change tools and processes, your performance should actually be worse because it will be your first time using them. That’s okay if the changes are small or the benefits are big. Just don’t kid yourself about the impact.

You do want to compare the current project to a prior project with similar scope. The better the match, the more accurate the estimate. The big differences between estimation techniques are how they produce matches.

Oh no, not again

The final excuses your manager or project lead will have to reject your hard schedule data from the previous project are all the “weird” things that happened last time. There was that unexpected security patch, the feature that was far more complex than anticipated, the reorganization and associated project reset, not to mention the snowstorm, and that earthquake, yeah, the earthquake. There’s no way you should count the earthquake!

You count the frigging earthquake. There’s always a surprise patch, feature, reorganization, and natural disaster waiting for you over the course of a project. Always. Random events happen, but their impact on the schedule isn’t as unpredictable. Thanks to Lyapunov’s central limit theorem, their overall impact averages out. However long it took last time is likely to be nearly the same this time. That is, as long as you don’t pretend this time will be different.

Same old wine

Okay, we’ve proven your project leads and managers are in denial. As a result, they force you to make ridiculous estimates you don’t believe, only to blame you later for missing them. We’ve shown that accurate estimates are almost trivial to make. The big question remains, “How can you turn your trivial and accurate estimates into ones your project leads and managers will believe?”

That’s where task hours are so handy. Instead of making your estimates in calendar days, you make them in task hours—the number of hours it would take if there were no earthquakes, e-mail, or bathroom breaks. Without those distractions, your estimates look far smaller and more reasonable, even though they’re no different.

Task hour estimates are slightly harder to make because you don’t have the data lying around in your inbox. However, you can estimate task hours quickly, easily, and accurately with a simple technique like planning poker (or it’s more accurate and sophisticated sibling, Wideband Delphi).

In planning poker, three or more engineers each estimate the same task privately around a table. They all reveal their estimates simultaneously so no one exerts undue influence. If the estimates match, you’re done. If they differ, the high and low estimators explain themselves, the group discusses their thinking, and then the process repeats until the estimates agree. The process also surfaces assumptions before they become a problem.

Once you have believable estimates in task hours, the argument isn’t about how long the tasks will take. It’s about how many hours you spend on-task in a week. Even after subtracting vacation, training, and big group meetings, most teams spend less than half of working hours on task. The rest of their time is in meetings, answering e-mail, lunch breaks, and so on. If your project leads and managers don’t believe it, simply have the team spend two weeks tracking their hours. The numbers don’t lie.

Your results may vary

As I mentioned earlier, there’s a certain amount of randomness or “variance” that asserts itself over the course of a project. It averages out, but any one estimate has a chance of being off by some standard deviation. That deviation is a percentage; it scales with the size of the estimate. Thus, a two-day estimate will be accurate give or take a few hours, a two-week estimate might be off by a couple of days (in either direction), and a three-month estimate could be off by a couple of weeks.

As long as you avoid being overly optimistic, the randomness will even out. By the end of the project, your project deviation will be about the same as the deviation from any individual task. If you are overly optimistic (“It can’t take this long next time!”) your deviations will keep adding up, not averaging out.

The point is that there’s little point in estimating a two-day task to the minute or a three-month task to the day. You just need order-of-magnitude estimates, like I talked about in my very first column seven year ago, Dev schedules, flying pigs, and other fantasies.

I want to believe

How would you estimate? Focus with your peers on understanding the tasks at hand and their order of magnitude using a technique like planning poker or Wideband Delphi (poker for well-understood tasks, Delphi for others). That’s the easy part.

What truly matters in the end is believing and accepting your estimates, then scheduling accordingly. As I’ve written about before, over-committing is foolish. What’s worse is that over-commitment can lead to Marching to death. As I said then, death marches are a strong indicator of weak, cowardly, deceitful, and irresponsible management.

Scheduling trouble is diabolical, but completely avoidable. When you prioritize your work properly, putting what’s first first, you reduce pressure on your schedule. When you use your estimates to drive realistic commitments, you can deliver reliably to your customers and partners, build trust, and enhance your group’s and our company’s reputation. Deriving good estimates is easy. Trusting them and yourself is the challenge.