Debt and investment

 We all have friends or relatives with money problems. There are three sources of those problems: a lack of income, a catastrophe, or a lack of self-control. There are whole industries devoted to solving the income issue—I’m not going to cover that here. Insurance and the kindness of friends and family are built to cover catastrophes. As for issues with self-control, they evade my empathy and rile my wrath.

Friends who live well beyond their means don’t gain more than my condolences when a minor market shift forces a major lifestyle reset. Relatives who spend every penny they earn on instant gratification don’t get my sympathy when a minor setback becomes a major issue. And neither compel my compassion when their situations only get worse over time, even as mine gets better. Maybe I’m heartless, but keeping your debt within your means and investing your resources for future growth seem like responsible rules everyone should know and follow.

So why do so many teams carry enormous bug debt and fail to fix architectural issues, while instead only shipping nifty, new knickknacks? Why do teams ignore investments in infrastructure that enable future growth and productivity, while instead spending all their resources on sexy, slick, fresh features? Why? Because they are irresponsible, short-sighted fools, for whom shipping will become more and more of a struggle, even as my team’s area gets faster and full-featured.

Eric Aside

Ward Cunningham came up with the notion of technical debt back in the early 1990s. The basic idea is that unfixed bugs and infrastructure issues cost you more and more over time, like unpaid interest on loans. The longer you take to fix these issues, the harder they are to fix and the more resources you must devote to working around them. Soon most of your development time is spent working around a poor architecture and stabilizing the product, instead of adding new functionality.

No respect at all

I admit it. Paying off technical debt and investing in infrastructure are far from sexy and lack the instant gratification of cool new feature work. In addition, people who work on refactoring code or enhancing engineering systems rarely get the respect and admiration enjoyed by engineers creating hot new scenarios. It’s not surprising that people choose to ignore debt and investment, but it’s still irresponsible.

Of course, you don’t want to devote all your engineering time to architecture and infrastructure. New features and scenarios are critical—no one buys a product for its newly factored kernel and the amazing internal cloud that tested it. Just as in your financial life, there needs to be a healthy balance between paying off debt, investing in the future, and accumulating cool features you need or want.

Payback’s a big itch

Technical debt can be a nagging problem, but how do you know how much to pay off and when? That depends on the type of debt.

• High priority bugs: Your team should have a minimum quality bar. Any bugs or issues that put your product below that bar will have to get fixed eventually. Therefore, you’re best off fixing them immediately. The longer you wait, the more debt accumulates and the more costly the fixes will be.
• Other bugs: Other issues don’t demand immediate attention. The decision to fix is based on the tradeoff between enhancing the product and raising the bar. Your business situation and customer priorities should dictate your choices.
• Architecture issues: Architectural improvements are worthless in a vacuum. Purity for purity’s sake is a special form of evil—distracting you from customer value. However, if new feature work is hampered by an area of poorly factored code, paying off the debt for that area will reap great returns. Stay focused, and try not to fix the whole world at once.

Any time you pay off monetary or technical debt, you want real value as a result. Are you saving time and money, or is the product code just prettier? This isn’t a hair salon. The value needs to be there.

Eric Aside

Most people know bugs become far more costly over time. There’s the overhead of logging a bug, triaging it, resolving it, regressing it, and closing it versus just fixing it immediately. There’s the added research and debugging time necessary when the code is no longer fresh in your mind. There’s the entanglement of workarounds that pile up over time. And there’s the indicator of bad design that a bug can represent, which if left alone allows the bad design to entrench itself, becoming far more expensive to repair.

All the cost associated with not fixing issues immediately begs the question, “Should you simply fix every bug instantly, regardless of priority?” You’d certainly save the bug logging overhead, the code would always be fresh in your mind, and fewer workarounds or bad designs would fester. Unfortunately, the question is somewhat pointless since bugs will always be found later, and those bugs will be more costly to fix. Nonetheless, minimizing time spent on bugs is worthwhile, so immediately fixing as many of them as you can should pay off.

The future

Paying off technical debt invests time now in order to save even more time in the future. You also want to make other investments for future growth:

• Enhancing build, test, and deployment systems for faster performance, shorter cycle times, and greater ease of use.
• Improving code review and check-in tools.
• Automating more steps in the engineering process.
• Simplifying access to customer data, analysis, and engineering metrics.

Where do you find time to make this investment? There are two common approaches:

1. Devote an early period in the product cycle to reducing technical debt and investing in infrastructure. People call this time “Milestone Q” (for quality). Milestone Q, aka MQ, is typically run during early product planning, when many front-line developers and testers have little else to do. However, you don’t want a wild free-for-all—you want to focus on real value. MQ works best when a cross-discipline team prioritizes and distributes the work to be done and then validates that the quality improvements are themselves of high quality.
2. Mix infrastructure investments in with feature work. Prioritize investments just you do features—how much value do they deliver relative to the work necessary to complete them? Once you get past the “Aaaahhhhhh!” critical features, you’ll find that many investments are quite comparable in value to additional feature work. A balanced mix sets you up for long-term success.

Using either or both approaches works. Find the approach that best fits how your team operates.

It’s important to me

Whichever way you choose to work on infrastructure investments, prioritize each one based on your team’s biggest headaches:

• Is it a gnarly tar pit of spaghetti code that must be untangled?
• Is it an error-prone deployment procedure that must be automated?
• Is it a constantly broken build that needs check-in tests?

Choose the biggest source of pain that you believe can be relieved quickly with focused effort. That’s your top priority investment.

Anyone who claims you don’t have time for infrastructure is flinging male bovine excrement. You don’t have time for all your feature work either. Heck, you never have enough time for just about anything. Life and work are about prioritization. Infrastructure investments can change the game for your product and your team, giving you far more time in the future. Ignoring them will only leave you further and further behind.

I will remember you

Annual review is coming soon, so you’re probably wondering how you get proper credit for blasé debt reduction and infrastructure investment. The key to visibility is to track the before and after.

• If you were fixing bugs, how many did you fix? Even better, how far did you extend mean time to failure, or what percentage of Windows error reports did you resolve?
• If you were speeding up builds, what was the percentage of build time you cut?
• If you were automating engineering steps, how much did you reduce the cycle time?

If you simply fix things and claim victory, no one will care. After all, we shipped before, so how bad could it have been? You must know what you’re improving, measure the baseline, and then report the tangible improvement using relatable metrics.

Finally, be wary of any improvement that takes more than a few months to show results that people can appreciate. Those kinds of projects are seen as pure overhead and will often be cut before they show results, or simply have their perceived impact diminished. (“It ought to be good—you spent six months on it!”)

Eric Aside

How do you create real value through dramatic infrastructure investments on short timelines? Steal from others (MS teams or approved open source). The less you reinvent, the more you can accomplish within the same time period. Remember, it doesn’t have to be perfect—it just has to be impactful.

I deliberately keep my teams small so that they can’t afford to create new stuff from scratch. They have to reuse and repurpose to survive. At review time, this means that everyone on my teams did important work (no time for fluff), contributed significant value (by utilizing the work of others), and did so quickly with few teammates (super-efficient). My teams love this routine because they are constantly completing impactful improvements, and my org adores it because my teams seem to do more with less.

Pay now

When it comes to technical debt, you can pay now or pay later—with interest. It’s true, fixes are not sexy work, but the more you take on early, the more time you have later to add sexy features at a faster pace.

As with any investment of time and resources, you want real value for your efforts. You don’t fix every bug or architectural issue, just the ones essential to meeting your product’s quality bar and unblocking your team’s feature development.

To prove the value you deliver with debt payments and infrastructure investments, you measure the before and after in terms people appreciate (time, effort, cost). You deliver that value quickly and efficiently by utilizing all the great prior work available at Microsoft (or from open source with the right legal due diligence).

In the end, we produce higher quality products more quickly with less effort, continually improving by building on each other’s advances. It’s a proven path to prosperity that keeps on growing and giving. Be responsible. Be successful. Be part of it.