How’s it going?

 Microsoft is broadly and deeply adopting data-driven decision making. This is great to see, but with renewed focus on data comes renewed focus on metrics. As I discuss in How do you measure yourself?, many metrics are faulty or easy to game. You can avoid misleading metrics by measuring desired end results, using baselines and trends to help you discover issues and make informed decisions.

Unfortunately, measures of desired end results are often trailing indicators. They tell you how it’s going after you went. Predictive measures provide leading indicators, but they also can be misleading—they are predictions. To gain a reliable and balanced view of your business success (or failure), you should use a healthy mix of trailing and leading indicators.

For product decisions, nothing’s better than measuring real business results: items sold, revenue generated, and costs accrued. These sorts of trailing indicators can help you run your business. However, will you continue to enjoy those results? Are you as efficient as you can be? What about the health of your product and staff? Can your team respond quickly to your changing market? There are proven measures that address these concerns to create a balanced scorecard. If you’re too focused on trailing indicators (or leading ones), you could end up walking off a cliff.

Better learn balance

Great activity, sales, engagement, and satisfaction today can vaporize with an outage or misstep tomorrow. Low efficiency and high costs open the door for competitors to beat you on price, while slow market response gets you beat on new capabilities, customer loyalty, and new customer segments. To get ahead and stay ahead, you need a balanced scorecard that shows you how you’re doing today and predicts how well you’ll do tomorrow.

There are four basic measures on your balanced scorecard: volume, health, cost, and responsiveness. Let’s break down the metrics for each one.

What are you doing?

The first measure you need is volume. This might seem simple, but it’s subtle. What are you actually producing for your customers? Overall for Microsoft, it might be licenses, purchases, or subscriptions, but for feature teams—the core operating unit of Microsoft—the answer isn’t as obvious. For components, it might be external function calls or times loaded. For development kits, content, and apps, it might be downloads. For services, it might be external requests.

Even after you settle on a measure of volume, you need to decide on how you’ll aggregate the data. Will you add up all calls, downloads, and requests? Or will you sum only a few high-level calls, downloads, or requests to represent the rest? Consider which approach best represents how you and your customers think about your team’s product.

Finally, you need to choose a time period for which the volume is collected. Most teams choose monthly with a daily tracking value. However, some teams have weekly or even daily views with breakdowns by the hour, minute, or second—whatever works best for their business cadence.

Volume is a trailing indicator that maps to revenue and value delivered. Modern teams also typically have secondary measures related to volume, like monthly active users (MAU), engagement (days users are active per month), and customer satisfaction (NSAT), all of which can be leading indicators for future volume. Consider collecting all these metrics, since they each measure an important aspect of volume.

Check my pulse

The second measure you need is health. The most common health metric is availability. It measures the percentage of the time that you delivered as promised. What percentage of calls didn’t result in a crash? What percentage of download attempts succeeded? What percentage of requests got responses?

You’d expect availability percentages to be quite high for a healthy product, typically measured by number of nines (99.9% would be three nines). For services that get millions of requests an hour, availability is usually measured by percentage of time without a failure, with a common target of three nines or better.

Health is a trailing indicator with broad business impact—if you’re not alive, you’re dead. Modern teams also look at trends in secondary measures of health, like number of severe incidents/crashes and mean time to failure (MTTF), which can predict future reliability.

What are your overheads?

The third measure you need is cost. The total cost of producing your product includes subscriptions (like Azure), lab equipment, vendors, maintenance, and you and your teammates. That’s right—you’re not free. In fact, each engineer costs roughly $250K a year (salary, benefits, air conditioning, and the rest), and principal band engineers cost $350K a year.

Much of your costs will remain the same month over month, but as I discuss in It’s business time, all costs are under your team’s control; you decide how efficiently you can and should run. Typically, costs are expressed as currency per unit of volume, like dollars per call, download, or request. Modern teams also typically measure cost per active user. Cost trendlines can alert you to opportunities for a competitive advantage—for you or against you.

What time is it?

The fourth and final measure you need for your balanced scorecard is responsiveness. How quickly can your team respond to a customer issue or opportunity? Response time is closely related to cycle time and throughput. For details with helpful pictures and suggestions for improving your response time, read Productivity mechanics.

Responsiveness is a leading indicator for volume, active users, engagement, satisfaction, and health, and can even drive down cost. Teams also use related secondary measures, like bug count per engineer, average bug age, and code complexity—all of which measure the technical debt that slows your responsiveness and hurts team morale. Reducing debt and shortening cycle time is a recipe for happier customers and happier teams.

I can see clearly now

Years ago, Microsoft teams had no idea how much their code was used, how often it failed, how much it cost to produce and maintain, or how quickly they could respond to opportunities. In hindsight, that’s crazy. Fortunately, that was in the early days of software markets when showing up, being competitive, and providing value was enough. Since then, the software market has matured, as have our competitors. We now have the telemetry and data to better manage our businesses.

Volume, health, cost, and responsiveness interact with each other. More volume leads to more cost and more pressure on health, which can impact responsiveness. Reducing costs haphazardly can reduce health and responsiveness. Improving responsiveness and reducing technical debt can improve health, increase scale, and reduce cost. In other words, you want to track all four measures.

Hopefully, your team has a balanced scorecard, but many teams only look at volume and health. Those that do look at cost and responsiveness often ignore whole categories of contribution to those metrics. It’s up to you to drive your business based on real data. Avoid surprises resulting from ignorance. See the whole picture. You can be balanced or blindsided. Choose wisely.