Aggregation aggravation

Data is a big part of our professional lives these days. Great teams use it to analyze decisions, detect incidents, and maximize our productivity. However, even with all this big data, great teams still make bad decisions, miss incidents, and seem unresponsive. How can that be?

Some people say we need more data. Maybe, but there’s so much already. Some people say we need more charts and alerts. Maybe, but our current charts fill up walls of monitors, and our existing alerts are overwhelming and hard to maintain. Some people say we need to focus on key indicators. Maybe, but it sure seems like we’re doing that today. And yet, sometimes our decisions, incidents, and responsiveness negatively impact customers.

Oh, wait! Customers! That’s right—we’re here to serve and delight customers. That’s what is missing from our data—customers. Customers are aggregated in our data to protect their privacy, which is important to everyone. However, aggregate data may hide customer variation. In 2020, there were 1.9 kids per family in the US. That aggregate value is useful, but the value is lower for single-parent families. It also breaks down differently for different family incomes and different parts of the country. Similarly, a little-used feature (in aggregate) may be loved by new customers, a highly available service (in aggregate) may be down for everyone in Atlanta, and a low bug count (in aggregate) may include a year-old pain-point for customers who take lots of photos. We’ve got the data, the charts, and the indicators; we just need to reacquaint ourselves with our customers.

The missing link

Let’s look at three examples in detail to see how aggregation is hiding customer insight.

• You think you’ve made a great data-driven decision. A recently released feature isn’t getting the overall engagement you’d hoped, but your site is healthy and gaining active users, so you kill the feature to devote engineering time elsewhere. Shortly afterward, you notice that your active-user increase has subsided. It turns out your recently released feature was attracting new customers. Your aggregate feature usage stats were accurate, but if you had sliced the data by key customer types, such as new active users, you’d have seen the impact and kept the feature.
• You think you’ve got great automated alerts that detect incidents in minutes. Your site is healthy and active with no high-priority incidents for weeks. Suddenly, customer support lights up, and you’re getting a call from your VP claiming the CEOs of Coca-Cola and Delta airlines are furious that your service is down—how did you miss this? It turns out some edge certificates in Atlanta, home to Coca-Cola and Delta, were stale yet weren’t detected. You fix the certificates and add checks across all regions, but you also wonder how you missed the drop in availability. Your aggregate health checks were nominal, but if you had sliced the data by key customer regions, an alert would have been triggered and would have detected the incident immediately.
• You think you’re keeping your bug counts down and technical debt low. Your team can respond quickly to feature requests and fixes. Yet marketing and business development folks are livid about your lack of support for customers who take lots of photos with their phones or cameras. They point to nasty online forum threads talking about long-neglected bugs. Your aggregate bug count was low, but if you had sliced the data by key customer scenarios, such as photography, you’d have raised the priority of the old bugs and fixed the problem.

The data is there. The charts and alerts are there. The key indicators are there. What’s missing is slicing the data, charts, alerts, and indicators by key customer attributes, such as type, region, and scenario. How do you pick the attributes and use them to slice the data? Read on.

Who are you?

Who are your key customers, and how are they differentiated? That’s the fundamental question you should answer to determine how to slice your data.

In the data-driven decision example, your team was concerned with engagement (increasing monthly active users). You measured the usage of a new feature (well done), but you didn’t slice that data by key customer engagement types, such as loyal longtime users, new active users, occasional returning users, and untapped users in new target markets.

In the automated alerts example, your team was concerned with availability. You could have differentiated key customers by amount of business (small, medium, and large usage) or by region (major cities or distribution points). Slicing your data either way would have triggered the alert.

In the bug count example, your team was concerned with customer experience. Key customers are differentiated by how they use your product—that is, by customer scenario. That’s how you should have sliced your data.

Regardless of how you categorize your customers, you want the number of segments to be small enough to manage yet large enough to differentiate customer needs and desires. Five to 16 customer segments are usually enough to differentiate your customers while preserving their privacy and maintaining a manageable portfolio.

You look familiar

You’ve chosen customer segments that align with your business goals and plans. Now you need to slice your data by customer segment for charts, alerts, and indicators. How is that done?

Sometimes the process is simple. You can tie your application telemetry directly to customer attributes that align to segments. You update your chart, alert, and indicator queries, and you’re done.

Sometimes the process is complicated. Perhaps there’s no connection between your telemetry and customer segments, either due to privacy concerns or data architecture. You could rearchitect your system, but that’s costly and time-consuming. Instead, you can use customer proxies—available data that is closely associated or correlated with your customer segmentation. For example, say you own an internal service and segment customers by organization. It’s hard to access org data, but orgs typically share the same repo or service accounts, so segment by those instead. Say you own a third-party service and segment customers by region, but you don’t have direct access to customer demographics. You can still segment by their request’s IP address or edge endpoint.

Regardless of the customer segmentation or customer proxy you consider, be sure to choose one that is stable over time and protects the privacy of individual customers.

I could lose a few

Once you’ve linked your charts, alerts, and indicators to your key customer segments, you’ll notice that some charts, alerts, and indicators quickly become your favorites. They provide the clearest signal and greatest insight. Your other charts, alerts, and indicators add context and help you discern details, but you usually reference them only after your favorites expose an issue.

Having favorite charts, alerts, and indicators is a good thing, since slicing your data by customer segment multiplies your available views, which can get overwhelming. Focus your balanced scorecards and reviews on your favorites, and leave the rest for reference when you drill down as needed.

I can see clearly now

Data aggregation is essential to protect privacy and spot trends. However, when it disconnects you from real customers, you can easily make poor decisions, miss critical incidents, and appear unresponsive to your customers. Use your product plans and goals to determine who your key customers are and how they are differentiated. Divide your customers into five to 16 segments that differentiate the customers’ needs and desires. If you don’t have data to directly map customers into segments, use proxies that are closely associated with each segment. Slice your charts, alerts, and indicators by customer segment to gain deeper insight into decisions, catch incidents that aggregated data misses, and stay responsive to every customer segment. Simplify your scorecards and reviews by focusing on the charts, alerts, and indicators that provide the clearest signal.

It’s aggravating to spend so much time on data collection, analysis, and reporting only to have aggregation hide the key information you need to serve your customers well. By slicing your data into customer segments aligned with your business strategy, you regain connection to your customers and their experience. These days, staying connected is more important than ever.