Ninety-four charts
Brindlepath’s main analytics dashboard had ninety-four charts. Logins by day. Page views by screen. Average session length. Tickets by category. Each had been added by someone, for a reason, at some point. Nobody looked at most of them. When the CEO asked, at a board meeting, whether the new ownership queue was working, it took the data team three days to answer, because none of the ninety-four charts measured whether patients actually had an owner.
The problem was not a lack of data. It was a lack of agreement about which numbers mattered, and why. Maya replaced the dashboard with a single page: one measure that captured the value Brindlepath created for customers, four measures that drove it, and a handful of health checks. The page had eleven charts. People looked at it every day.
This chapter is about choosing the numbers that matter, collecting them well, and using them to make better decisions, without pretending that data can make decisions for you.
Data-informed, not data-driven
“Data-driven” is a popular phrase, and a slightly dangerous one. Data cannot tell you what you should want, which customers to serve, or whether a trade-off is worth making. It can tell you, with varying confidence, what is happening and sometimes why. The decisions are still yours.
A better phrase is data-informed: decisions are made by people, using evidence alongside judgment, strategy and customer understanding. Data-informed teams do three things well:
- They agree in advance what they will measure and why, so they are not tempted to find a metric that flatters the result afterward.
- They combine quantitative and qualitative evidence. Numbers say what is happening; conversations say why.
- They know the limits of their data: what it does not capture, how it could mislead, and how confident they can be.
Choosing what to measure
A North Star and its inputs
A useful approach, popularized by the analytics company Amplitude and by growth practitioners such as Sean Ellis, is to identify a single North Star metric: the measure that best captures the value your product delivers to customers, and that leads to long-term business results.1 A good North Star:
- reflects value to the customer, not just activity;
- leads revenue, rather than lagging it;
- can be influenced by the product team’s work;
- is understandable by everyone in the company.
For Brindlepath, the North Star was high-risk discharged patients with a confirmed owner within 24 hours, per month. It captured the core value, a patient someone was looking after, and it led directly to the outcomes customers bought the product for: fewer missed follow-ups and fewer avoidable readmissions.
A North Star on its own is too broad to act on. Below it sit a few input metrics: the measures teams can move directly, which together drive the North Star. Figure 10.1 shows Brindlepath’s metrics tree.
Each product team owned one or two inputs. The integration team owned discharges received within two hours. The navigator experience team owned patients reviewed by 10 a.m. and time to confirm or reassign. The adoption team owned active navigators per customer. Every team could see how its work connected to the North Star, and from there to the outcomes customers cared about and the revenue the business needed.
Other useful frameworks
Two other frameworks are worth knowing.
Pirate metrics (AARRR), introduced by Dave McClure, follow a customer through Acquisition, Activation, Retention, Referral and Revenue.2 They are useful for finding the weakest stage in a growth funnel, especially in products with self-service sign-up.
HEART, developed by researchers at Google, measures user experience through Happiness, Engagement, Adoption, Retention and Task success, each paired with goals, signals and metrics.3 It is especially helpful for measuring the quality of a specific feature or redesign.
Whichever framework you use, the discipline is the same: start from the goal, decide what signal would show progress toward it, and only then choose the metric.
Beware vanity metrics
Vanity metrics go up and to the right and tell you nothing useful: total registered users, cumulative downloads, page views. They are easy to report and hard to act on. The test is simple: if this number changed, would we do anything differently? If not, it is decoration.
Brindlepath’s old dashboard tracked logins per day, which rose every month. It took a while to notice that much of the increase came from navigators logging in repeatedly because sessions timed out too quickly, which was a usability problem, not a success.
Retention: the number that tells the truth
For subscription products, retention is the most honest measure of value. Customers can be persuaded to buy once; they keep using and paying only if the product keeps helping them.
Cohort analysis shows retention clearly. A cohort is a group of users or customers who started in the same period. Tracking each cohort over time shows whether people keep using the product, and whether recent cohorts retain better than earlier ones, which is the clearest sign that product changes are working.
Look for two things in a retention chart. First, does the curve flatten? A curve that keeps falling toward zero means people are trying the product and leaving. A curve that flattens means a group has found lasting value. Second, are newer cohorts higher than older ones? If so, the product is improving.
In business software, measure retention at two levels: the user (are navigators still using it each week?) and the account (is the customer renewing and expanding?). Account retention lags user retention by months, so a drop in user engagement is your early warning.
Analyzing data well
Correlation is not causation
The most common analytical mistake is to treat a relationship as a cause. Customers who use the reporting module renew at higher rates. Does reporting cause renewal? Or do larger, more committed customers both use reporting and renew? Without more careful analysis, you cannot tell.
When the decision matters, find stronger evidence: an experiment, a comparison of similar groups, or a before-and-after analysis with a clear control.
Experiments
A/B tests, which randomly show different versions to different users and compare outcomes, are the strongest everyday tool for establishing cause. They work best when you have enough users, when outcomes can be measured quickly, and when showing different versions is acceptable.
In business software, and especially in healthcare, those conditions often do not hold. You may have a few hundred users per customer, outcomes that take months to appear, and clinical workflows where showing different versions to different nurses is inappropriate. Alternatives include staged rollouts across customers or units, with careful before-and-after comparison; pilot and comparison sites; and simple, well-designed observation of real use.
Whatever method you use, decide your measure and your success threshold before you look at the results.
Pair numbers with stories
Quantitative data tells you what is happening at scale. Qualitative evidence, from interviews, support tickets and session observations, tells you why. The best insights come from moving between the two. When the numbers show that navigators at one customer review far fewer patients than others, a single call with that customer usually explains why in ten minutes.
Building a measurement foundation
Good analysis depends on good data. A few foundations matter:
- An event taxonomy: a consistent, documented set of events and properties that the product records, such as patient_assigned, owner_confirmed and patient_reassigned, with clear definitions.
- Instrumentation as part of done: every significant feature ships with the events needed to measure its outcome, as Chapter 9 recommends.
- One definition of each key metric, written down, so that “active navigator” means the same thing in every report.
- Privacy by design: collect only what you need, avoid sending personal or health information to analytics tools that are not approved for it, and respect customers’ data agreements. In healthcare, analytics events should never contain patient identifiers unless the analytics system is covered by the right agreements and controls.
- Self-service access for the product team to the data they need, with the data team supporting rather than gatekeeping.
Bringing financial and customer data together
The richest insights often come from joining product usage with financial and customer data. Which features do customers who expand use most? Which usage patterns predict churn six months before renewal? What does it cost to serve customers who use the product heavily versus lightly?
Brindlepath’s most useful analysis joined three sources: usage data, support tickets and contract data. It showed that customers whose navigators confirmed ownership for more than 80 percent of high-risk patients in the first 90 days almost always renewed, and those below 50 percent often did not. That single finding reshaped onboarding: the customer success team’s first goal became getting every new customer above 80 percent within 90 days.
Brindlepath in practice: finding the leading indicator
Brindlepath’s most valuable analysis started with a question from the CEO: could the company tell, months in advance, which customers were likely to renew? Renewal decisions came once a year, far too late to change anything. The team needed a leading indicator.
Maya asked the data team to assemble every customer’s history in one place: daily usage by navigator and site, the North Star and input metrics by month, support tickets, implementation milestones and contract outcomes. With only about forty customers at the time, the analysis was as much careful reading as statistics. The team plotted each customer’s ownership rate over its first year, colored by whether it had renewed.
The pattern was striking. Customers whose navigators confirmed ownership for more than 80 percent of high-risk patients within their first 90 days had, almost without exception, renewed. Customers that stayed below 50 percent through the first 90 days had mostly churned or renewed at a reduced scope. The middle group was mixed, and the difference usually came down to whether an executive sponsor had stayed engaged.
The team was careful not to overclaim. With forty customers, the finding was a strong signal, not proof, and it could reflect other differences between customers, such as how well-staffed their navigator teams were. So they treated it as a hypothesis and tested it: for the next ten new customers, customer success made reaching 80 percent within 90 days its first explicit goal, with weekly tracking and a playbook for customers who fell behind. Eight of the ten reached it, faster on average than any earlier cohort, and all eight renewed.
The 90-day ownership rate became the single most-watched number in customer success, appeared on the first page of every account review, and shaped the onboarding work described in Chapter 14. One well-chosen metric, built from joined data and tested before being trusted, did more than ninety-four charts ever had.
Common mistakes
Measuring what is easy instead of what matters. Page views and logins are easy to collect and rarely tell you whether customers got value.
Too many metrics. A dashboard with dozens of charts means nobody knows which ones matter. Fewer, better metrics get looked at.
Deciding the measure after seeing the results. Choosing the metric that makes a launch look good is a common, quiet way of fooling yourself.
Confusing correlation with cause. Customers who use a feature may succeed for reasons that have nothing to do with the feature.
Numbers without stories. Quantitative data shows what is happening; it rarely shows why. A single customer call often explains a puzzling chart.
Leaking sensitive data into analytics. Patient identifiers or other personal data in analytics events create privacy and compliance risk that far outweighs their analytical value.
Try this
Write down your product’s North Star metric and three or four input metrics that drive it, in the shape of Figure 10.1. For each input, name the team that owns it and the date it was last reviewed. Then open your main dashboard and remove, or move to a secondary page, every chart that does not connect to the tree. Finally, pick one input metric that moved unexpectedly last month and call two customers to ask what changed for them. Compare what they say with what the numbers suggested.
How AI changes this
Anyone can ask the data a question. AI tools that turn a plain-English question into a database query or a chart put analysis in the hands of everyone on the product team, not just analysts. That is powerful, and risky. The tool will happily answer a question built on the wrong definition of “active.” Shared, documented metric definitions, the foundation described above, become more important, not less, because AI tools should use them rather than invent their own.
AI finds patterns quickly. Models can scan usage data for unusual behavior, cluster users by how they use the product, and flag early signs of churn. Treat these findings as leads to investigate, not conclusions. Pattern-finding tools are especially prone to mistaking correlation for cause.
AI summaries need checking. An AI-generated summary of last month’s metrics may read beautifully and contain a wrong number. Any AI-produced analysis that will inform a real decision should show its sources and calculations, so a person can check them.
AI features need their own metrics. For AI features, measure quality, not just usage: how often the output is accepted, edited or rejected; how often it is wrong in ways that matter; and what it costs per use. Chapter 17 covers evaluation in detail.