A Data Product's North Star Metric

If you’ve ever heard an executive ask “How will we know if it is worth it?”, you’ll also remember the uncomfortable silence and blank stares around the room. Slowly, people start to offer up suggestions, “we’ll have saved this much money” or “we’ll be able to do something new we couldn’t before.” Inevitably someone says, “it is just the right thing for us to do.” The meeting ends, usually with an agreement to come back at a later date to answer the “worth” question. Sometimes this sounds like “we need to find our north star.”

We all want to know what success means. No one walks into their day saying “I don’t want things to go well.” The challenge is agreeing on what success looks like. Investing in data products is no different. Data products are often hugely expensive (into the millions) either because of third-party vendor costs or internal team investment (engineering, product, data). No matter what company you work at, this “how will we know if it is worth it” question is coming for you. Are you ready?

Based on what I’ve observed, and my own struggles in this area, the answer is probably not. While I don’t love the concept of a north star metric, I think the exercise of identifying “one” metric to evaluate product success/growth does some useful things. First, it focuses the product team. Many product teams use multiple metrics because they aren’t forced to prioritize, not because they actually need multiple. Second, it creates an easier path to messaging to the organization. Other business units have an easier time evaluating growth through a simple lens. Third, if the north star becomes “out of touch”, it is much easier to notice that with one metric rather than when juggling 5+.

The challenge with identifying a data product’s north star is that we are dealing with some unique challenges: the products are often internal facing, the products don’t always have an obvious connection to revenue and we are more likely to measure external user behavior carefully than our internal users. Here are some ideas/considerations that hopefully help move your conversation about a data product’s north star forward:

1. If we took this data product away, what would our customers lose? The key to a data product (and any product) is identifying its differentiating factors or value add. A north star metric should reflect this value add - otherwise, why are you doing it? This question helps you hone in on “what” the product adds for the user without requiring you to immediately come up with a numerical measure.

2. What data related processes/outcomes are we trying to make better for our customers right now? I’ve noticed many teams get caught in the trap of trying to identify a north star metric that will “live on” forever. But most teams operate in a place where they are trying to change something specific about their product/user experience. Don’t be afraid to make your north star metric specific. It doesn’t need to last 5 years - most people don’t anyway :)

3. Do the customers agree that this measure reflects what matters to them? Sometimes I see data product teams, especially those internal, come up with metrics that sound good in an echo chamber but make little to no sense to the customer. While a customer isn’t the owner of a north star metric, it seems reasonable to think they should be able to see growth/improvement in a north star metric as good for them. Otherwise you’ll be caught in a place where you see growth in a metric and they’ll say “huh?”

4. What can we measure consistently and reliably? This doesn’t mean only looking at events that are captured now. However, many attractive metrics are nearly impossible to measure without sophisticated tracking that itself may take a quarter to implement. A good metric requires that people trust and believe in its accuracy and reliability. Simplicity is better than complexity. Make sure you can provide that.

5. Get commitment around the metric. Sometimes data product teams come up with a great metric. Then they forget to tell anyone about it. At the end of the quarter, they report growth/change and other teams say something like “so what?” Those moments are painful, but avoidable. Once you’ve decided on a metric, make it public (internally). Help others understand why you are choosing to use it and set expectations about the insight it provides. You have to do a little PR - it pays off.

There’s no perfect answer to “How will we know if it is worth it?”, but by thinking about the issues and questions above, you’ll move closer to something that is viable. The north star metric concept is useful to get you started, but it shouldn’t be your end game. After all, you’re responsible for the data product.. The metrics for it should be yours as well.