Data Products Aren't All the Same: 5 Critical Examples

I saw a job posting on LinkedIn this week for a data product manager at a big tech company. My first reaction was “cool!” My second reaction, after reading more deeply, was “what is this person supposed to actually do?” The job description was so generic to the point that I had no idea what product they were actually responsible for. Then it hit me - just like product management frameworks tend to abstract away details to fit “general concepts”, talking about “data product” in a generic way can produce the same result. Data product is a useful idea, but to make it really create value, we have to get into the specifics - and that means talking about types of data products.

Why is it so important to get specific? Because there’s no way a company or group can succeed by doing data product in a generic way. We do not create value with generic frameworks, we create value by making individual products higher quality, more useful and more impactful for the business. That can come in many forms depending on the business: in some places that is improving the sustainability of dashboards, in others, it means making a transformational data platform for all analysts across the company. One size does not fit all.

So, let’s get specific. What are these different data products that we should be thinking about? I’ll focus on 5 examples that I’m most familiar with (and that I see repeated across industry right now). As always, I’m here to start a conversation, and that means this list is *not* comprehensive. I’d love to hear from you about the data products that you manage and think about day to day.

1. Experimentation platform. I’m a bit biased because this is where most of my headspace at Yelp goes, but the ability to have a real product experimentation platform is transformational for many companies. Whether this product is internal (like it is at Yelp) or external (with many options like Amplitude Experiment, Launch Darkly or Optimizely), the experimentation platform enables companies to identify causal relationships between product changes and user behavior (or other metrics). This space is *extremely* competitive right now, and I’m a huge fan of many startups that have launched here. For data product managers, experimentation platform requires a vast array of context: from experimental design, to cohort allocation, to logging, to guardrails metrics, to experiment readouts.

2. Machine learning platform (and models). ML as a product has picked up dramatically in the past 3-4 years, and many companies are grappling with how to build, serve, manage, update and monitor ML assets that are in production. As with experimentation, an ML platform is not just one product, it is a set of many data products that enable the business to create value from machine learning. The ML talent market is already competitive, and product minded ML engineers or data scientists have a huge advantage here - they understand the technical details of modeling, tuning and monitoring that enable them to build really useful data products at scale.

3. Product metrics. The word metrics strikes a combination of fear, confusion and admiration into people. For those who work closely with data, the challenge of creating, maintaining, updating, monitoring and deprecating metrics is well understood. Given how many people use metrics to understand business health and make critical operating and strategic decisions, treating metrics as a critical data product has never been more important. In fact, I’d argue that before ever doing something like experimentation and ML as a product, metrics need to be fairly well solidified and managed. Product managers successful in this space tend to have a deep understanding of how the business operates, while understanding the affordances and constraints of what metrics can do. I believe that treating metrics like a data product results in fewer, more streamlined metrics, rather than increasing N.

4. Product analytics. I’ll start by saying that product analytics has about 50 different definitions depending on who you ask. For now, I think about product analytics as leveraging data to create informed hypotheses about user (or customer) behavior. Most often, companies cannot succeed at product analytics broadly speaking. Success happens within vertical areas of the business (e.g. product analytics for marketing, product analytics for consumer growth). Treat product analytics like a product isn’t about maintaining the UI or ensuring people are logging in to use the product on a specific time schedule. It means ensuring that the right data is available to the right people so that they can leverage it to make informed hypotheses and decisions (potentially leading to an experiment).

5. Data quality. If I had a dime for (okay quarter, given inflation) each time I heard about the need for data quality yet little action/investment to make progress on it, I probably would be on an island not writing this newsletter. Fortunately, (or unfortunately for you, maybe) products that support data quality haven’t permeated the market yet. But they will, much sooner than we think. The reason I mention data quality here is that for any of the upstream products I mentioned above to succeed, reliable and well documented data is not only nice to have, it is foundational. My belief is that for data quality efforts to succeed, very few companies can build the human power they need to do it all in house. They need to leverage external solutions. But the success of an external solution is not guaranteed - it depends on a data product manager focusing on its success.

Remember that job posting that I mentioned above? It didn’t say anything about what type of data product the company was interested in managing. Because of that, I can guarantee they got less interest and engagement (and fewer qualified candidates) than they would have with a specific call for experimentation, ML or analytics. As you think about building a data product capability at your company, think about being specific about what you want. No one can win in every domain all at once. We need to carefully sequence our investments to see success.

Thank you so much for reading and engaging!