Generalist vs Subject Matter Expert: Companies Grapple with How to Use Data Product

The Market for Data Product

The number of “data product manager” roles available on LinkedIn has increased at least three-fold since I started writing this newsletter. It has been fantastic to see companies grapple with the need for this type of role and formalize it internally. Yet there is an interesting dichotomy I’ve observed in how companies define and develop the data product manager role. Some go generic, while others get very specific. For example, I saw a post this morning for “data product manager” that “works on all data platforms and makes data come together for downstream customers”. I saw a different post that described the role as “focused on maintaining the performance and strategy for our key ML models”.

As I searched through job ads and internal role descriptions over the past 3 months, I’ve seen this distinction arise repeatedly. Some companies want data product managers to do very generic things with data like “make it higher quality”, “make it more available”, “focus on data platform strategy”. Others want area experts in experimentation, machine learning, metrics, analytics, data governance (among others). Rather than judge whether one is better than the other, I thought I’d collect some data on what I found to illustrate the difference, and let you think about the value proposition of generalist vs SME.

Data Product Roles Specifically Defined

1. Experimentation. I’m a bit biased because this is where most of my headspace goes, but the ability to have a real product experimentation platform is transformational for many companies. An experimentation platform enables companies to identify causal relationships between product changes and user behavior (or other metrics). This space is *extremely* competitive right now, and I’m a huge fan of many startups that have launched here. For data product managers, experimentation platform requires a vast array of context: from experimental design, to cohort allocation, to logging, to guardrails metrics, to experiment readouts.

2. Machine learning platform (and models). ML as a product has picked up dramatically in the past 3-4 years, and many companies are grappling with how to build, serve, manage, update and monitor ML assets that are in production. As with experimentation, an ML platform is not just one product, it is a set of many data products that enable the business to create value from machine learning. The ML talent market is already competitive, and product minded ML engineers or data scientists have a huge advantage here - they understand the technical details of modeling, tuning and monitoring that enable them to build really useful data products at scale.

3. Product metrics. The word metrics strikes a combination of fear, confusion and admiration into people. For those who work closely with data, the challenge of creating, maintaining, updating, monitoring and deprecating metrics is well understood. Given how many people use metrics to understand business health and make critical operating and strategic decisions, treating metrics as a critical data product has never been more important. Product managers roles in this space tend to have a deep understanding of how the business operates, while understanding the affordances and constraints of what metrics can do.

4. Product analytics. I’ll start by saying that product analytics has about 50 different definitions depending on who you ask. For now, I think about product analytics as leveraging data to create informed hypotheses about user (or customer) behavior. Most often, companies cannot succeed at product analytics broadly speaking. Success happens within vertical areas of the business (e.g. product analytics for marketing, product analytics for consumer growth). Treat product analytics like a product isn’t about maintaining the UI or ensuring people are logging in to use the product on a specific time schedule. It means ensuring that the right data is available to the right people so that they can leverage it to make informed hypotheses and decisions (potentially leading to an experiment).

5. Data quality and governance. If I had a dime for (okay quarter, given inflation) each time I heard about the need for data quality yet little action/investment to make progress on it, I probably would be on an island not writing this newsletter. Fortunately, (or unfortunately for you, maybe) products that support data quality haven’t permeated the market yet. But they will, much sooner than we think. The reason I mention data quality here is that for any of the upstream products I mentioned above to succeed, reliable and well documented data is not only nice to have, it is foundational.

Data Product Roles More Generically Defined

Keep in mind here that generically defined does not mean better or worse than specifically defined roles (like above). They are just different.

1. Data platform. The reason I put data platform into this space is that platform itself is a fairly generic and hard to define word. What counts as a platform? Who decides that? It is often company and person dependent. Roles that focus on data platform product often have a broader scope than the area specific roles I mentioned above. In these roles, data is thought about as an asset that the product manager has responsibility to turn into value for the company. This could be through making data higher quality, more available, more easily searchable or well understood. It could even encompass some of the specific areas I mentioned above. My guess is that internally these roles are actually more specific (e.g. it turns into more of a metrics specific role), but that is hard to see when you are on the outside looking in.

2. No definition at all. Simply “data product manager” without further elaboration. These are the areas I worry about the most because I think they setup both the person and the company for failure. I think there are companies out there who value both product management and data and assume that by creating a data product manager role, they will suddenly have a better data strategy and direction. In some cases, the job descriptions make me think they are directly reacting to a perception that data scientists and those working directly with the data won’t be able to manage data as a product.

Which Is Better? It Depends

Every time I say “it depends” I’m reminded of statistics students rolling their eyes at me. I don’t say this to get away with a cheap answer. Instead, it depends on the the company and what value they are trying to create from data. Early in the journey, data product roles might be more generically defined because the company isn’t entirely sure what they need (and what they don’t). As they mature (and their products/focus mature as well), perhaps they better understand the data investments that provide the highest ROI. This could be in metrics, data quality, or experimentation.