Improving a Data Product: Is it the Data or the Experience?
“So is a data product just anything that has to do with data?” I didn’t get this question from anyone. Instead, it is something that I’ve been struggling with from the moment I started writing this newsletter. The truth is, I’m not sure exactly how to draw the line, though I’ve tried in some previous posts. As I read more about data product and discuss it with others, I’ve started asking a different question - what is the value proposition of data for your product, and how does it lead to improving the product? Here’s why I’ve done that.
About two months ago, I met with a former colleague and as we discussed data product, we started honing in on the idea of how to make a product better. This was specifically about a fraud detection system they were using for an e-commerce related platform. We got around to the discussion of “how to make this product better”. After an hour long discussion, the list of improvements was almost entirely based on model improvements, data quality enhancements and fraud methodologies. It had little to do with the end user experience. I think it is fair to say that the improvements for this product were mostly about the data and models underlying it, rather than any UI or experience.
Contrast this with some of the work I’ve done on experimentation in the past few years. In these situations, there are certainly key improvements to make on the underlying data and metrics (there is always room to do it better). There is a need to ensure allocation is happening correctly and ways to check that the data logging is happening in the ways we’d expect. Yet many of the discussions around improving experimentation also focus on the end user experience. I’ve been part of many discussions about how to make it easier for new users to launch trustworthy experiments. Product improvements are often about the UI, about reading out the results, or providing users more insight into what is happening with an experiment.
I give these two examples above to say that it is easy to define both of the above products (fraud detection system and experimentation platform) as data products. The key question to me is that if you have a data product you must be intentional about what it means to improve the product. “Building better data products” isn’t always about the data and it isn’t always about the user experience. But it could be either (or both in some cases). This is where the role of a data product manager is critical. You must be the person to answer “what does it mean to make this product better” and “what do we need in order to do this”.
In a time where companies are crunched for budget and prioritization is the key to success, there isn’t room to do everything. Most likely, you won’t be able to make everything as good as you want it to. So let’s say you’re in a situation where you have one hire to make. You have 10 things you’d like to make better. Do you need a data scientist? Do you need a platform engineer? Do you need a UX designer? You probably can’t have all three and if you find someone who is all 3, well, please let me know because I’d like to hire them :)
In these newsletters, I typically try to include 5 points or things to think about. But today, I’ve just got one: look at the product you are responsible for. Look at the list of things you wish could be better about it. Then cut that list down to 1 thing. Yes, just one. It might not have anything to do with improving the data and everything to do with the product experience itself.
Why am I taking the time to say all of this? Because the key to being a data product manager is knowing when it isn’t all about the data. At some point, you may get no improvement from technical enhancements. In the coming 12-24 months, we’ll only get to do fewer things, not more. It won’t be all about the data, even if your job is to own a data product.