Improving a Data Product: Is it the Data or the Experience?
“So is a data product just anything that has to do with data?” I didn’t get this question from anyone. Instead, it is something that I’ve been struggling with from the moment I started writing this newsletter. The truth is, I’m not sure exactly how to draw the line, though I’ve tried in some previous posts. As I read more about data product and discuss it with others, I’ve started asking a different question - what is the value proposition of data for your product, and how does it lead to improving the product? Here’s why I’ve done that.
About two months ago, I met with a former colleague and as we discussed data product, we started honing in on the idea of how to make a product better. This was specifically about a fraud detection system they were using for an e-commerce related platform. We got around to the discussion of “how to make this product better”. After an hour long discussion, the list of improvements was almost entirely based on model improvements, data quality enhancements and fraud methodologies. It had little to do with the end user experience. I think it is fair to say that the improvements for this product were mostly about the data and models underlying it, rather than any UI or experience.
Contrast this with some of the work I’ve done on experimentation in the past few years. In these situations, there are certainly key improvements to make on the underlying data and metrics (there is always room to do it better). There is a need to ensure allocation is happening correctly and ways to check that the data logging is happening in the ways we’d expect. Yet many of the discussions around improving experimentation also focus on the end user experience. I’ve been part of many discussions about how to make it easier for new users to launch trustworthy experiments. Product improvements are often about the UI, about reading out the results, or providing users more insight into what is happening with an experiment.
I give these two examples above to say that it is easy to define both of the above products (fraud detection system and experimentation platform) as data products. The key question to me is that if you have a data product you must be intentional about what it means to improve the product. “Building better data products” isn’t always about the data and it isn’t always about the user experience. But it could be either (or both in some cases). This is where the role of a data product manager is critical. You must be the person to answer “what does it mean to make this product better” and “what do we need in order to do this”.
In a time where companies are crunched for budget and prioritization is the key to success, there isn’t room to do everything. Most likely, you won’t be able to make everything as good as you want it to. So let’s say you’re in a situation where you have one hire to make. You have 10 things you’d like to make better. Do you need a data scientist? Do you need a platform engineer? Do you need a UX designer? You probably can’t have all three and if you find someone who is all 3, well, please let me know because I’d like to hire them :)
In these newsletters, I typically try to include 5 points or things to think about. But today, I’ve just got one: look at the product you are responsible for. Look at the list of things you wish could be better about it. Then cut that list down to 1 thing. Yes, just one. It might not have anything to do with improving the data and everything to do with the product experience itself.
Why am I taking the time to say all of this? Because the key to being a data product manager is knowing when it isn’t all about the data. At some point, you may get no improvement from technical enhancements. In the coming 12-24 months, we’ll only get to do fewer things, not more. It won’t be all about the data, even if your job is to own a data product.
If one is building a data product for a non-tech business, for example Hospitality, then apart from UX and data, process becomes most important. A lot has to be aligned in the process to see success of the data product
(Note: written while drowsy on Benadryl for a bee sting, please excuse typos).
Re: “what it means to improve the product”
Data or experience; I’d say it’s the environmental goals that beneficiaries of the product have for their environment that influences a data product manager’s focus at a higher level, which turns into outcome goals that customers must be able to achieve through both the user experience and the underlying data platform capabilities.
Environment goals, like “always be aware of and empowered to protect my product domain from harmful experiments,” are not outcome goals (like “analyze impact of a single experiment”), but higher level goals about an invariant quality of the beneficiaries’ desired environment.
The adoption of new unregulated (unmet) Environment Goals by a beneficiary creates the demand for outcome goals to be satisfied, which motivates improving user experiences and the underlying data platform.
The release of environment goals seems rarer, but also causes the release out outcome goals the beneficiary had that aimed to meet the now removed environmental goals.
For a CRM data product, “always know the status of my deals” is an environment goal, it’s not true today and false tomorrow, it’s a desired invariant of your environment, that may not always be met by your current CRM.
Outcome goals are a means to an end, and describe how the beneficiaries’ product experiences can help bring them closer their desired environment.
Example outcome goals for CRM: “get an email when the status of my deals change in the CRM,” “always notify my manager when the status of my deals change but I’m OOO”, etc.
Similarly for clothing, an environmental goal could be “always have ready-to-wear clothes in my closet to create desirable and suitable outfits for my day’s activities that express my identity,” while an outcome goal might be “acquire clothing—to build, maintain, or optimize a type of outfit.”
A key insight: sometimes it doesn’t matter how much better you make a product (internally: data platform, or externally: experience), if the environmental goals of the beneficiary are met or otherwise does not consider the improved ability to reach an outcome goal as an improvement to their desired environmental goals.
Example 1: A novice photographer doesn’t have a strong environmental goal of “never losing a single photo I take.” On the other hand, a pro wedding photographer does have this environmental goal as one of their top concerns when photographing weddings, and they only use cameras that simultaneously use two memory card slots and write every photo to both cards for backup, to be protected against the rare but inevitable corruption of one card’s data.
Adding the second card slot was of value to the wedding pro photographer because their strong environmental need was unmet by cameras using only a single card (due to infrequent memory card data corruption rates), but is not of value to the novice photographer who is just learning to take photos.
Further, the novice photographer may move on to only use an iPhone for photography, instead of a point and shoot or mirrorless camera, because their level of skill didn’t grow their environmental goals, such that mobile phones are actually able to satisfy their more limited environmental and outcome goals. Had most notice photographers been both motivated and empowered through easy photography education to level up their photography skills to a degree whereby their environmental goals would now demand the capabilities of more powerful cameras, the iPhone would not have been considered a competitive option by the beneficiary, because it wouldn’t have met their upgraded environmental goals (such as: always be able to create natural bokeh—the iPhone tries but isn’t as good at this, at least for now…).
Example 2: If I don’t have changing circumstances or motivations in my life for which I need new clothes to create outfits that differ from what I already can create with the clothes I already own, and I am satisfied with my current clothes and their outfits possibilities, then my (described above) clothing environmental goal is satisfied.
If my environmental goals are satisfied, no amount of “improvement” to a clothing store will convince me to buy more clothing, until I either first pick up new unregulated (unmet) environmental goals, or one or more of my existing environmental goals becomes unsatisfied due to a change in my circumstances (change in life stage, change in social events, change in physical demands of clothing, etc) or motivations, such as: develop talent at constructing outfits with more clothes, or a shift from nurture myself with gifts of non-clothing to gifts of clothing (perhaps because it became harder to gift non-clothing, or because the hunt for and choosing of clothing became more enjoyable than the alternative ways to nurture myself).
In summary, to improve the product, one must know which environment goals are unmet and of high value to the beneficiary of the job that involves the product, only then can one assess which outcome goals are unmet and should be prioritized.
—Note: Written on mobile, while on Benadryl for a bee sting, please excuse typos.
PS: My environmental goal of “always be healthy” became unmet, promoting outcome goals to the forefront of my priority to address, such as “reduce swelling”, which led to several product hires (Benadryl, anti itch cream, etc)).