Product Data Interview: Making Better Decisions With Data

This is part of our Product Data Interviews series, where we ask product managers across the industry the same set of questions about how they use data, what slows them down, and what helps them make better product decisions.

Our interviewee is an experienced product manager with a prior background in software development at a consumer technology company. Their work spanned product discovery, measurement, experimentation, user-interface improvements, and monetization.

This interview has been lightly edited for clarity and anonymized. Company and product names, internal organizations, career timelines, and product-specific operational details have been removed or generalized.

1. What kind of product decisions were you personally responsible for in your previous role?

I worked across consumer-facing experiences, data instrumentation, user-interface redesigns, monetization, testing, and experimentation. I also worked on systems that helped people discover relevant options and supported broader product-integration work.

My background helped me work across both the technical and product sides because I worked as a developer before moving into product management.

2. When you're evaluating whether something is working or worth building, what signals matter most to you?

I prefer a blend of quantitative behavioral data and qualitative user research. Behavioral data matters, but so do direct customer quotes, bug reports, and even emotionally worded feedback. When a customer cares enough to send a detailed complaint, there is often a useful signal underneath the intensity. The exact severity may not generalize to every customer, but the underlying problem can.

I also think companies can over-focus on whatever leadership has labeled the highest priority. Teams need room to build small "delighter" features as well. A product people enjoy can earn more patience when other parts of the experience are imperfect. In an ideal setup, teams would reserve some capacity for promising small ideas.

3. Walk me through a recent product decision you made that involved data. What did the process actually look like?

We were evaluating a product change intended to serve a particular customer segment better. Rather than use company-wide averages, we asked data scientists to isolate that cohort and analyze its specific behavior. The goal was to understand what that segment valued and design an option that would actually fit its needs.

The main lesson was that averages can hide a collection of dissimilar customers whose behaviors may even conflict. Good product analysis starts by identifying meaningful customer groups, learning what each expects, and then evaluating an option against the relevant cohort instead of an undifferentiated average. User research is essential because it helps define those groupings before the quantitative analysis begins.

4. How confident do you generally feel in the data available to you when making product decisions? What tends to increase or reduce that confidence?

Access to dedicated data science and user-research teams increased my confidence. Data scientists could design more sophisticated analyses and challenge assumptions in the numbers, while researchers could provide the qualitative context needed to understand why customers behaved a certain way. Having both perspectives made it easier to cross-check a conclusion instead of relying on a single signal.

In practice, though, my confidence was often low because those teams were spread too thin and frequently redirected by ad hoc executive requests. A PM might wait weeks or months for analysis, and user research could take even longer to reach the front of the backlog. That made it difficult to get evidence on the timeline of a real product decision.

The dashboards were not always maintained reliably, either. For example, a filter could appear to work while still returning unfiltered results. Unless someone noticed that the numbers looked suspicious and asked the owner, a team could make a decision from the wrong slice of data.

Confidence also increases when the underlying pipeline is maintained, filters are validated, definitions are visible, the result can be cross-checked against another source, and the data scientist or researcher has enough time to understand the question. It decreases when data is passed through several layers of storytelling, when the source or agenda is unclear, or when teams are pressured to find a favorable interpretation.

5. What's the most frustrating or time-consuming part of getting the insights you need to make a decision?

The hardest part was combining data across systems. Some information had to be separated by design to prevent teams from seeing competitively sensitive data. Restricting that data was the correct and ethical policy. At the same time, demonstrating a valid business reason and securing approval made access personally time-consuming.

Compounding that, different parts of the customer journey were measured through several analytics systems rather than one end-to-end pipeline. Those systems used incompatible collection methods, which meant their numbers could not simply be joined. A PM might understand one part of the journey but be unable to connect it reliably to another. On top of that, we still had to wait for the data science team to prioritize the request.

6. How self-serve is data access for product managers at your company today?

Basic dashboard exploration was self-serve once a PM had been granted permission. We could use prebuilt dashboards to pull common insights without asking an analyst to answer every question.

The limits appeared when we needed to create a new analysis, validate a dashboard that might be broken, join data from different systems, or do more sophisticated work. That was not self-serve. It required dedicated data scientists, and they generally produced deeper and more reliable insights than a PM could get from the dashboards alone.

The company also experimented with using LLM-based generative AI to produce dashboards and write-ups from datasets. Those tools sometimes hallucinated figures that did not reflect the underlying data.

The problem was not automation, or even AI more broadly. It was applying LLMs to tasks they were poorly suited for without enough validation. The team had not drawn a clear enough boundary between work suited to an LLM and work better handled through deterministic automation, traditional machine learning, or dedicated data scientists. Giving more people access is useful only if the system also provides trustworthy definitions, lineage, validation, and guardrails.

7. What's the hardest thing about turning data into action rather than just more dashboards or reports?

The hardest part was deciding what outcome actually represented customer success, especially when a company-level metric pointed in the opposite direction. Time spent finding something is a good example. A company might treat more time in a product as better engagement, but the purpose of a discovery experience is to help customers find what they need quickly. If people spend less time searching because the product helped them complete their intended action sooner, the lower number may represent a better customer outcome.

There was also pressure to turn every result into a positive story. In one feature experiment, the interaction volume was suspiciously low while completion among those few users appeared perfect. Leadership wanted to emphasize the completion rate. The more likely explanation was that internal testers were overrepresented and instrumentation for real customers was not working correctly.

We also found unexpected imbalances between treatment and control populations after certain filters were applied, which could invalidate the result. But identifying that problem could be treated as obstruction, while a favorable slice was easier to present.

Turning data into action requires an environment where teams can say, "This experiment is inconclusive," discard invalid results, and investigate instrumentation before making a decision. Storytelling should make sound evidence understandable; it should not convert weak evidence into certainty.

8. Are there product metrics or definitions that people at the company regularly interpret differently?

Yes. Engagement time was a common example. If average engagement falls, that could reflect a product change, a discovery problem introduced by a redesign, an outage, seasonality, or a change in customer routines. The metric describes the shape of the outcome, but it does not identify the cause.

People also treated time spent as a clean measure of preference. That overlooks differences in experience length, shared use, activity elsewhere, accidental launches, one-time trials, and changing tastes. A customer can spend a long time with something without valuing it most, or complete a short experience and value it highly. Explicit feedback and implicit behavioral data answer different questions and should be used together.

Experiment results were also interpreted differently depending on sample size and duration. Small expected effects require larger samples or longer tests, and some product effects need time to stabilize. Shipping pressure can make teams want an answer before the experiment has enough statistical power to provide one.

9. What's a surprising or overlooked source of product insight that you think more teams should pay attention to?

Public conversations on social media, forums, and community sites are often dismissed as a vocal minority, but they can be extremely useful. I once grouped a broad set of public comments into recurring themes. That made the feedback much more actionable than a handful of isolated screenshots.

The important thing is not to interpret every comment literally. Look for the emotion and motivation underneath it. If many people express frustration or distrust, ask which unmet expectations could have produced that feeling. Public discussion is organic evidence of how customers frame the product in their own language, and one person who comments may represent many others who feel the same way but stay silent.

10. What advice would you give another PM at a startup trying to make better product decisions with data?

Always ask whether you are seeing correlation or causation, and immediately brainstorm the confounding variables that could create the same data shape. Make it a habit—or even a team exercise—to generate as many plausible explanations as possible. An explanation that sounds unlikely at first can turn out to be the real one.

Question the data itself, too. Bad data is worse than no data because it creates unearned confidence. For example, an average-engagement metric can look absurdly low if it includes accidental launches and people who sampled an experience briefly and left. If that noise is not filtered, a team could draw the wrong conclusion about the product's value.

Cross-reference different sources, validate instrumentation, inspect filters and denominators, and make sure the measurement matches the product behavior you are trying to understand. With no data, people tend to be cautious. With bad data, they can make a seriously damaging product decision with complete confidence.

Editor's note: Why these lessons matter to Anamap

Several themes in this conversation reflect why we created Anamap. Product teams should not have to make important decisions without knowing whether their instrumentation is reliable, what a metric means, or where the underlying data came from. Greater access to data only helps when it is paired with shared definitions, context, and ways to validate what the numbers appear to show.

Anamap is our attempt to make that foundation more visible and easier to maintain. The goal is not to remove healthy skepticism from product decisions, but to give teams enough context to ask better questions, catch measurement problems earlier, and use data with more confidence.

This note reflects our perspective on the lessons from the conversation and should not be read as an endorsement of Anamap by the interviewee.