Product Data Interview: Randy Young

This is part of our Product Data Interviews series, where we ask product managers across the industry the same set of questions about how they use data, what slows them down, and what helps them make better product decisions.

Randy Young is a forward-deployed product manager at Cresta, where he works on voice AI agents that replace interactive voice response systems. He has previously worked in product roles at Autodesk, Splunk, Amazon, and Bugcrowd, with experience spanning customer-facing product management, product-led growth, experimentation, and enterprise customer workflows.

Connect with Randy on LinkedIn.

1. What kind of product decisions are you personally responsible for in your day to day work?

My role at Cresta is very different from anything I have done before. Previously, I worked as a core, customer-facing product manager with a focus on product-led growth. That work was very much about experimentation: building features, testing outcomes, and using the results to guide the roadmap.

Today I am a forward-deployed product manager building voice AI agents that replace IVRs. It is a little more like being a consultant product manager for customers. In many cases, you are working with call center operations teams that do not have a product manager, so you are helping them refine requirements, manage the project, and run user acceptance testing.

Most of the decisions are around setting up a voice agent so it can meet the business goals the customer has. That could mean deflecting minutes from human agents through authentication, intent routing, or answering simple questions with an AI agent, such as account balances. It could also mean warming leads for sales or containing users completely when a human decision or human help is not necessary.

The interesting thing about this position is that the product changes based on each customer's needs, which means the decisions are changing constantly as well.

2. When you're evaluating whether something is working (or worth building), what signals matter most to you?

You can measure a lot of things inside a call, especially the successful outcomes of the use cases you are trying to solve.

There are also downstream effects you can measure, like a reduction in call abandonment, containment of the user inside the AI agent after the call is successfully handled, and a reduction in mean time to handle for the human agent after the call is handed off from the AI agent.

3. Walk me through a recent product decision you made that involved data. What did the process actually look like?

I had a customer that did not want to contain calls inside the voice AI agent because they thought their human agents would handle things better. At the same time, they had a problem with long hold times because their human agents were handling too many calls, and they had a high abandonment rate on calls that were put on hold.

We loaded a few thousand of their human-agent calls into our system. Using AI, I evaluated those calls to see which questions could be answered easily and which issues could be solved inside a voice agent without ever needing to go to a human. I also evaluated the mean time to handle for each call.

With that analysis, we could recommend a set of FAQ questions and knowledge-based articles that would solve some of their calls without requiring a human agent, while freeing up their human agents to solve the more complicated customer issues.

4. How confident do you generally feel in the data available to you when making product decisions? What tends to increase or reduce that confidence?

Sometimes you have a challenge getting to historical calls, and you have to make a judgment call based on other projects you have worked on.

In those cases, you really cannot measure it until you launch, experiment with changes, and then measure the outcome.

5. What's the most frustrating or time-consuming part of getting the insights you need to make a decision?

In many cases, it can be challenging to get customers on board with making changes to their existing processes. They have a lot of experience running their businesses, but not a whole lot of experience with voice AI agents.

In some cases, customers are not comfortable making decisions in ambiguity or with a limited amount of data.

6. How self-serve is data access for product managers at your company today?

Out of all the places I have worked, we probably have the most access to information about the performance of our product.

I think it is important to have a combination of aggregated data across multiple customers, the ability to filter by industry or specific customer type or size, and a process for measuring specific customers and outcomes based on their AI agent's performance.

It is also nice to have AI overlays that can give you additional insights.

7. What's the hardest thing about turning data into action rather than just more dashboards or reports?

I think this comes with experience and domain knowledge. You get more comfortable working in some ambiguity because you have some idea about how different decisions will affect things downstream.

8. Are there product metrics or definitions that people at the company regularly interpret differently?

Yes. There are some challenges around what is considered containment inside a voice agent.

For example, if someone does not complete a task but gets an FAQ question answered and then hangs up, that customer did not get to a human agent, so they were contained. You have to make a judgment call on whether that is acceptable containment. Some customers only consider it containment when a happy path was completed, like making a payment or getting account information.

There are also judgment calls around cases where someone gets a balance, their next payment due date, and the amount of that payment, but then hangs up before saying "thank you." In many of these cases, you need to decide whether that counts as containment.

Every product manager in our organization handles these differently, and it really depends on what the customer wants or thinks are acceptable outcomes.

Recently we have been looking into first call resolution, or seeing if a customer calls back in three to five days for the same or a similar issue. Measuring this is more about true containment and a completely solved inquiry.

9. What's a surprising or overlooked source of product insight that you think more teams should pay attention to?

In every company I have worked at, it has always been a challenge to aggregate all of the different venues where a customer might touch your brand.

It is about the single view of your customer: understanding all the different avenues where they have touched your company and the success of those different avenues. That could include your support organization, social media, a conference, sales outreach, a chat or phone system, talking to a human support agent, or even their satisfaction with your end product in general.

10. What advice would you give another PM at a startup trying to make better product decisions with data?

Initially, especially when you are building a zero-to-one product, you need to listen to your customer. You need to do lots of interviews, listen for signals, and then build your product based on those signals.

As you are building, you need lighthouse customers to bounce ideas off and measure the success of what you are building and prototyping. It is really important to define success before you even start building and to have a way to measure against that.

Success can be revenue. Success can be customer adoption. Success can even be customer satisfaction based on use of the product. It really depends on the product you are building and the intended outcome.

For example, enterprise software is usually about revenue or the number of Fortune 500 companies that are referenceable, which I like to call your "NASCAR slide." Consumer products are usually about active users, subscribers, or ad revenue.

It is all about setting realistic goals and then measuring, refining, and remeasuring your progress.

A final irony about data companies

One irony Randy called out is that companies considered "data" companies are sometimes the places where internal data is hardest to get to. Or, as he put it: "The cobbler's kids have the worst shoes."