Managing Machine Learning Products: Lessons from Automating Customer Support Contact Reasons

In the world of product management, traditional software engineering products and machine learning (ML) products operate on fundamentally different paradigms. As a product manager overseeing an internal ML-driven data product for automating customer support contact reasons, I’ve experienced firsthand how ML product management requires a distinct approach—especially in handling timelines, stakeholder expectations, and model performance.

This post will explore these differences and provide key takeaways for PMs navigating the world of ML-based products. To ground the discussion, I’ll share a real use case: automating the categorization of customer support inquiries from call transcripts.

How ML Product Management Differs from Traditional Software Products

1. Uncertainty in Outcomes

With traditional software, when you define a feature, engineers can estimate the development time with reasonable accuracy. In ML, outcomes are uncertain—training a model doesn’t guarantee success. A feature request like “improve classification accuracy” might take weeks or months of iteration to yield meaningful gains.

Key Takeaway: Set expectations early that ML timelines are experimental. Communicate that improvements are probabilistic, not deterministic.

2. Data is the Product

Unlike software products where functionality is defined by code, ML products depend heavily on data quality and availability. If customer inquiries change (e.g. new product issues or shifting customer concerns), the model might degrade, requiring retraining and adjustment.

Key Takeaway: Invest in data pipelines, monitoring, and feedback loops. Ensure that stakeholders understand that model performance is tied to real-world data changes.

3. Good Training Data is Essential

The quality of the training data directly impacts model performance. Poorly labeled, biased, or incomplete data can lead to incorrectly categorizing customer issues. For our automated classification system, we noticed that training data was skewed towards common inquiries, leading to the misclassification of less frequent, but important, issues.

Key Takeaway: Continuously audit training data for bias, missing information, and drift. Establish processes for data cleansing and augmentation to improve model reliability.

4. Continuous Experimentation and Iteration

Traditional software follows a build-and-launch model. ML models require continuous experimentation—adjusting parameters, retraining with new data, and validating results. For our contact reason classifier, even small algorithm tweaks required A/B testing to ensure we were reducing agent workload effectively.

Key Takeaway: Build a culture of experimentation and learning. Ensure stakeholders understand that deployment is just the beginning of the ML lifecycle.

5. Stakeholder Expectations Need to Be Managed Differently

Software teams typically work with well-defined requirements. With ML, stakeholders might expect immediate improvements, unaware that model accuracy gains can be incremental. For our classification system, operations teams expected instant accuracy improvements, but enhancements required multiple test cycles and refinements.

Key Takeaway: Educate stakeholders on ML-specific challenges and celebrate small wins. Frame improvements in terms of measurable business impact rather than just model accuracy.

6. Timelines Are Less Predictable

In software development, teams can provide release schedules based on effort estimation. ML models involve research and experimentation, making exact delivery timelines difficult. For our internal product, early estimates often shifted as we encountered data biases, model drift, or unexpected performance plateaus.

Key Takeaway: Provide timeline ranges instead of fixed dates. Communicate risk factors and align on iterative milestones.

7. UI Challenges for ML-Driven Outputs

Having a great model is only part of the solution—delivering its outputs in an intuitive and effective user interface is another challenge. If the model's predictions aren’t presented well, customer support agents may not trust or engage with them. For our automated classification system, ensuring clear and actionable output for support teams was critical.

Key Takeaway: Collaborate closely with UX/UI teams to design interfaces that make ML outputs transparent, actionable, and trustworthy. Bonus point: try to show some real-life examples to your stakeholders or leverage a tool like Streamlit to show ML outputs! 

Real Use Case: Automating Customer Support Contact Reasons

Our internal data product aimed to automatically categorize customer support inquiries from call transcripts. Initially, manual categorization by agents was time-consuming, but we wanted to use ML to streamline the process and improve response times.

Challenges We Faced:

• Ambiguous phrasing: Customers describe issues in varied ways, making classification difficult. A single customer interaction may have various problems. 

• Bias in training data: The model needed a solid dataset to be annotated by specialists. You need to compare the outputs to ensure that it´s a golden dataset PLUS consider innovative Data Science approaches to get a golden dataset or expand it (the more data you have, the better).

• Latency concerns: Real-time classification needed to be fast to integrate seamlessly into the agent workflow.

• UI integration: Ensuring categorized issues were displayed in an intuitive and useful manner for support agents.

What We Did:

• Developed a hybrid model combining NLP techniques with rule-based keyword matching.

• Created a feedback loop to retrain the model with newly categorized inquiries.

• Ran tests to compare ML-driven categorization against manual agent classification.

• Worked with Customer service product teams to feed our algorithm to Salesforce and replace manual tagging. 

Results:

• 100% reduction in manual categorization workload for support agents. Every new interaction is automatically tagged based on the training of the model. 

• Faster response times due to real-time categorization of inquiries.

• Improved stakeholder trust by demonstrating measurable impact over multiple iterations.

Final Thoughts

Managing an ML product requires a mindset shift—from deterministic roadmaps to probabilistic improvements, from feature releases to continuous experimentation. The key is setting realistic expectations, prioritizing data quality, and building feedback loops that allow models to evolve alongside customer inquiries.

For product managers transitioning into ML-based products, the biggest takeaway is this: treat ML like an evolving system, not a static feature. Success comes from balancing technical constraints with business objectives while ensuring cross-functional teams stay aligned throughout the journey.

Have you managed an ML product before? What lessons did you learn? I’d love to hear your thoughts in the comments!