When “Better AI” Is the Wrong Product Goal
When a model’s recommendations are ignored, it is tempting to improve the model.
Add more signals. Tune the threshold. Improve accuracy. Those can be valid investments—but only if accuracy is what prevents people from using the output.
On a real-time decisioning platform, analysts were overriding recommendations at a high rate. A data-enrichment initiative was already underway to improve the model. Before committing more engineering time, I wanted to understand what analysts did after receiving a recommendation.
Follow the output into the user’s workflow
I spoke with three analysts about the decisions they made, the cases they handled, and what happened when a recommendation did not fit. The pattern was consistent: they were not simply rejecting inaccurate output. They were overriding recommendations because the output did not include context they were required to consider.
That distinction changed the problem statement. The issue was not just whether the model could predict well. It was whether an analyst could understand and responsibly use the recommendation.
Adding more behavioral signals might make the model more complex without resolving that gap. Optimising the input would not necessarily improve trust in the output.
Make the trade-off explicit
I documented the finding and recommended redirecting engineering effort toward an explainability layer. The proposal was not “stop improving the model.” It was “address the observed barrier to adoption before investing in more signals.”
The team also had a performance constraint: the platform needed to stay under 500 milliseconds at the 99th percentile. A proposed real-time enrichment path risked pushing response times beyond that limit under peak load. We considered a staged alternative: batch enrichment first, with real-time enrichment later if the infrastructure could support it.
This made the decision less abstract. We were balancing interpretability, latency, adoption and cost—not choosing between “AI quality” and “no AI.”
Build evidence into the operating model
A product decision about AI should connect the model’s performance to how people use its output. Useful questions include:
- What does the user do with the recommendation?
- When do they accept, override, or escalate it?
- What information do they need to understand the output?
- Which constraints—latency, auditability, policy, or cost—are non-negotiable?
- What would change our mind about the next investment?
For this platform, experimentation standards mattered too. Earlier tests lacked holdouts, so the team could not reliably separate real impact from novelty. We established pre-registration, primary and guardrail metrics, holdout expectations, and a consistent readout. The framework supported more credible decisions about engagement and platform changes.
Measure the system, not just the model
After redirecting work toward explainability, the analyst override rate fell from 40% to under 20%. The platform maintained its sub-500ms latency target. The experiment practice supported measured improvements in engagement and churn, and three other teams adopted the standards.
Those outcomes do not mean explainability is always the right answer. They show why observing the user’s job can be more valuable than assuming a model metric explains product adoption.
Before asking how to make an AI system more accurate, ask what people do with its output—and what makes them trust, question, or override it. The answer may point to a different product problem entirely.