Experimentation Practice

TD · Product Owner

Experimentation Practice

1M+

Users Served

+18%

Engagement

-12%

Churn

ExperimentationA/B TestingProduct AnalyticsDecisioningStakeholder Alignment

Overview

Created a rigorous experimentation practice on a real-time decisioning platform, replacing unverified “wins” with pre-registered tests, holdouts and measurable outcomes.

The Challenge

The platform served more than one million users but had no structured experimentation process or defined success metrics. Earlier “wins” lacked holdout groups, and an expensive data-enrichment initiative was not addressing why analysts overrode recommendations.

The Approach

Interviewed analysts, identified interpretability as the issue, and redirected engineering toward an explainability layer while protecting the sub-500ms latency requirement. Co-authored experiment standards with data science and engineering, then ran the first test as a public proof of concept.

Key Outcomes

  • Built pre-registration, holdout and significance standards for experiments.
  • Ran 15+ experiments; engagement improved 18%, conversion 18% and churn fell 12%.
  • The experimentation framework was adopted by three other product teams.
  • Redirected engineering from a low-value data-enrichment initiative, avoiding about $2M in annual spend.

The Result

Fifteen-plus experiments produced attributable engagement, conversion and churn improvements. Analyst overrides fell from 40% to under 20% after the explainability work, and the practice was adopted by three other teams.