When AI ships with capabilities it cannot actually deliver

What this risk is

The systematic gap between AI performance measured in controlled evaluation conditions and performance in real-world deployment. AI systems that perform at or above human level on benchmarks may perform significantly worse in production, often without this gap being visible until after deployment consequences have materialized.

How it occurs · Mechanisms

Benchmark contamination: Training data may contain benchmark test sets, inflating measured performance.

Distribution shift: Real-world inputs differ from benchmark inputs in ways that are hard to anticipate — different phrasing, different context, different populations, different languages or dialects.

Benchmark gaming: Optimization pressure to perform well on benchmarks leads to models that are overfit to benchmark-specific patterns.

Evaluation conditions vs. deployment conditions: Benchmarks evaluate models in isolation; production involves integration with other systems, real users, and variable input quality.

Selective reporting: Published benchmark results favor models and conditions that show good performance.

Real-world incidents

IBM Watson for Oncology (2017–2019)

Marketed as achieving expert oncologist performance. Internal MSKCC documents leaked to Stat News showed it provided “unsafe and incorrect” treatment recommendations in many cases. Multiple hospitals discontinued use after paying millions in licensing fees.

Gap source: Benchmark performance on structured cases with clean data vs. real clinical cases with incomplete records, unusual presentations, and treatment constraints.

COVID-19 AI Diagnostic Tools (Wynants et al., 2020 — BMJ)

Systematic review of 232 AI tools developed for COVID-19 diagnosis. Conclusion: nearly all had methodological issues that inflated reported performance. None were recommended for clinical use based on available evidence.

Autonomous Vehicle Disengagement Rates

CAV companies report dramatically different performance in PR materials vs. California DMV disengagement data. Real-world performance in varied weather, road conditions, and edge cases consistently underperforms controlled demonstrations.

Mitigations · Governance

• Prospective validation — Evaluate on data collected after model development, not data from the same period
• Real-world pilot before full deployment — Deploy at limited scale with intensive monitoring before full rollout
• Independent evaluation — Third-party evaluation on held-out data not shared with the model developer
• Production monitoring — Track actual performance metrics post-deployment, not just at launch
• Honest capability communication — Require disclosure of evaluation methodology alongside performance claims
• Sunset clauses — Require re-evaluation at regular intervals; performance that was valid at deployment may degrade

—