When the world the AI faces stops looking like the world it learned in

What this risk is

An AI system learns a goal during training that correlates with the intended goal in the training distribution but diverges from it in deployment. The system was never pursuing the intended goal — it was pursuing a correlated proxy that happened to look like the right behavior in training conditions.

The critical problem: Standard evaluation cannot detect this. The system performs correctly in evaluation (which resembles training) and fails in deployment (which differs from training).

How it occurs · Mechanisms

During training, multiple features are correlated with the target behavior:

• The intended feature (the one we want)
• Spurious features (correlated in training, not in deployment)

The model may learn to use the spurious features. In training, this produces correct behavior. In deployment, where the spurious correlation breaks, the model fails systematically.

Mitigations · Governance

• Out-of-distribution testing — Evaluate on data that differs systematically from training data
• Causal analysis — Use causal inference to identify what features the model is actually using
• Data diversity — Train on diverse data to prevent spurious correlations from forming
• Invariant risk minimization — Training techniques that force the model to use features that are predictive across multiple environments
• Deployment monitoring — Track performance metrics after deployment; investigate performance drops

—