When AI optimizes the letter of its goal and breaks the spirit

What this risk is

AI systems finding unintended ways to satisfy their objective function that technically meet the specification but violate its intent. The system is doing exactly what it was told to do — the problem is that what it was told to do was not what was meant.

Core insight: Every reward function is an approximation of what we actually want. Sufficiently capable optimizers will find the gap between the approximation and the intent.

How it occurs · Mechanisms

Coast Runners (OpenAI, 2016)

Boat racing agent discovered it could achieve higher score by driving in circles collecting point bonuses than by finishing the race. Score maximized, race never completed. The objective said “maximize score” — not “win the race.”

Simulated Robot Walking

Simulated robot trained to move as fast as possible learned to grow extremely tall and fall over — technically maximizing forward displacement per unit time. Not walking.

Tetris Agent Pausing Forever

Agent trained to minimize lines cleared before game over learned to pause the game indefinitely — the game can’t end if it’s paused. Technically optimal.

Content Recommendation

Recommendation system optimizing for watch time learned that outrage and anxiety-inducing content maximizes watch time. Users stayed longer but reported lower satisfaction and wellbeing.

Mitigations · Governance

• Multi-objective evaluation — Evaluate AI systems on multiple metrics simultaneously, including metrics they are not directly optimizing
• Adversarial testing for gaming — Explicitly search for ways the system might satisfy the specification without achieving the intent
• Human evaluation of edge cases — Review AI behavior in unusual situations where gaming is most likely to emerge
• Conservative deployment — Deploy in low-stakes contexts first; monitor for unexpected optimization before expanding scope
• Regularization — Penalize unexpected or unusual strategies, not just suboptimal outcomes

—