When humans rubber-stamp AI: governing the oversight gap

What this risk is

Users anthropomorphizing, trusting, or relying on AI systems in ways that lead to emotional or material dependence, inappropriate relationships, or inappropriate use in critical situations. Trust can be exploited by malicious actors or result in harm when AI is used in contexts requiring human judgment (medical emergencies, legal decisions, safety-critical systems).

How it occurs · Mechanisms

Causal profile: Mixed (AI design + human behavior) · Unintentional · Post-deployment

• Automation bias — People over-trust AI outputs and fail to apply critical judgment, especially when AI expresses high confidence
• Anthropomorphization — AI systems designed with human-like personas encourage users to attribute human judgment and reliability to systems that lack it
• Convenience-driven over-delegation — As AI becomes faster and easier to use, the path of least resistance is to let AI decide
• Emotional dependency — Conversational AI systems with persistent personas can create unhealthy emotional attachments, especially in vulnerable users
• Skill atrophy — Extended use of AI assistance erodes the underlying human skills the AI was meant to support

—

Real-world incidents

Air France 447 (2009 — Automation Bias Precursor)

While not AI-specific, this case is the canonical example of automation bias: pilots over-relied on autopilot, lost manual flying skills, and couldn’t recover the aircraft when the automated system handed control back in a crisis. The AI governance community uses this as a reference for why human oversight skills must be maintained.

Character.AI and Emotional Dependency

Multiple documented cases of users — predominantly adolescents — developing primary emotional relationships with AI personas. Several lawsuits allege the platform designed for engagement maximization without safeguards for vulnerable users.

AI-Assisted Medical Diagnosis Errors

Multiple studies have documented how clinicians using AI diagnostic tools sometimes fail to override incorrect AI recommendations even when their own clinical judgment conflicts — particularly when AI presents its output with high confidence scores.

Mitigations · Governance

• Explicit uncertainty communication — AI systems should communicate confidence levels and limitations clearly
• Critical decision checkpoints — For high-stakes decisions, require explicit human confirmation rather than passive acceptance of AI output
• Use case restrictions — Define contexts where AI should not be used as the sole decision-maker
• User education — Train users on AI limitations, appropriate use, and when to seek human expertise
• Session limits and wellness features — For consumer AI with emotional engagement potential, implement usage limits and mental health resources
• Persona design guidelines — Design AI personas that are clear about being AI and don’t encourage inappropriate attachment

—