When clinical AI flips the diagnoses physicians had right

What this risk is

Clinicians over-trusting AI diagnostic or clinical decision support recommendations — failing to apply independent clinical judgment or override incorrect AI recommendations, even when their own assessment or objective clinical indicators contradict the AI output.

Automation bias is not a software bug; it is a human cognitive pattern. It is most pronounced when: the AI expresses high confidence, the clinician is fatigued or time-pressured, or the AI’s recommendation aligns with the path of least resistance.

How it occurs · Mechanisms

Goddard et al. (2012) — Automation Bias in Medicine

Systematic review establishing that clinicians across specialties exhibit automation bias when using decision support systems — accepting incorrect computer recommendations at significantly higher rates than would be expected from independent decision-making.

AI Radiology and Over-acceptance (Multiple studies, 2020–2024)

Multiple controlled studies showed radiologists who were shown AI-generated annotations accepted false positive findings at higher rates than radiologists who reviewed images independently. The AI’s visible marking of a finding increased the probability it would be reported.

Sepsis AI Alert Fatigue and Override Errors

Hospitals deploying sepsis prediction AI reported both directions of automation bias: alert fatigue causing under-reliance on valid alerts, and over-reliance on alerts leading to inappropriate treatment in borderline cases.

Mitigations · Governance

Design Controls

• AI-last, not AI-first — Have clinicians form independent assessments before seeing AI recommendations where feasible; reduces anchoring
• Uncertainty communication — AI systems display confidence intervals and known failure modes, not just point recommendations
• Challenge prompts — System asks clinician to articulate why they agree with AI recommendation before accepting it for high-stakes decisions
• Disagreement visibility — When AI and clinician assessments diverge, make divergence visible and require acknowledgment

Operational Controls

• Human oversight requirements — Define which decisions require independent clinical assessment regardless of AI recommendation
• Regular calibration training — Train clinicians on when AI systems fail; maintain clinical skill that AI is meant to support
• Monitoring for over-acceptance — Track rates at which clinicians override AI vs. accept; high acceptance rates warrant investigation

—