When the same AI gives different answers to the same question

What this risk is

AI systems producing different outputs for semantically identical or near-identical inputs — including contradicting themselves within a single session, giving different answers depending on question phrasing, or behaving differently in edge cases not covered by testing. In high-stakes applications, behavioral inconsistency is itself a safety risk, independent of whether any individual response is correct.

How it occurs · Mechanisms

Same Question, Different Answers

LLMs are stochastic systems. The same question asked at different times, with different conversation history, or with minor phrasing variations may produce substantively different answers. This is acceptable for creative tasks; it is problematic for clinical, legal, or financial applications.

Self-Contradiction Within Sessions

Models can make statements early in a conversation that they contradict later — particularly in long conversations where early context is effectively forgotten or in complex multi-step reasoning.

Language and Format Sensitivity

Studies have documented that LLMs give different answers to the same question asked in different languages, different dialects, or different formats (bullet points vs. prose). This suggests the model is responding to surface features rather than semantic content.

Prompt Sensitivity

Small changes in how a question is phrased can produce large changes in the answer: adding “as an expert” to a prompt, changing “can” to “should”, or altering question order all affect outputs significantly.

Mitigations · Governance

• Deterministic inference — Set temperature to 0 for applications requiring consistency; sacrifice diversity for reliability
• Consistency testing — Systematically test outputs for consistency across prompt variations, languages, and phrasings
• Output validation — Secondary validation layer that checks outputs for internal consistency
• Structured prompting — Use structured prompt templates that reduce surface variation
• Version locking — Lock to specific model versions in production; avoid automatic updates
• Human review for high-stakes inconsistencies — When AI contradicts a previous answer or expresses uncertainty, escalate to human review

—