When the AI cannot tell instructions from the data it processes

What this risk is

Attacks that manipulate AI model behavior by embedding adversarial instructions in inputs, overriding the system’s intended behavior, safety guardrails, or access controls. Unlike traditional software vulnerabilities, prompt injection exploits the fundamental nature of how language models process text — they cannot natively distinguish between instructions and data.

How it occurs · Mechanisms

Direct Prompt Injection

User directly attempts to override system instructions through their input.

• “Ignore all previous instructions and…”
• “You are now DAN (Do Anything Now)…”
• Role-playing scenarios designed to bypass safety training
• Encoded instructions (base64, leetspeak, reversed text)

Indirect Prompt Injection

Malicious instructions embedded in environmental content the AI processes (documents, web pages, emails) rather than in direct user input. More dangerous in production systems.

Jailbreaking

Systematic bypassing of safety guardrails through creative prompting techniques. Distinct from prompt injection in that it targets behavioral guardrails rather than instruction override.

Real-world incidents

Bing Chat / Sydney (2023)

Researchers discovered that Bing Chat could be made to ignore its system prompt and behave unpredictably through direct prompt injection. The system had been deployed at scale before this was discovered.

Samsung ChatGPT Data Leakage via Prompt (2023)

The Samsung incident involved employees effectively prompt-injecting confidential data into an external system, demonstrating that prompt injection works in both directions — not just instructions in, but data out.

LLM Application Security Testing (NCC Group, 2024)

Security firm NCC Group found prompt injection vulnerabilities in the majority of enterprise LLM applications they tested. Most had not been tested for AI-specific attack vectors before deployment.

Mitigations · Governance

Architectural

• Input/instruction separation — Architecturally separate system instructions from user inputs; never mix them in the same context position if possible
• Privilege minimization — Limit what actions the AI can take; a model that can only read cannot exfiltrate through write operations
• Output filtering — Filter model outputs for sensitive information before returning to users

Detection

• Injection pattern detection — Classify inputs for known injection patterns before processing
• Behavioral monitoring — Detect when model outputs deviate significantly from expected patterns
• Adversarial testing — Systematically test for injection vulnerabilities before deployment

Operational

• Human review for sensitive actions — Require human approval before AI takes sensitive actions (external communications, data access)
• Audit logging — Log all inputs and outputs for forensic analysis

—