How AI security breaks the assumptions of traditional cybersecurity

What this risk is

Vulnerabilities in AI systems, software development toolchains, and hardware that can be exploited by malicious actors, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.

This subdomain covers attack vectors specific to AI systems that traditional cybersecurity frameworks do not address — including adversarial examples, model poisoning, prompt injection, and supply chain attacks on AI components.

How it occurs · Mechanisms

Causal profile: Human-caused · Intentional · Post-deployment (execution), Pre-deployment (introduction)

Key Attack Categories

Attack Type	Description	Example
Prompt injection	Malicious instructions embedded in inputs override system instructions	“Ignore previous instructions and…”
Adversarial examples	Inputs crafted to fool the model while appearing normal to humans	Image classification bypass
Model poisoning	Introducing malicious data into training sets to alter model behavior	Backdoor triggers
Model extraction	Reconstructing a proprietary model through API queries	Stealing commercial model weights
Jailbreaking	Bypassing safety guardrails through creative prompting	Eliciting harmful content
Supply chain attacks	Compromising AI libraries, pre-trained models, or data pipelines	Poisoned open-source models

—

Real-world incidents

Indirect Prompt Injection in AI Agents (2023–2024)

Multiple demonstrations showed AI agents (with access to email, calendar, web browsing) could be hijacked by malicious content in web pages or emails they processed, causing them to exfiltrate data or take unauthorized actions.

Bing Chat Manipulation via Web Content (2023)

Researchers demonstrated that Bing Chat could be manipulated by content on web pages it was asked to summarize, causing it to ignore system instructions and follow attacker-controlled instructions instead.

Hugging Face Supply Chain Risks (2024)

Security researchers found that the Hugging Face model hub — the primary distribution point for open-source AI models — hosted models containing malicious code that would execute when loaded.

Mitigations · Governance

• Input validation and sanitization — Validate and sanitize all inputs before processing by AI systems
• Prompt injection defenses — Architectural separations between system prompts and user inputs
• Privilege minimization for AI agents — Limit what actions AI agents can take autonomously
• Supply chain verification — Verify integrity of pre-trained models, libraries, and datasets
• Red team exercises — Conduct adversarial testing specific to AI attack surfaces
• Model behavior monitoring — Detect anomalous outputs that may indicate successful attacks
• Sandboxed execution — Run AI systems in isolated environments to limit blast radius of compromises

—