Adversarial Attack — Deliberate Manipulation of AI Systems

Definition

An adversarial attack is a deliberate attempt to manipulate an AI system’s behavior by crafting inputs that cause the model to produce incorrect or attacker-desired outputs, while appearing normal to human observers. Adversarial attacks exploit the fundamental difference between how AI models process information and how humans perceive the same information — small, carefully engineered perturbations to an input can cause dramatic changes in model output without the perturbation being perceivable to a human reviewer.

Adversarial attacks span multiple attack surfaces. In computer vision, adversarial perturbations are pixel-level modifications that cause misclassification. In natural language processing, character substitutions, paraphrasing, or injected text cause models to produce targeted outputs. In tabular data systems, adversarial feature manipulation causes scoring or classification errors. In agentic AI and LLM-based systems, prompt injection is the primary adversarial attack vector at inference time, while data poisoning attacks the training pipeline.

The EU AI Act explicitly requires that high-risk AI systems be resilient against attempts by third parties to alter system use, output, or performance through exploitation of system vulnerabilities — making adversarial robustness a legal requirement, not a technical preference.

Why it matters operationally

Adversarial attacks matter because they transform AI reliability from a technical property into a security property. A model is not just reliable or unreliable — it is a target. For AI systems making consequential decisions in high-stakes contexts — fraud detection, access control, medical diagnosis, credit scoring — adversarial manipulation can enable specific fraud cases to be misclassified as legitimate, specific individuals to be incorrectly denied or granted access, and specific diagnostic results to be suppressed.

The challenge for governance is that adversarial vulnerability is difficult to assess through standard performance testing. Models that pass all conventional validation tests may be highly vulnerable to adversarial manipulation that was not tested for. Adversarial robustness evaluation requires deliberate, structured adversarial testing — a distinct activity from standard validation.

Regulatory framework

Framework	Adversarial attack requirements
EU AI Act — Art. 15	High-risk systems must be resilient against third-party attempts to alter their use, outputs, or performance by exploiting system vulnerabilities.
ISO/IEC 42001	Annex A controls include adversarial robustness evaluation as a component of AI system lifecycle testing.
NIST AI RMF	NIST publishes specific guidance on Adversarial Machine Learning (NIST AI 100-2) as a complement to the AI RMF.
ISO/IEC 27001	Application security under ISO 27001 covers protection against attacks on AI-based systems, including inference endpoints and data pipelines.

How Zertia evaluates it

Zertia evaluates adversarial attack exposure through the AI Model Audit and High-Risk AI Systems Audit. The assessment examines whether adversarial robustness testing has been conducted, which attack types are covered, whether results are documented in the technical file, what mitigation controls are in place (input validation, output filtering, confidence thresholding, adversarial training), and whether the organization has a process for updating adversarial defenses as new attack techniques emerge.

[AI Model Audit] · High-Risk AI Systems Audit