Where AI capability crosses into mass-harm territory

What this risk is

Using AI systems to develop cyber weapons (e.g., coding cheaper and more effective malware), develop new or enhance existing weapons (e.g., lethal autonomous weapons or CBRNE — chemical, biological, radiological, nuclear, explosive), or use weapons to cause mass harm to individuals, infrastructure, or ecosystems.

How it occurs · Mechanisms

Causal profile: Human-caused · Intentional · Post-deployment

• AI-assisted vulnerability discovery — AI can scan codebases and identify exploitable vulnerabilities faster than human researchers
• Automated malware generation — LLMs can generate and customize malware code, lowering the technical barrier to sophisticated attacks
• CBRNE uplift — AI can potentially accelerate the development of biological, chemical, or nuclear weapons by providing synthesis routes or optimization
• Lethal autonomous weapons (LAWs) — AI-powered weapons that can identify and engage targets without human authorization
• Critical infrastructure attacks — AI optimizes attack strategies against power grids, water systems, financial systems

—

Real-world incidents

AI-Assisted Cyberattacks (Multiple, 2023–2025)

Microsoft, Google, and OpenAI all documented nation-state actors (Russia, China, North Korea, Iran) using LLMs to support cyberattack operations — including researching targets, writing phishing content, and developing attack tools.

BioSecurity Concerns (RAND, 2024)

RAND Corporation and Johns Hopkins researchers demonstrated that frontier AI models could provide meaningful uplift to non-experts attempting to develop biological weapons, leading to emergency discussions at the US AI Safety Summit.

Gaza: AI in Military Target Selection (2024)

Reports from Israel’s use of AI targeting systems (Lavender, Gospel) in Gaza raised urgent questions about AI in lethal decision-making, the adequacy of human oversight, and accountability for AI-assisted military decisions.

Mitigations · Governance

• Dual-use research restrictions — Implement biosecurity guardrails in AI systems to prevent CBRNE uplift
• International agreements — Support multilateral treaties on lethal autonomous weapons
• Export controls — Restrict export of AI systems that could be weaponized
• Red lines in model training — Define categories of capabilities that AI systems should never provide
• Usage monitoring — Monitor API usage for patterns consistent with weapons research

—