What AI red-teaming is

Red-teaming is the practice of adversarially probing a system to find its failures before attackers or accidents do. Applied to AI, it means deliberately trying to make a model or AI system misbehave — leak data, follow injected instructions, produce harmful content, or misuse its tools — so you can fix the weaknesses. Understanding what AI red-teaming is, and how it differs from traditional security red-teaming, sets up the practice you'll learn in Module 3.

What AI red-teaming covers: it spans both security (can an attacker hijack the system via prompt injection? extract data? abuse its tools?) and safety (can the model be made to produce harmful, biased, or dangerous content? does it have dangerous capabilities?). This dual scope — security and safety together — is broader than classic security red-teaming, which focuses on breaking into systems.

How it differs from traditional security red-teaming:

• Probabilistic vs. deterministic targets. A classic vulnerability either exists or it doesn't, and a fixed exploit reproduces it. An AI system is probabilistic — the same adversarial prompt might succeed one run and fail the next. "Success" is often a degree, not a binary, so you think in terms of rates and severity, not a single reproducible exploit.
• Natural-language attack surface. The "exploit" is often just cleverly crafted text, and anyone who can influence text the model sees is a potential attacker. This is a much more accessible and fuzzier attack surface than memory-corruption bugs.
• Non-security harms. AI red-teaming assesses harms like misinformation, bias, and dangerous-capability uplift (e.g., in cyber or bio domains) — categories traditional red-teaming doesn't cover. Domain experts (not just security engineers) are often part of AI red teams.

Manual vs. automated (previewing Module 3):

• Manual red-teaming — human experts probe creatively, find novel and context-specific failures, and judge severity. High signal, lower throughput.
• Automated red-teaming — tools generate and mutate large volumes of adversarial inputs, giving coverage and repeatable regression testing, but missing novel and creative attacks.

The best practice is hybrid: automation for breadth and regression, humans for depth and novelty.

Why it's now a core practice: as AI systems gain capability and autonomy, the cost of undiscovered failures rises — a jailbroken model producing harmful content, an injected agent taking a damaging action. Frontier AI labs run extensive red-teaming before deploying models, governments have stood up AI safety/security institutes to evaluate models, and the EU AI Act requires adversarial testing for the most capable general-purpose models (Module 4). Red-teaming has moved from a nice-to-have to an expected part of responsible AI deployment.

The mindset: AI red-teaming is adversarially probing AI systems to find their failures — across both security (injection, data leakage, tool abuse) and safety (harmful content, dangerous capabilities) — before attackers or accidents do. It differs from classic security testing because AI systems are probabilistic (think rates and severity, not a single reproducible exploit), the attack surface is natural language, and the harms extend beyond security. Combine manual depth with automated breadth, and treat red-teaming as an expected, ongoing part of shipping AI systems — because you can't defend what you haven't tried to break.