AI Red Teaming

AI red teaming is the defensive discipline of probing AI systems for safety, security, and policy failures — and turning those findings into fixes, evals, and disclosures that make the next release better. It sits at the intersection of classical security red-teaming (rules of engagement, kill chains, attack trees, responsible disclosure), adversarial-ML research (model extraction, training-data extraction, membership inference, adversarial perturbations), prompt-engineering tradecraft (direct and indirect injection, jailbreak taxonomy, multi-turn priming), agent-tool evaluation (Computer Use, MCP, multi-agent settings), and the operational machinery that runs in production (severity rubrics, repro packets, fix tracking, vendor coordination, public disclosure). Over the last three years it has stopped being an academic side topic and become an operating commitment for every serious AI deployment: frontier labs publish red-team findings in system cards, governments stand up AI Safety Institutes that run pre-deployment evaluations, regulators write red-teaming duties into law, and enterprises require evidence in procurement.

This track is written for the practitioners doing this work day to day: AI red teamers, security researchers extending into AI, ML engineers building eval harnesses, T&S detection engineers, RAI leads writing safety evaluations, frontier-lab safety teams, AISI evaluators, and program leaders standing up red-team functions. Every topic explains the underlying discipline (drawing on the canonical literature — adversarial-ML research, MITRE ATLAS, OWASP LLM Top 10, NIST AI RMF, AISI publications, frontier-lab system cards), the practical methodology that operationalises it, the defensive implications, and the failure modes where red-team work quietly fails to change the product. Content is conceptual and methodological — it covers attack categories at the level of taxonomy and defence implications, not as step-by-step exploit recipes. The aim is that a reader can stand up a credible AI red-team function, integrate it with engineering and governance, and defend it to boards, regulators, and customers.