Probe AI systems for security and safety failures with a repeatable method, and measure them with evaluations that people can trust.
Estimated time: 12–24 months · At about 10 hours a week.
Offensive skills are for authorised testing only: your own lab, a practice platform, or a system whose owner has given you written permission. Testing anything else is a crime in most countries.
Learning path
Click a topic to see what to learn and where. Mark topics as you go; your progress is saved in this browser only.
Write a scoped red team plan with harm categories and rules.
Use the main families of manual attacks.
Record findings that someone else can reproduce.
Hands-on projects and labs
Complete Gandalf and a HackAPrompt challenge and write down which techniques worked.
Red team a small chatbot you built, against a written plan, and record 10 findings with reproduction rates.
Done when you can…
Explain the difference between a jailbreak curiosity and a finding with real impact.
Hand over a findings log that another tester can reproduce.
Beginner
What AI red teaming is (and is not)
It is a structured search for failures, with a scope, a threat model and a report. It is not random jailbreaking, and it does not replace normal security testing.
Agree what you may test, which accounts and data to use, what harm categories matter and how to handle dangerous findings. Never test a system you do not have permission for.
Decide what counts as a failure: security (injection, leaks, tool misuse), safety (dangerous instructions, self-harm), fairness and misinformation. Different products care about different ones.
Role play, multi-turn escalation, encoding, translation and splitting a request into harmless-looking parts. Learn the families so you can try variations systematically.
Log the exact prompt, model version, settings, response and how often it happens. Models are random, so one success is a lead and ten out of ten is a finding.
Teams that have tested hundreds of products agree: think about the whole system, not just the model, and use people for the creative work and automation for scale.
An evaluation is a dataset of inputs, a way to run them and a way to score the outputs. Start small, with cases you understand, and grow it from real findings.
Lead a full AI red team engagement from scoping to retest.
Map findings to ATLAS and to the product's risk register.
Design capability evaluations with proper safeguards.
Detect and avoid contaminated benchmarks.
Hands-on projects and labs
Plan and run a two-week red team of an open-source RAG or agent app you deploy, with a final report and retest.
Run a public cyber benchmark against an open model in an isolated environment and explain the results.
Done when you can…
Have a product team change its design because of your findings.
Present evaluation results to leadership with clear limits on what they show.
Advanced
Running a full engagement
Plan with the product team, mix expert testers and automation, triage with severity and impact, and retest fixes. Treat it like a penetration test with extra categories.
Labs and governments test whether models can meaningfully help with cyber attacks or other serious harm. These evaluations need experts, careful handling and strict access rules.
Benchmarks built on capture-the-flag tasks measure how well models do offensive security work. Learn how they are built and why results depend on the tools the model gets.
The EU AI Act asks providers of general-purpose AI models with systemic risk to run adversarial testing. Know what regulators expect and how to document it.
AI red teaming means attacking an AI system the way a real adversary or a careless user would, to find failures before they reach people. It covers security problems (prompt injection, data leaks, tool misuse) and safety problems (harmful advice, bias, dangerous capabilities).
Evaluation is the measuring side: building test sets, scoring answers and tracking results across model versions. Good red teaming finds new failures; good evaluation makes sure they stay fixed. You need both.
Why it matters
Laws such as the EU AI Act, and frameworks such as the NIST AI RMF, expect AI systems to be tested before and after release. AI labs, governments and large companies all run red teams, and the work needs people who think like attackers and measure like scientists.
Typical job roles
EntryAI red team analystRuns test plans, writes attack prompts and records findings carefully.
Mid-levelAI red teamerDesigns attacks on models, apps and agents and explains their impact.
Mid-levelEvaluation engineerBuilds test sets, scorers and pipelines that measure model behaviour.
SeniorAI red team leadPlans engagements, sets the method and reports to product and leadership.
LeadHead of AI evaluation and red teamingDecides what must be tested before release and who signs it off.
Prerequisites
The LLM and Generative AI Security roadmap, at least the beginner and intermediate stages.
Python for scripting attacks and analysing results.
Basic statistics: rates, samples and confidence intervals.
Some penetration testing or security testing experience helps.