Understand how machine learning models are attacked, through their data, their inputs and their outputs, and how to make them robust and private.
Estimated time: 15–26 months · At about 10 hours a week.
Offensive skills are for authorised testing only: your own lab, a practice platform, or a system whose owner has given you written permission. Testing anything else is a crime in most countries.
Learning path
Click a topic to see what to learn and where. Mark topics as you go; your progress is saved in this browser only.
Draw an ML pipeline and mark where data, code and models can be tampered with.
Describe attacks with the NIST and ATLAS vocabulary.
Load and share models safely, avoiding untrusted pickle files.
Hands-on projects and labs
Train a small image or text classifier in PyTorch or scikit-learn and write a one-page threat model for it.
Map two ATLAS case studies to the tactics and techniques they used.
Scan a model file with ModelScan or picklescan and explain what it found.
Done when you can…
Explain the difference between evasion, poisoning, extraction and inference attacks.
Write a threat model for an ML system that names the attacker, their access and their goal.
Beginner
The ML lifecycle and its attack surface
Data collection, labelling, training, evaluation, deployment and monitoring are each a place an attacker can step in. Draw the pipeline first, then ask who can touch each step.
Ask what the attacker knows (white box or black box), what they can change (data, inputs, the model file) and what they want (wrong answers, stolen model, private data).
NIST's taxonomy groups attacks by when they happen (training or deployment), the attacker's goal and their knowledge. Use it as a shared vocabulary for findings.
The OWASP AI Exchange lists the threats to AI systems, from input manipulation to model theft, with the controls for each. A good checklist for your first reviews.
ATLAS documents real incidents, such as evading malware classifiers and copying commercial models. Read a few to see that these attacks happen outside the lab.
Accuracy on clean test data says little about behaviour under attack. Learn precision, recall and calibration, and why you need a separate test for robustness.
A backdoored model behaves normally until it sees a hidden trigger, then does what the attacker wants. Backdoors can come from poisoned data or a tampered model file.
These attacks reveal whether a person's record was in the training data, or rebuild what the training data looked like. They turn a model into a privacy leak.
Training on adversarial examples is the most reliable defence against evasion, but it costs accuracy and compute. Measure robustness against strong attacks, not weak ones.
Train a model with differential privacy and explain the accuracy and privacy trade-off.
Assess a federated learning design for poisoning and leakage.
Secure a training pipeline end to end, from raw data to registered model.
Hands-on projects and labs
Train the same model with and without Opacus and compare accuracy and membership inference results.
Simulate a federated setup with Flower and one malicious client, then add a robust aggregation rule.
Train a simple malware classifier on EMBER features and try to evade it with feature changes.
Done when you can…
Choose and justify a privacy budget for a real use case.
Write a robustness and privacy test plan that runs before every model release.
Advanced
Differential privacy
Differential privacy adds calibrated noise so a model cannot reveal much about any one person. Learn the privacy budget (epsilon) and train with DP-SGD.
Federated learning trains on data that stays on devices, but clients can send poisoned updates and the updates can leak data. Learn secure aggregation and robust aggregation.
Watch for drift, unusual query patterns and sudden accuracy drops. They can mean the world changed, or that someone is probing or extracting the model.
Design and run an adaptive attack against a proposed defence.
Write organisation-wide standards for ML robustness and privacy.
Publish or present original ML security research.
Hands-on projects and labs
Reproduce a recent adversarial ML paper, then try an adaptive attack on its defence.
Report a real vulnerability in an open-source ML library through its disclosure process or huntr.
Done when you can…
Have your research, tools or standards used by others.
Lead the security review of a high-risk ML system.
Expert
Reading and doing research
The field moves through papers at venues such as IEEE S&P, USENIX Security, NeurIPS and SaTML. Reproduce results before you trust them, and publish your own.
Machine learning models learn from data, so anyone who can influence the data, the inputs or the outputs can influence the model. Attackers can poison training data, craft inputs that fool a classifier, copy a model through its API, or pull private training records back out of it.
Machine Learning Security is the study of these attacks and the defences against them: robust training, data checks, privacy techniques such as differential privacy, and careful control over what a model reveals. It sits between data science and security, and you need a working knowledge of both.
Why it matters
Models now make decisions about fraud, malware, faces, loans and medical images, and attacks on them are real, documented and catalogued in MITRE ATLAS. Few people understand both ML and security, so the skill is rare and in demand.
Typical job roles
EntryML engineer with a security focusBuilds models and adds the basic checks: data validation, safe file formats, access control.
Mid-levelAdversarial ML researcherTests models with evasion, poisoning and privacy attacks and measures how robust they are.
Mid-levelAI security engineerDesigns defences and monitoring for ML systems in production.
SeniorPrivacy engineer (ML)Applies differential privacy, federated learning and data minimisation to training pipelines.
LeadPrincipal ML security scientistSets the research agenda and the organisation's standards for model robustness and privacy.
Prerequisites
Python, NumPy and one ML framework such as PyTorch or scikit-learn.
Basic linear algebra, probability and calculus: gradients, vectors and distributions.
How models are trained and evaluated: loss, overfitting, train and test splits.
Security basics: threat models, the CIA triad, and how attackers think.