Secure chatbots, copilots and other apps built on large language models, from prompt injection and data leaks to RAG and output handling.
Estimated time: 12–23 months · At about 10 hours a week.
Offensive skills are for authorised testing only: your own lab, a practice platform, or a system whose owner has given you written permission. Testing anything else is a crime in most countries.
Learning path
Click a topic to see what to learn and where. Mark topics as you go; your progress is saved in this browser only.
Explain prompt injection and why filtering alone cannot stop it.
Review an LLM feature against the OWASP Top 10 for LLM Applications.
Find and demonstrate an injection in a practice lab.
Hands-on projects and labs
Complete the Gandalf levels and the PortSwigger Web LLM attacks labs.
Build a small chatbot with a free or local model (for example with Ollama) and try to make it reveal its system prompt.
Hide an instruction in a web page and show that a summarising bot you built follows it.
Done when you can…
Explain the difference between direct and indirect prompt injection with an example of each.
Write a short security review of a chatbot design using the OWASP list.
Beginner
How LLMs and chat apps work
Tokens, context windows, system prompts, temperature and tool calls. You do not need the maths, but you do need to know what the model sees on each request.
The standard list of LLM app risks: prompt injection, sensitive information disclosure, supply chain, poisoning, improper output handling, excessive agency and more.
Jailbreaks trick a model into ignoring its safety training, through role play, encoding tricks or automated search. They matter most when the model's output reaches other people or systems.
Models can repeat secrets from their prompt, their context or their training data. Never put credentials in prompts, and only give the model data the user may see.
Build a RAG app that enforces document permissions per user.
Encode and validate model output before it reaches a browser, database or shell.
Add guardrails and measure how many attacks still get through.
Run garak or promptfoo against an app and triage the findings.
Hands-on projects and labs
Build a RAG app over a few documents with different owners, then try to read another user's document through the chat.
Make a chatbot render model output as HTML, exploit it with XSS, then fix it.
Add promptfoo red-team tests to a small app's CI pipeline.
Done when you can…
Ship a small LLM app that passes your own security test suite.
Explain to a developer why the system prompt is not a security control.
Intermediate
Improper output handling
Treat model output like untrusted user input. If it becomes HTML, SQL, a shell command or a URL, encode or validate it first, or you get XSS, injection and data theft.
Retrieval-augmented generation feeds documents into the prompt. Check permissions at retrieval time, so the model only sees what this user may see, and treat retrieved text as untrusted.
Embeddings can leak the text they came from, and shared vector stores can mix tenants' data. Apply access control, tenant separation and encryption like any database.
Input and output classifiers catch many attacks and harmful outputs, but they are a layer, not a fix. Measure what they miss and keep hard limits in code.
Scanners send thousands of known attack prompts and score the replies. Run them in CI so a new prompt or model does not quietly bring back old problems.
AI coding assistants can suggest insecure code and packages that do not exist, which attackers then register. Review and scan generated code like any other code.
Run a scoped penetration test of an LLM application and write a report with impact.
Design logging and detection for prompt injection and data exfiltration.
Assess a model provider's security and privacy terms.
Hands-on projects and labs
Test an intentionally vulnerable LLM app you deployed yourself, and write a full report.
Run PyRIT against your own app and compare its findings with manual testing.
Build a dashboard of prompts, tool calls and blocked requests for a small app.
Done when you can…
Lead the security review of a production LLM feature.
Show a detection that catches a real indirect injection in your lab.
Advanced
Testing LLM applications
Plan an LLM test like a web app test: scope, threat model, then injection, data exposure, output handling and abuse of integrations. Report impact, not just a funny reply.
Tools can generate and mutate attack prompts at scale and score the results with another model. Check the scoring yourself, since judges make mistakes too.
Check data retention, training on your data, regional hosting, abuse monitoring and the shared responsibility split. Use the provider's own safety settings where they exist.
The strongest defences limit what an injected instruction can do: separate trusted and untrusted data, restrict tools after reading untrusted text, and keep a human in the loop.
NIST's profile lists the risks specific to generative AI and the actions to manage them, organised by the AI RMF functions. Use it to structure a programme.
Large language models (LLMs) now sit inside search boxes, help desks, coding assistants and internal tools. They take text from users, documents and websites, and they cannot reliably tell instructions from data. That single weakness, prompt injection, is behind many of the attacks in this field.
LLM security is about the whole application around the model: what data it can see, what it can do with its output, how it is grounded with retrieval (RAG), and how you test and monitor it. Much of it is classic application security, applied to a new and unpredictable component.
Why it matters
Almost every organisation is shipping or buying generative AI, and most of the security problems are still being worked out. OWASP, NIST and national cyber agencies now publish guidance, and teams need people who can apply it.
Typical job roles
EntryApplication security engineer (AI apps)Reviews LLM features for prompt injection, data leaks and unsafe output handling.
Mid-levelAI security engineerDesigns guardrails, access control and monitoring for LLM applications.
Mid-levelLLM penetration testerTests chatbots and copilots for injection, jailbreaks and data exposure, with permission.
SeniorAI security architectSets patterns for how teams build RAG, assistants and integrations safely.
LeadHead of AI securityOwns AI security strategy, policy and the review process for new AI features.
Prerequisites
How web applications and APIs work: HTTP, JSON, authentication and sessions.
Basic Python, enough to call a model API and write small scripts.
The OWASP Top 10 for web applications, at least at a beginner level.
No ML maths needed to start; the Machine Learning Security roadmap goes deeper.