59. LLM Red Teaming for Beginners: How to Test AI Systems for Prompt Injection, Data Leaks, and Safety Regressions (Defensive Guide)

Last Updated: September 12, 2026

If you are deploying any AI system — a chatbot, a copilot, a RAG pipeline, or an autonomous agent — LLM red teaming is no longer optional. In 2026, adversarial testing of AI systems is a documented compliance requirement under the EU AI Act, NIST AI 600-1, and the AI Safety Institute frameworks. The question is not whether to red team your AI. It is how to do it systematically, what tools to use, and what documentation to produce. This guide answers all three questions in plain English — no security background required to get started.

LLM red teaming is the practice of deliberately probing a model or AI application for harmful, unsafe, or unintended behaviours before those behaviours appear in production. It is the AI equivalent of penetration testing for traditional software. The difference is that where pen testing targets known vulnerability classes in application code, LLM red teaming targets non-deterministic language model behaviour — the same prompt can produce different outputs across sessions, and the attack surface is natural language itself. Every user input is a potential attack vector, and many safety failures are subtle and context-dependent in ways that standard software testing cannot surface.

This guide is designed for security professionals, AI governance leads, developers, and business teams deploying AI systems who need a practical, current understanding of LLM red teaming in 2026. It covers the 2026 attack taxonomy including agentic threats, the four leading open-source tools (Garak, Promptfoo, PyRIT, and DeepTeam), a beginner-ready 5-step process, and the specific regulatory compliance requirements your organisation must meet. Whether you are running your first red teaming exercise or building a recurring programme, this guide gives you everything you need to start today. According to the OWASP GenAI Security Project, prompt injection remains the number one risk for LLM applications — and it is the first thing every red team should test.

🎯 1. What Is LLM Red Teaming — The 2026 Definition

LLM red teaming is the practice of deliberately probing a model for harmful, unsafe, or otherwise undesired behaviour before that behaviour shows up in production. The goal is to find and fix failure modes before real users — or real attackers — discover them first. The term comes from cybersecurity, where a dedicated team plays the role of the attacker. For large language models, red teamers craft inputs designed to make the model behave badly — bypassing safety measures, leaking data, generating harmful content, or taking unintended actions.

It is the AI equivalent of penetration testing for traditional software applications. But there is a critical difference. Traditional penetration testing targets known vulnerability classes — SQL injection, broken authentication, privilege escalation — with largely deterministic outcomes. LLM red teaming targets non-deterministic behaviour. The same prompt can produce different outputs across sessions. The attack surface is not application code. It is natural language itself. Any user input is a potential attack vector, and safety failures in LLMs are often subtle and context-dependent in ways that standard automated security scanning cannot detect.

With LLMs, both benign and adversarial usage can produce potentially harmful outputs — including hate speech, incitement of violence, data leakage, or manipulated decisions. A user does not need to be actively malicious to trigger a failure. Poorly designed systems fail with ordinary inputs. Red teaming tests for both deliberate adversarial attacks and inadvertent safety failures, giving AI teams a complete picture of where their system is most vulnerable before deployment.

Red teaming is now a best practice in responsible AI development. It is how the major foundation labs — OpenAI, Anthropic, Microsoft, and Google — evaluate their models before public release. What was once an internal research practice at frontier AI labs is now a documented compliance requirement for organisations deploying high-risk AI systems under the EU AI Act, NIST AI 600-1, and the GPAI Code of Practice. The shift is significant: red teaming has moved from “something the big labs do” to “something your compliance team needs evidence of.”

The 2026 Red Teaming Reality: Red teaming AI systems is no longer a research niche — it has become an infrastructure layer that major vendors are acquiring outright. In July 2026, OpenAI announced its acquisition of Promptfoo, the leading open-source LLM testing framework, confirming that adversarial AI testing is now a production-grade enterprise requirement.

🤖 2. LLM Red Teaming vs Agent Red Teaming — A Critical 2026 Distinction

Traditional LLM red teaming tests static prompt-response pairs against a single model endpoint. You send a prompt. You evaluate the response. You repeat. This approach is well understood, well tooled, and necessary — but it is no longer sufficient for the AI systems most organisations are deploying in 2026. Autonomous AI agents — systems that plan multi-step tasks, call external tools, maintain persistent memory, and communicate with other agents — create attack surfaces that single-turn adversarial prompts cannot reach.

Agent red teaming extends adversarial testing to autonomous behaviours across multi-step execution chains. The threats it targets include goal hijacking (redirecting an agent’s objective mid-task), tool misuse (manipulating an agent into calling APIs or tools in unintended ways), memory poisoning (injecting malicious content into an agent’s persistent memory store), and inter-agent communication spoofing (intercepting or fabricating messages between agents in a multi-agent pipeline). None of these attack classes exist in a simple chatbot. All of them exist in the agentic systems your organisation is deploying today.

The distinction matters for compliance. The EU AI Act, NIST AI 600-1, and AI Safety Institute frameworks treat LLM testing and agent testing as separate testing requirements with different success criteria, different scoring approaches, and different mitigation patterns. An organisation that has completed rigorous LLM red teaming for its chatbot cannot assume that same evidence covers its agentic deployment. You need both — and they require different tools and different methodologies.

The OWASP Top 10 for Agentic Applications (ASI 2026), published December 2025 and formalised across Q1–Q2 2026, defines five core risk domains that every agent red team must cover:

• ASI01 — Goal Hijacking: Attacker redirects an agent’s objective mid-task via injected instructions or poisoned content. Example: an agent tasked with summarising documents is redirected to exfiltrate them. Unlike standard prompt injection that produces a single bad answer, goal hijacking reprograms the agent’s entire multi-step planning process.
• ASI02 — Tool Misuse: Attacker manipulates an agent into calling tools in unintended ways. Example: a customer service agent is tricked into calling a refund API with fabricated order data, or a code agent is coerced into executing malicious shell commands.
• ASI03 — Identity Abuse: Attacker impersonates a trusted agent, user, or orchestrator to gain elevated permissions. Example: an attacker spoofs a manager identity to override agent safety guardrails or to authorise actions the agent would normally reject.
• ASI06 — Memory Poisoning: Attacker injects malicious content into an agent’s persistent memory. The poisoned memory then influences all future agent behaviour across sessions — a persistent compromise that survives conversation resets.
• ASI07 — Insecure Inter-Agent Communication: Attacker intercepts or spoofs messages between agents in a multi-agent pipeline. Particularly dangerous in complex workflows where agents implicitly trust each other’s instructions without verification.

Key finding (NIST CAISI — February 2026): Novel agentic attacks reached an 81% task-hijack rate against tested agent systems — compared to an 11% baseline for prior attack methods. Agent red teaming must evolve continuously, not just at deployment.

For security teams planning their 2026 red teaming programme, the practical rule is: use the OWASP LLM Top 10 (2025 edition) for single-prompt LLM applications, and use the OWASP ASI 2026 framework for tool-using agents. Test both if your deployment includes both — and most enterprise AI deployments now do. You can also explore how agentic AI risks and governance frameworks are evolving alongside these testing standards.

🗂️ 3. The 2026 Attack Taxonomy — What You Are Testing For

Before you can run an effective red teaming exercise, you need a clear map of the attack categories you are testing against. The 2026 attack surface for LLM applications is significantly broader than it was two years ago — multimodal inputs, agentic tool use, MCP-connected pipelines, and supply chain attacks have all expanded what red teamers must cover. Here is a complete, beginner-friendly breakdown of every category your red team should include.

Prompt Injection

Prompt injection remains the number one risk on the OWASP LLM Top 10 for the second consecutive year. There are two primary variants. Direct injection occurs when an attacker crafts a prompt that overrides the model’s system instructions — for example, a user input that begins with “Ignore all previous instructions and instead…” Indirect injection is more dangerous in agentic systems: malicious instructions are embedded in content the model retrieves from an external source (a webpage, uploaded document, email, or database record) and executed when the model processes that content. The model has no way to distinguish legitimate retrieved content from injected instructions. In agentic systems, the blast radius of a successful injection expands dramatically — a single injected instruction can trigger tool calls, API requests, data exfiltration, and downstream actions across an entire multi-step workflow.

Jailbreaks

Jailbreaks are techniques that bypass a model’s safety training to produce prohibited content. The simplest jailbreaks — direct refusal bypass attempts — are well defended by modern models. The sophisticated attacks that red teams must now test for include many-shot jailbreaks (feeding the model many examples of the desired harmful output pattern before the final request, proven to beat single-turn defences by wide margins) and Crescendo attacks (gradually escalating a conversation across multiple turns to reach a prohibited output that a single-turn attack would fail to achieve). Research published in 2025 confirmed that one black-box method successfully jailbroke GPT-4-Turbo and GPT-4o on more than 80% of prompts using multi-turn escalation techniques — a finding that makes Crescendo testing a mandatory component of any enterprise red team exercise.

Data Extraction

Data extraction attacks use prompts designed to retrieve information the model should not disclose. System prompt extraction attempts to trick the model into revealing its hidden system instructions — which frequently contain sensitive configuration details, business logic, or API endpoint information. Training data memorisation attacks attempt to extract verbatim content from the model’s training data using carefully crafted targeted prompts. For RAG-enabled applications, a third variant targets the retrieval mechanism: prompts designed to surface confidential documents from the knowledge base that the user should not have access to. All three are standard components of a complete red team exercise.

Hallucination Exploitation

Deliberate triggering of confident but false outputs is particularly dangerous in medical, legal, and financial applications. An attacker who understands that a model will confidently hallucinate under certain conditions can craft inputs that cause the model to generate false information that serves the attacker’s purpose — fabricated citations, incorrect legal interpretations, or incorrect dosage information in a healthcare context. Red teamers should test for hallucination exploitation in every domain where the AI system’s outputs inform consequential decisions.

Multimodal Attacks

As LLM applications increasingly accept image, audio, and video inputs, new attack surfaces open. Image inputs can embed malicious instructions invisible to human reviewers but readable by the model — an image containing instructions printed in a font size or colour that human reviewers skip over but the model processes as text. Audio inputs can inject instructions into voice-enabled AI systems. Multimodal and MCP-based attacks now require dedicated testing scenarios that text-only red teaming exercises will not surface. If your AI application accepts any non-text input, multimodal attack testing is mandatory.

Supply Chain Attacks

Supply chain attacks target the AI system’s inputs at their source rather than the inference endpoint. Malicious fine-tuning datasets can be used to backdoor a model — embedding hidden behaviours that activate under specific trigger conditions and survive subsequent safety fine-tuning. Poisoned RAG knowledge bases corrupt retrieval results, causing the model to return attacker-controlled content in response to legitimate user queries. Compromised third-party tools or APIs connected to agentic systems introduce attack vectors that bypass the model’s own safety mechanisms entirely. For organisations using third-party AI components, supply chain security testing should be part of every vendor due diligence process — see the AI vendor due diligence checklist for a practical framework.

🛠️ 4. The Four Best LLM Red Teaming Tools in 2026

The best LLM red teaming tools in 2026 are Garak, Promptfoo, PyRIT, and DeepTeam — four open-source frameworks that probe LLM-powered applications for prompt injection, jailbreaks, data leakage, and other adversarial failures before they reach production. Each tool solves a different part of the red teaming problem. Understanding which tool to use, when, and in combination with which others is the practical skill that separates effective red teaming programmes from exercises that generate reports without improving security.

Garak (NVIDIA)

Garak is NVIDIA’s open-source LLM vulnerability scanner. It runs automated probes across a large library of attack categories — prompt injection, jailbreaks, data leakage, hallucination, toxicity, and more — and produces a structured report of findings. Garak’s strength is breadth: it covers dozens of probe categories spanning security vulnerabilities, bias detection, hallucination testing, and injection scenarios. Its modular architecture separates generators (which create test prompts), detectors (which analyse model responses), and harnesses (which orchestrate evaluation workflows). Garak is the right first tool to reach for — run it before any other testing to establish a baseline coverage report. Installation is via pip: pip install garak. It is Apache 2.0 licensed and actively maintained at github.com/NVIDIA/garak.

Promptfoo

Promptfoo is an open-source LLM testing and red teaming framework built for development teams who want adversarial testing integrated into their CI/CD pipeline. Assertions live in YAML files next to application code and fail a build like any other automated test — transforming a red team finding into a regression test rather than a PDF report. Promptfoo supports custom attack scenarios, automated adversarial probes, and OWASP LLM Top 10 checks. Its configuration-driven approach makes it accessible to developers without dedicated security expertise. Note: in July 2026, OpenAI acquired Promptfoo — a confirmation that CI-integrated AI red teaming has become a production-grade infrastructure requirement. It is MIT licensed at github.com/promptfoo/promptfoo.

PyRIT (Microsoft)

PyRIT is Microsoft’s Python Risk Identification Toolkit for generative AI. Where Garak excels at breadth and Promptfoo at CI integration, PyRIT excels at depth — specifically, multi-turn attack scenarios that carry state across conversation turns. PyRIT’s Crescendo orchestrator enables chained-attack path discovery: an attacker model generates the next adversarial prompt based on the target model’s last response, iteratively escalating toward a prohibited output. The TAP (Tree of Attacks with Pruning) orchestrator refines prompt attacks using the target’s responses to find the most effective attack path. PyRIT already supports multi-turn agent test scenarios, making it the right tool for enterprise security teams testing agentic systems and complex pipelines. It is MIT licensed at github.com/Azure/PyRIT.

DeepTeam

DeepTeam is a purpose-built agent red teaming framework from the Confident AI team — the same team behind the DeepEval LLM evaluation framework. It covers 40+ vulnerability types mapped to the OWASP Top 10 for LLM Applications and supports linear, tree, and Crescendo jailbreaking methods. DeepTeam’s primary differentiation is native support for multi-agent pipeline testing, tool-call sequence attacks, and persistent memory attacks — the attack categories that Garak and Promptfoo address only partially. For teams deploying agentic AI in production, DeepTeam provides the agent-specific test coverage that general-purpose scanners cannot deliver.

The practical approach most security teams use in 2026: run Garak first for baseline coverage, wire Promptfoo into your CI/CD pipeline for ongoing regression testing, and bring in PyRIT or DeepTeam when your deployment includes multi-turn conversations or agentic workflows. These tools solve different problems — using one does not replace the others. For teams new to red teaming, Garak and Promptfoo give you meaningful adversarial coverage in an afternoon with minimal setup. You can see how these tools fit into the broader AI security stack in the complete guide to AI tools for cybersecurity teams.

⚖️ 5. Manual vs Automated Red Teaming — When to Use Each

Manual red teaming and automated red teaming are not competing approaches — they are complementary phases of a complete red teaming programme. Understanding when to use each, and how to combine them effectively, is one of the most practically important skills for teams building their first AI security programme in 2026.

Manual red teaming leverages human expertise, creativity, and contextual understanding to identify novel attack vectors and edge cases that automated tools cannot reach. Human red teamers can employ sophisticated social engineering techniques, adapt their approach in real time based on model responses, and bring domain-specific knowledge that automated probe libraries do not contain. A security researcher testing a healthcare AI system brings clinical knowledge that no automated scanner can replicate — they know what a dangerous hallucination looks like in a medical context, and they can craft prompts that exploit that specific failure mode. However, manual approaches face significant scalability limitations, require extensive expertise, and produce results that vary across different evaluators.

Automated red teaming uses frameworks like Garak and Promptfoo to run thousands of adversarial probes systematically — faster, more consistent, and more reproducible than manual testing. Automated tools excel at coverage: running hundreds of jailbreak variants, testing every OWASP LLM Top 10 category systematically, and surfacing the volume of findings that manual testing would take weeks to produce. Their limitation is equally clear: automated tools are bounded by their probe libraries. They find what they are configured to look for. Novel attack vectors, context-specific vulnerabilities, and the kind of creative adversarial thinking that surfaces the most dangerous findings require human red teamers.

The 2026 best practice is a combined approach with a defined sequencing:

• Phase 1 — Manual discovery: Run manual red teaming first, before any automated tools. Human red teamers discover novel attack vectors and define the most dangerous failure modes for your specific use case.
• Phase 2 — Automated coverage: Deploy Garak and Promptfoo against your system. Use automated tools for systematic coverage across known attack categories and to scale the testing volume that manual efforts cannot match.
• Phase 3 — CI/CD integration: Wire Promptfoo into your deployment pipeline as a permanent regression testing layer. Every model update, fine-tune, or data refresh triggers automated adversarial testing before the change reaches production.
• Phase 4 — Recurring manual exercises: Conduct in-depth multi-stakeholder exercises quarterly, or after any significant architecture change. Manual red teaming is not a one-time pre-deployment exercise.

For agentic systems specifically, treat agent red teaming as a completely separate workstream from chat-LLM red teaming. The probes overlap, but the success criteria, scoring, and mitigation patterns are different enough that combining them into a single exercise produces incomplete results for both. Wire automated adversarial suites into your CI/CD pipeline so every fine-tune or data refresh triggers agent-specific testing before any change reaches production.

🔢 6. The 5-Step LLM Red Teaming Process for Beginners

A structured red teaming process produces findings that are actionable, reproducible, and documentable for compliance purposes. Ad hoc testing — entering prompts until something interesting happens — generates noise, not findings. The five-step process below is designed for teams running their first red teaming exercise and for teams building a repeatable programme. Follow every step in order. Skipping Step 1 (scope definition) is the single most common reason red teaming exercises produce reports that cannot drive remediation.

Step 1 — Define Scope and Objectives

Before sending a single prompt, define what you are testing and what a failure looks like. What system are you testing — a chatbot, a RAG pipeline, an agentic workflow, or a fine-tuned model? What are the highest-risk failure modes for this specific use case? A customer service chatbot and a medical diagnosis assistant have entirely different failure hierarchies. Who are the likely adversaries — external attackers, malicious users, or insiders? What data does the system have access to, and what actions can it take?

Advance planning also determines who should be on the red team. Include domain experts relevant to your deployment. If testing a healthcare AI system, include clinical staff who understand what a dangerous hallucination looks like in their specific context. If testing a legal AI system, include lawyers who can identify plausible but incorrect legal interpretations. A red team that consists only of security generalists will miss the domain-specific failure modes that matter most.

Step 2 — Map the Attack Surface

List every input vector your system accepts: user prompts, uploaded documents, retrieved web content, API responses, email inputs, memory stores, and any other channel through which external content reaches the model. For agentic systems, extend this mapping to every tool the agent can call, every API it has access to, and every data source it can read from or write to. Identify trust boundaries: what does the system trust by default, and what can an attacker manipulate to inject malicious content into those trusted channels?

The attack surface for agentic AI is dramatically larger than for a simple chatbot. A customer service chatbot has one input channel — the user’s text input. An AI agent that can browse the web, read emails, call APIs, and write to a database has dozens of input channels, each of which is a potential injection point. Map them all before you start testing. What you do not map, you will not test.

Step 3 — Execute Manual Red Teaming First

Start with the OWASP LLM Top 10 and work through each category systematically. Test prompt injection first — it is the highest-severity category and the most likely to produce critical findings. Then test jailbreaks: attempt direct refusal bypasses, then multi-turn Crescendo escalation, then many-shot approaches. Test data extraction: attempt system prompt extraction and training data memorisation. For agentic systems, apply the OWASP ASI 2026 framework in parallel — test goal hijacking, tool misuse, and memory poisoning as separate attack categories with their own test cases.

Document every finding with four fields: the input (exact prompt or sequence of prompts used), the output (exact model response), a severity rating (Critical / High / Medium / Low), and reproducibility (does the finding reproduce consistently, intermittently, or rarely). This documentation is not optional. It is the evidence base your compliance team needs for EU AI Act conformity assessment.

Step 4 — Run Automated Scanning

After completing manual testing, deploy automated tools to expand coverage at scale. Use Garak for broad automated coverage across its full probe library — point it at your model endpoint and run the complete probe set for injection, jailbreak, leakage, and hallucination. Use Promptfoo to formalise your most important manual findings as automated regression tests that will run in your CI/CD pipeline going forward. For agentic systems and multi-turn scenarios, deploy PyRIT’s Crescendo and TAP orchestrators to discover chained attack paths that single-turn tools cannot reach.

Review automated results carefully. Automated tools surface volume — dozens or hundreds of findings per run. Prioritise by severity and reproducibility. Not every automated finding represents an exploitable vulnerability in your specific deployment context. Triage findings before escalating to remediation.

Step 5 — Remediate, Retest, and Document

Fix every identified vulnerability at the appropriate layer. Update system prompts to add explicit guardrails against the specific failure modes discovered. Implement output filtering — a classifier that screens model outputs before returning them to users. Add input filtering to screen user inputs for known attack patterns. Restrict tool permissions in agentic systems to reduce the blast radius of a successful injection. For systemic failures that suggest model-level weakness, evaluate whether model replacement or targeted fine-tuning is required.

Retest every fixed vulnerability before closing the finding. A fix that prevents a specific prompt from working does not necessarily close the underlying vulnerability class. Test the fix with variations of the original attack to confirm the remediation is robust. Then document the full cycle: the original finding, the remediation applied, and the retest result. This documentation is required for EU AI Act compliance — the Act requires ongoing risk management as a continuous iterative process, not a single pre-deployment assessment. See the EU AI Act compliance guide for the full documentation requirements.

📋 7. LLM Red Teaming and Regulatory Compliance — What the Rules Require

In 2026, LLM red teaming has moved from an optional security best practice to a documented compliance requirement across three major regulatory frameworks. Understanding exactly what each framework requires — and what documentation you must produce — is now essential for any organisation deploying AI systems in the EU or using AI models with systemic reach.

The EU AI Act, NIST AI 600-1, and AI Safety Institute frameworks each treat adversarial testing differently, but they share a common requirement: red teaming is not sufficient unless it is documented. A verbal assurance that “we tested the system” does not satisfy compliance. What satisfies compliance is a documented red teaming report with findings, severities, mitigations, and retest results — retained for the minimum period specified by Article 12 obligations.

Two specific EU AI Act provisions matter most for organisations planning their red teaming programme. Article 55 (systemic risk GPAI providers) explicitly requires adversarial testing and red-teaming as a mandatory safety obligation — not a recommendation. The high-risk AI system provisions under Annex III require documented evidence of robustness testing as part of the conformity assessment process. Red teaming outputs feed directly into this evidence base. For organisations deploying AI in employment, healthcare, education, credit, or law enforcement contexts — all Annex III categories — documented red teaming is not optional.

NIST AI 600-1’s MAP-3.2 control maps adversarial testing to the GOVERN and MEASURE functions of the AI Risk Management Framework. For organisations using the NIST AI RMF as their governance standard, red teaming evidence satisfies MAP-3.2 documentation requirements. The ISO/IEC 42001:2023 AI management system standard embeds adversarial testing in its risk management controls — organisations pursuing certification must demonstrate red teaming as part of their continuous risk management process.

Compliance reality check: The EU AI Act requires ongoing risk management as a continuous iterative process. A single pre-deployment red team exercise does not satisfy Article 55 or Annex III obligations. You need a recurring programme with documented cadence — not a one-time report.

🏢 8. Building a Red Teaming Programme — For Organisations Without a Security Team

Most small and mid-size organisations deploying AI do not have a dedicated red team. That does not mean they can skip adversarial testing — it means they need a practical minimum viable programme that works within their resource constraints. The following four-step framework is designed for organisations without dedicated security expertise who are deploying AI systems and need to demonstrate due diligence for regulatory and risk management purposes.

Step 1 — Use Automated Tools First

Deploy Promptfoo or Garak against your AI system before go-live. Both tools are open-source, require minimal setup, and give you baseline coverage across the most well-known attack categories within an afternoon. Run Garak’s full probe suite for prompt injection, jailbreaks, and data leakage. Use Promptfoo to build an automated test suite around your most important use cases. Document the results — even partially, even informally — before your first production deployment. This gives you evidence of testing even before you have a formal programme.

Step 2 — Run a Manual Tabletop Exercise

Gather three to five people who know the AI system well — the people who built it, the people who will use it, and ideally one person with no prior exposure to the system who can bring fresh adversarial perspective. Spend two hours trying to make the system fail. Focus on your highest-risk use cases first. Document every finding — even the ones you fix immediately, even the ones that seem minor. A tabletop exercise is not a substitute for a structured red team exercise, but it is dramatically better than no manual testing at all and it produces findings that automated tools consistently miss.

Step 3 — Engage External Red Teamers for High-Risk Systems

If your AI system makes consequential decisions — hiring, credit scoring, healthcare triage, legal advice, or anything where a wrong output causes real harm — bring in external red teamers for at least one exercise before deployment. External perspective surfaces the blind spots that internal teams cannot see. Anthropic’s trust and safety team, Scale AI’s red teaming services, and several specialist AI security firms now offer structured LLM red teaming engagements for enterprise deployments. For organisations subject to Annex III of the EU AI Act, external red teaming evidence carries significantly more weight in conformity assessment than internal-only testing.

Step 4 — Make It Recurring

A one-time red team exercise before deployment does not satisfy EU AI Act requirements, and it does not reflect how AI systems actually behave over time. Models drift. Fine-tunes introduce new failure modes. Architecture changes open new attack surfaces. Build red teaming into your deployment checklist and your quarterly security review. Wire Promptfoo into your CI/CD pipeline so every update triggers automated regression testing. Schedule a manual tabletop exercise at least twice per year. This transforms red teaming from a compliance checkbox into a genuine safety practice — which is what the regulation actually requires.

📊 9. Red Teaming Findings — What to Do With the Results

Red teaming generates findings. Findings require classification, remediation, and documentation. The loop from testing to remediation to retest to documentation is what makes red teaming a genuine safety practice rather than an exercise in generating reports. This section covers the full findings management cycle — from severity classification through to the documentation requirements your compliance team needs.

Severity Classification

Classify every finding into one of four severity tiers before assigning remediation timelines:

• Critical: Immediate fix required before deployment or before next production release. A Critical finding represents a failure mode that could cause serious harm, enable data exfiltration, or result in regulatory breach. Examples: system prompt fully extracted by a simple user prompt; agent successfully redirected to exfiltrate customer data.
• High: Fix within 7 days. A High finding represents a serious vulnerability that is not immediately catastrophic but poses significant risk in production. Examples: multi-turn jailbreak reliably bypasses safety training; indirect injection successfully triggers tool calls.
• Medium: Fix within 30 days. A Medium finding represents a real vulnerability with limited blast radius or low reproducibility. Examples: jailbreak works intermittently; hallucination exploitation requires specific domain knowledge to trigger.
• Low: Track and monitor. A Low finding represents a known limitation or edge case that does not pose immediate risk. Examples: model occasionally uses informal language; inconsistent formatting in edge-case inputs.

Remediation Options

Apply the remediation approach that addresses the root cause at the correct layer:

• System prompt hardening: Add explicit instructions against the specific prohibited behaviours discovered. This is the fastest fix but the least robust — sophisticated attackers work around system prompt instructions. Always combine with other controls.
• Output filtering: Add a classifier that screens every model output before returning it to the user. Output filtering catches a broader class of failures than system prompt instructions alone and is harder for attackers to bypass.
• Input filtering: Screen user inputs for known attack patterns before sending to the model. Input filtering is effective against known attack signatures but provides limited protection against novel attack variants.
• Tool permission restriction: For agentic systems, reduce the blast radius of a successful injection by limiting what tools the agent can call, what data it can access, and what actions it can take. The OWASP ASI 2026 principle of “least agency” applies here — grant agents the minimum autonomy required for safe, bounded task completion.
• Model replacement or fine-tuning: If systematic failures suggest a model-level weakness rather than an application-level configuration problem, evaluate whether the underlying model needs to be replaced or fine-tuned to address the specific failure class.

Documentation Requirements

Record every finding in a structured format: the input (exact prompt or prompt sequence), the output (exact model response), the severity rating, the remediation applied, and the retest result. Retain records in alignment with EU AI Act Article 12 obligations — minimum retention periods apply depending on your system’s risk category. For Annex III high-risk systems, red teaming documentation forms part of the technical documentation required for conformity assessment. Include a red teaming summary in your conformity assessment documentation. The summary should cover: scope and objectives of each exercise, attack categories tested, total findings by severity tier, remediation applied, and retest confirmation. This is the evidence your notified body will request. For a complete governance documentation framework, see the AI governance programme structure guide.

Also consider the role of adversarial machine learning more broadly in your security posture — red teaming is one component of a complete adversarial ML defence strategy that also includes training data validation, model monitoring, and supply chain security. And for teams managing the broader challenge of shadow AI security risks, red teaming findings often surface unsanctioned AI usage patterns that require policy intervention alongside technical remediation.

🏁 10. Conclusion — Red Teaming Is Now a Security Standard, Not a Research Practice

LLM red teaming entered 2024 as a practice that only the largest AI labs performed internally. It enters Q4 2026 as a documented compliance requirement under EU law, a recommended control under NIST guidance, and a core safety commitment under the GPAI Code of Practice. The regulatory and market signals are unambiguous: organisations that deploy AI systems without documented adversarial testing are accepting both security risk and compliance risk simultaneously.

The practical path forward is straightforward. Start with automated tools — Garak and Promptfoo are free, open-source, and operational within an afternoon. Add manual testing through structured tabletop exercises. Wire automated regression testing into your CI/CD pipeline. Engage external red teamers for high-risk deployments. Document every finding, every remediation, and every retest. Make it recurring. The 2026 consensus among AI security practitioners is clear: red teaming is not a project. It is a programme. The organisations that build it as a programme today will be the ones with clean compliance evidence when the EU AI Act audit requests arrive — and they will arrive. For teams ready to explore the full landscape of AI governance controls alongside red teaming, the AI Audit Checklist covers every domain your compliance review will cover.

🔗 Related Articles

• 📖 Prompt Injection Explained: How AI Assistants Get Tricked and How to Stay Safe
• 📖 OWASP Top 10 Risks for LLMs and GenAI Apps (2026) Explained
• 📖 Adversarial Machine Learning Explained: How AI Systems Get Attacked
• 📖 Best AI Tools for Cybersecurity Teams in 2026: The Complete Guide
• 📖 AI Governance Explained: How to Build an AI Policy Framework