Setting up red teaming for AI agents: the three layers you need from 2026

You must now treat red teaming as a continuous, layered discipline rather than a periodic exercise. Document which tests ran against each workflow, what vulnerabilities they found, and what you did about them. This record becomes your evidence of due diligence.

The prompt is an analysis of 14 September 2026 of autonomous red-team agents and their deployment in production, which argues that organisations should build continuous red-team capability rather than relying on periodic testing. The analysis cites an autonomous agent that discovered over 17,000 unique findings across customer environments in its first month, including a BOLA vulnerability exposing years of passenger data at a major airline. In our assessment, the shift is not that AI finds vulnerabilities—scanners have done that for years—but that the test itself becomes autonomous, repeated and broadly deployable. This moves the security question from when do we test again to how do we keep the test layer itself under control.

What changes when agents have tools and persistent memory?

Classic red-team tests for language models focus on jailbreaks, prompt injection and data leakage from model outputs. Agent-specific red teaming asks different questions. An autonomous agent with email access, file systems, shared memory and shell capabilities presents attack surfaces that a chatbot does not. The adversarial-ML taxonomy of NIST AI 100-2, applied to agentic systems, requires you to test whether attacking instructions hidden in documents, emails or web content can steer an agent's behaviour. Can it bypass approval steps? Can it spread to other agents via shared memory? Can it be misled by instructions embedded in uploaded files? These are the tests that matter once you connect tools and integrations.

How should you structure red teaming across three layers?

Organise your red-team capability as a layered system. The first layer is automated LLM tests in your continuous integration pipeline—fast, repeatable checks for known failure modes. The second layer is periodic agent exercises that test your model and agents against real tools and data, simulating the workflows they will actually run. The third layer is autonomous red-team agents that continuously scan production, discovering vulnerabilities at speed and scale.

Each layer serves a different purpose. Automated tests catch regressions early. Periodic exercises validate that your mitigations work in realistic conditions. Autonomous scanning finds what you did not think to test. Together, they create a test surface that moves with your system.

What failure modes must each layer address?

• Prompt injection and jailbreaks — direct adversarial input to the model designed to override its instructions or safety guardrails.
• Data leakage and memorisation — personal data reproduced or inferred from model outputs or retained in agent memory.
• Tool misuse and privilege escalation — an agent using a connected tool beyond its intended scope or bypassing approval workflows.
• Instruction injection via documents — adversarial instructions embedded in files, emails or web content that an agent processes.
• Agent-to-agent propagation — an agent spreading malicious instructions or state to other agents via shared memory or integrations.
• Autonomous agent compromise — the red-team agent itself becoming a vector for attack if its access rights are not constrained.

What must you be able to demonstrate per workflow?

1. Record the model and its purpose — document which model each workflow uses, the lawful basis for the data it touches, and the tools it can access.
2. Log which tests ran — maintain a record of which red-team tests executed against each workflow, when they ran, and what they tested for.
3. Document vulnerabilities found — record what each test discovered, including the severity, the attack vector and the data or system at risk.
4. Track mitigations implemented — document what you changed in response to each finding, when the change deployed, and whether follow-up testing confirmed it worked.
5. Maintain audit evidence — keep this record available for internal risk committees and external audits, organised by workflow and by time.

What does a verification layer add?

A verification layer sits above your red-team architecture and makes visible per workflow which tests carried out, which vulnerabilities the tests found in models and agents, and which mitigations you implemented in response. This visibility layer does not replace the red-team tests themselves and does not guarantee their technical correctness. The professional judgement on findings and mitigations remains with your security and risk team. What it does is create a single source of record that you can present as evidence of testing and due diligence.

Automated tooling can track which tests ran, aggregate findings and flag when a vulnerability remains open. What it cannot do is decide whether a finding is real, whether a mitigation is sufficient, or whether the test itself was well designed. That remains your responsibility. Use tooling to make your testing visible and auditable; use your own expertise to make it sound.