An LLM red-team exercise is useful when it reveals a failure you can reproduce, explain and fix. Collecting dramatic prompts without an application boundary or an outcome definition produces little evidence about a deployed agent.
This guide focuses on authorized testing of systems you own or have permission to assess. It provides a runbook and result schema, not a claimed security score for a model or product.
Define the trust boundary
OWASP’s prompt-injection description covers adversarial instructions entering through prompts or external content. Its prevention guidance recommends layered controls instead of treating natural-language instructions as an authorization system.
PyRIT and garak provide tooling for organizing or running evaluations. Tool availability does not establish that a particular application passed a meaningful test.
For an agent, enumerate data sources, available tools, credentials and side effects. A model can refuse in prose while an application still executes a harmful action. Conversely, an alarming generated sentence may produce no unauthorized operation. Record both behavior and actual effects.
Write a bounded test charter
The red-team case sheet records the system version, entry point, asset, attempted policy violation, expected control and observed outcome.
Specify the allowed environment, accounts, execution budget and stop condition before starting. Use synthetic records and harmless sentinel strings instead of customer data or live credentials. Route any simulated external destination to a test sink you control.
Start with application-specific failures: another tenant’s record, an unauthorized tool operation, a retrieved document pretending to override instructions, or a repeated request that would duplicate a side effect. Include a valid authorized request so denial of everything is not mistaken for success.
Record enough to reproduce a failure
Retain model/version, system configuration, tool schemas, retrieved fixture, request sequence, sampling settings and timestamps. Save the actual tool request and the application’s authorization decision. Redact secrets before storing or sharing the record.
For a state-changing operation, inspect the resulting state rather than relying on the assistant’s narration. For an information disclosure, compare the returned content with the synthetic sentinel. For an availability issue, record the resource limit and recovery path.
Re-run cases because model outputs may vary. Preserve all attempts and the tested configuration. A repeatable application-policy failure can be deterministic even when the language model is not.
Metrics with honest denominators
Define “success” before counting it. Separate attempted unauthorized tool calls, actually authorized calls, completed harmful side effects and textual policy violations. Report the number of attempts and cases represented.
If one case is attempted repeatedly, do not count every attempt as an independent vulnerability. Report both case coverage and attempt-level outcomes. When a model refuses, verify that the application did not perform the disallowed action through another path.
Track valid-request completion too. A mitigation that blocks legitimate work can look secure while making the product unusable. Avoid claiming that a finite test set proves the absence of all prompt-injection vulnerabilities.
Turn findings into regression controls
Fix the failing boundary where possible: server-side authorization, bounded tool inputs, controlled data access and explicit approval for consequential operations. Keep prompt changes as one layer rather than the only control.
Add the minimized failure as a regression case and re-run both denied and allowed paths after model, retrieval or tool changes. The automated tool-boundary example supplies a small deterministic starting point. It is not a live model red-team result.
Close a finding when the relevant control demonstrably prevents the recorded failure and the normal workflow still works. Keep remaining uncertainty in the report.
Revision note
Updated October 20, 2026. Removed unsupported security-market and adoption projections. Replaced generic assurances with a scoped defensive runbook, evidence schema and explicit outcome definitions.
Primary sources
Checked October 20, 2026: owasp injection; owasp defense; pyrit; garak.
