New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 20+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
Someone in your organization can almost certainly name the guardrail protecting your LLM feature. It has a product name, a dashboard, and a line in the architecture diagram.
Here is the harder question. When was the last time somebody tried to get past it, on purpose, with the configuration in front of them?
If the answer is "we tested it before launch," you know what the guardrail did against the attacks you thought of in September. You do not know what it does against the attacks an adversary thinks of in November, and the gap between those two things is the entire subject of this post.
An LLM guardrail is a control that inspects what goes into or comes out of a language model and blocks, rewrites, or flags content that violates a policy. It sits beside the model rather than inside it. It is a filter, a classifier, or a policy engine — not a property of the model's weights.
That distinction matters, because it tells you what a guardrail can and cannot promise. A guardrail can reduce how often an unwanted output reaches a user. It cannot make the model incapable of producing that output, and it cannot make itself immune to the same class of manipulation it exists to catch.
This is not an ioSENTRIX opinion. It is the stated position of NIST, MITRE and OWASP.
Because the model has no architectural way to tell an instruction from data, and a guardrail bolted alongside it inherits that problem rather than solving it.
OWASP puts it plainly in the 2026 edition of its LLM Top 10. One practical note before you check: OWASP's own landing page still serves the 2025 list, so the quotations below are from the canonical 2026 source rather than the page a search will land you on. Under its prompt injection entry, the project writes that "LLMs make no architectural distinction between instructions and data, and their behavior is stochastic, so no reliable prevention mechanism exists today," and concludes that "defense is therefore architectural rather than interceptive." The same entry is specific about filters: "Semantic filters are evadable by rephrasing or encoding, and low-resource and code-mixed inputs degrade their accuracy."
NIST reaches the same conclusion from a different direction. In Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025, March 2025), NIST notes that detection systems guarding a model "are also vulnerable to attacks and may have correlated failures to the main models that they are monitoring." Correlated failure is the uncomfortable part: the guardrail and the model can be fooled by the same input, because they are frequently the same kind of thing.
MITRE encodes the attack as a technique rather than a theory, in the matrix we use in our AI red teaming work. In the ATLAS knowledge base, AML.T0068 LLM Prompt Obfuscation describes adversaries hiding prompt injections "to avoid detection from humans, large language model (LLM) guardrails, or other detection mechanisms," and states that this "may bypass LLM guardrails that identify malicious content." ATLAS also carries guardrails as a mitigation, AML.M0020 Generative AI Guardrails — and the mitigation's own text instructs that "guardrails should be continuously evaluated, red-teamed, and updated as adversarial techniques evolve."
So the framework position is not that guardrails are useless. It is that a guardrail is a control whose effectiveness decays and which therefore has to be re-tested, by someone trying to beat it, on a schedule. Alignment with any of these frameworks is not the same as compliance with anything, and none of them certifies that your particular guardrail held.
This is the part most buyers never read, and it is the most useful documentation in the category. The serious vendors document their own limits, and they deserve credit for it. Four examples, all from official product documentation.
NVIDIA's NeMo Guardrails documentation states that "jailbreak detection fails open" — when a configured detector cannot be reached, "the rail allows the request instead of blocking it," and "a detector outage removes this protection without a visible signal." The same page notes that its heuristics carry a usage note that they are "intended only for English language evaluation and will yield significantly more false positives on non-English text, including code." NVIDIA's runtime security FAQ adds that "no single rail guarantees protection against every adversarial prompt," and that "there is no deployment-independent precision, recall, or false-positive rate."
Meta's Llama Guard 4 model card lists, under limitations, that "as an LLM, Llama Guard 4 may be susceptible to adversarial attacks or prompt injection attacks that could bypass or alter its intended use." The companion Llama Prompt Guard 2 card names the exact failure mode under its Limitations heading, as Vulnerability to Adaptive Attacks: "While Prompt Guard enhances model security, adversaries may develop sophisticated attacks specifically to bypass detection."
Amazon's Bedrock Guardrails documentation is explicit about a coverage boundary that matters enormously for agents: content filters "do not evaluate tool results (`toolResult`), tool definitions (`toolSpec`), or model-generated tool call arguments (`toolUse.input`)." If your agent's risk lives in the tool chain — and in an agentic system, it does — the filter is not looking there.
Microsoft's Azure AI Content Safety documentation describes the streaming trade-off directly: with asynchronous filtering, "no content is buffered, which allows for a fast streaming experience with zero latency associated with content filtering," but "unsafe content might be briefly exposed before filtering is complete."
None of that is a reason to remove a guardrail. Every one of those four products reduces risk. It is a reason to stop treating the product's presence as the answer to "is this control effective," because the vendors are not claiming that, and the documentation says so in their own words.
It finds things, and the published research is specific about how much. The caveat worth stating up front: the figures below are per-system and per-technique from named academic benchmarks, not an industry average. Anyone quoting you a single industry-wide "X% of guardrails fail" number has invented it; the published figures are always tied to a named set of defenses and a named set of techniques.
An empirical study of evasion attacks against prompt injection and jailbreak detection systems (arXiv:2504.11168, revised July 2025) measured attack success as evasion of the classifier while preserving the adversarial payload, across six detection systems. Using character-injection techniques, emoji smuggling achieved a 100% attack success rate against the tested detectors for both prompt injections and jailbreaks. Unicode tag smuggling and bidirectional text averaged between the high 70s and high 90s across the six detectors, depending on the technique and on whether the payload was an injection or a jailbreak; individual detectors ranged from 0% to 100%. One hardened detector in the set — a prompt-injection model that is not trained to detect jailbreaks, and so was tested on injections only — averaged roughly 20% across the character-injection techniques. That average is misleading on its own: it held nine of the eleven techniques below 20%, most of them near zero, and was bypassed only by the two smuggling variants, which both reached 100%. The authors note their own limitation: only one of the six systems was tested as a true black box, which "limits the generalizability of our findings."
A second study is more important methodologically. The Attacker Moves Second (arXiv:2510.09023, October 2025) applied adaptive attacks — gradient, reinforcement learning, search-based, and human red-teaming — against twelve recent defenses. Its abstract reports bypassing those twelve "with attack success rate above 90% for most," and notes that "the majority of defenses originally reported near-zero attack success rates." On stacking detectors, the paper is blunt: "simply adding more filters or stacking additional detectors does not resolve the underlying robustness problem." And on method: "Human red-teaming is still effective! We find that human red-teaming succeeds in all the cases we evaluated, even when the participants have only limited information and feedback from the target system."
OWASP cites that second paper in the current LLM Top 10, where its prompt injection mitigation guidance tells readers to "test against adaptive attackers who have read the deployed defense, and reject static-only attack-success claims."
That phrase — reject static-only attack-success claims — is the whole brief for a guardrail test, written by OWASP rather than by us.
Here is the method, with what each step proves. The point of naming the proof is that a test which does not change what you believe is not worth running.
1. Establish what the guardrail is supposed to refuse, in writing. Not "block harmful content" — the actual policy, per category, with examples of a pass and a fail. Proves: that a failure is identifiable. Without this, every result is arguable, and arguable results never get fixed.
2. Baseline with published benchmarks, disclosed. Run an established suite against the deployed configuration — OWASP's own mitigation guidance names AgentDojo and JailbreakBench as the starting point. Proves: the floor. A guardrail that fails a public benchmark will not survive anything harder, and this step is cheap enough to run in CI.
3. Re-run with the defense specification in the tester's hands. Give the tester the policy, the detector in use, the thresholds, and the prompt template. Proves: resistance to an informed adversary rather than a naive one. This is the step the research above says changes the answer, and the step most programs skip because it feels like cheating. An attacker who has read your vendor's public documentation has exactly this information.
4. Attack the encoding, not just the semantics. Homoglyphs, zero-width characters, bidirectional text, emoji and Unicode tag smuggling, full-width characters, diacritics, base64 and other encodings, and the same payloads in a second language. Proves: whether the control is matching meaning or matching strings. This is where the published success rates are highest, and it is mechanical to test.
5. Test the injection path you do not control. Indirect injection through retrieved documents, tool output, web content, and uploaded files — if retrieval is involved, start from RAG security, and if the tools come through a model context protocol server, from MCP security. ATLAS separates these deliberately: AML.T0051 LLM Prompt Injection has sub-techniques AML.T0051.000 Direct, AML.T0051.001 Indirect, and AML.T0051.002 Triggered. Proves: whether the guardrail inspects the content the model actually consumes, or only what the user typed. Read your vendor's coverage boundaries first — as the Bedrock example above shows, tool results may be out of scope by design.
6. Break the guardrail itself and watch what happens. Make the detector unreachable, slow, or rate-limited. Proves: fail-open versus fail-closed. NVIDIA documents its behavior here; most teams have never observed their own. A control that silently stops existing during an outage is a control you cannot rely on during an incident.
7. Check what the decision leaves behind. Does a block produce a log line with the input, the matched policy, the score, and the decision? Proves: whether you could reconstruct the event afterwards. ATLAS's guardrails mitigation asks for exactly this: "Guardrail decisions should be logged … and observed failures should be systematically incorporated into updated policies, evaluation datasets, detection logic, prompts, and Generative AI Model Alignment."
8. Test whether the rules themselves are recoverable. Attempt system-prompt extraction directly; probe the refusal messages differentially to infer which conditions trigger them; and read your tool descriptions and error strings for leaked policy text. If an attacker learns the rules, the rules become a map. OWASP's hidden-context entry in the current LLM Top 10 makes the point well: leakage "reveals the underlying triggers, conditions, and exceptions that led to that decision," which "allows attackers to craft inputs that intentionally avoid known refusal patterns or exploit gaps in enforcement." Proves: whether your system prompt and policy text are recoverable, which determines how much step 3 is worth.
9. Re-run after every model, template, or threshold change. Proves: that the result still holds. A guardrail test has a shelf life measured in deployments, not quarters.
Not a better product. A different relationship to the product.
Treat the guardrail as a control with a measured effectiveness rate that you own, rather than a feature with a vendor's name on it. Write down what it refuses. Measure how often an informed tester gets past it. Record the number. Change something. Measure again. Keep the series.
That series is the evidence. A guardrail with a dated bypass rate against a disclosed-configuration test, and a trend, is a control you can speak about honestly to an auditor, a customer, or a board. A guardrail with a product name and a green dashboard is a procurement decision, and it answers a question nobody asked.
The honest summary of the research is that detection-based guardrails are a useful, limited layer. The second paper above says so itself: detectors "can still provide practical value by blocking some unsophisticated or opportunistic attacks, making them a useful — but limited — component of a broader defense strategy." Useful and limited is a perfectly good thing for a control to be. It is only a problem when the architecture behind it was designed as though the guardrail were neither — the same mistake we described in compliant is not correct.
Which is why the architectural half matters more than the filter. If a successful injection can only make your agent do things that are individually authorized, logged, and reversible, the guardrail's bypass rate stops being existential. OWASP's conclusion that defense "is therefore architectural rather than interceptive" is the design instruction, and it is why the agentic questions worth asking are mostly about authorization, and testing the interceptive layer is how you find out how much the architecture has to carry.
Alignment is training that shapes what the model tends to produce. A guardrail is a separate control that inspects inputs or outputs at runtime and enforces a policy. Alignment can be elicited around; a guardrail can be evaded. They fail in different ways, which is why testing one tells you little about the other.
No, and the frameworks separate them. NIST AI 100-2 E2025 defines a jailbreak as "a direct prompting attack intended to circumvent restrictions placed on model outputs, such as circumventing refusal behavior to enable misuse." Prompt injection is the broader class of supplying input that the model treats as instruction; our prompt injection explainer covers the attack in detail. MITRE ATLAS tracks them as distinct techniques, AML.T0054 and AML.T0051.
It will block more of the unsophisticated traffic, which has real value. It will not fix the underlying robustness problem — the adaptive-attack research found that its attacks "continue to bypass both the target model and every detector we evaluated, even when multiple guardrails are combined."
After any change to the model version, the system prompt, the policy, the detector, or its thresholds — and on a routine cadence between changes, because the attack techniques move even when your system does not.
No. It means one control behaved a certain way against a stated set of attacks on a stated date. The LLM feature still sits on an application, an identity model, and a tool chain, each of which needs its own testing — see AI data leakage testing and AI penetration testing services.
ioSENTRIX is a CREST-accredited, ISO/IEC 27001 certified penetration testing firm. Our AI and ML penetration testing engagements test guardrails the way an adversary would — with the configuration disclosed, the encoding layer in scope, the indirect injection paths included, and a dated result you can show someone. We assure the stack you have chosen; we do not sell a competing one, which is what lets us report the number honestly whichever way it comes out. If you have shipped an LLM feature and nobody has tried to get past its controls since launch, that is the engagement.
