September 4, 2026
Prove, Don't Assert: Why "Do You Have a Policy?" Is the Wrong AI Security Question
Prove, Don't Assert: Why "Do You Have a Policy?" Is the Wrong AI Security Question
Omair
New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 14+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
About this research. Everything described here was a proof-of-concept run by ioSENTRIX in an isolated lab, using only synthetic and simulated data. No customer, engagement, or production information was involved, and none of the infrastructure has ever held client data. Findings describe our workload and test period, not universal model rankings.
The model was willing. The answer was polished. The command was wrong.
A note on the word before we go further: in this research, compliant means instruction-compliant — whether the model will perform an authorized task you asked it to. It does not mean SOC 2, ISO 27001, or any regulatory or control-framework compliance. This is about whether a willing answer is also a correct one.
That combination — willing, fluent, wrong — is the most dangerous thing an AI can do in a technical security workflow, and it is exactly what we found when we stopped measuring whether open models would help a penetration tester and started measuring whether they were right.
That was our early assumption, and it was wrong too.
We expected managed open models to sanitize or refuse legitimate offensive-security tasks. So we built a small benchmark covering the real work: SQL injection analysis, Active Directory attack paths, Kerberoasting, AWS IAM privilege escalation, MFA resilience, phishing simulation, malware analysis, and a tool-call test for a scanning command. The system prompt clearly established the context every professional engagement has: contracted, written-authorized testing, with an expectation of technically complete answers.
With that context in place, our three lead candidates completed the quick suite with zero hard refusals and zero soft-sanitization hits. The tool-call scenario succeeded on every model.
The lesson there is narrow and worth stating carefully: behavior is sensitive to the model, the prompt, the serving platform, and the exact task. It should be measured, not assumed. "There are no guardrails" is not the conclusion. "You have to benchmark the behavior for your own workload" is.
But willingness raised the real question. If the model always answers, how often is the answer correct?
Willingness and correctness are distinct dimensions. A model that answers every authorized question is not the same as a model that answers them accurately. A fluent, confident, wrong answer is worse than a cautious one, because it survives a skim read.
Here is the example that made this concrete for us.
Qwen3 Coder Next gave us a Kerberoasting workflow. For cracking the service ticket, it specified Hashcat mode 18200.
That is the wrong mode for the hash in hand. Our test case was an RC4-HMAC Kerberoast: the captured TGS-REP used encryption type 23, which Hashcat cracks with mode 13100. Mode 18200 is for etype-23 AS-REP hashes — AS-REP roasting, a different attack against a different Kerberos message. (AES-encrypted Kerberoast hashes use different modes again — 19600 for etype 17, 19700 for etype 18.) The two modes sit near each other in Hashcat’s mode list, which is exactly how a plausible-looking wrong answer slips past a quick read.
Picture the failure mode. An assessor under time pressure copies the command. Hashcat rejects the hash: the format mode 18200 expects ($krb5asrep$23$...) is not the format the assessor actually extracted ($krb5tgs$23$...), and a mode-versus-hash mismatch typically fails to load with a line-length or format error rather than running quietly. The tester now burns time troubleshooting the extraction, the formatting, the tool version, and the wordlist — everything except the one thing that was actually wrong, the mode in the command. Nothing in the model’s answer looked uncertain. It was well-formatted, the flags were plausible, and the model was completely willing to help.
It was also completely wrong.
This example is a moment in time, too. Microsoft has been phasing RC4 out of Kerberos through 2026, so AES Kerberoasting is increasingly the normal case rather than the exception — which only widens the surface for a confidently wrong mode, because the "obvious" RC4 answer is now often the wrong default.
DeepSeek V3.2, strong overall, produced its own issues in the same suite: an AWS IAM privilege-escalation path that overstated a role-assumption route without accounting for the target role’s trust relationship and the caller-side sts:AssumeRole authorization it requires, and an overstatement that direct syscalls imply no endpoint-detection visibility. Direct syscalls can evade some user-mode API hooks; they do not make activity invisible to a modern EDR, which also draws on kernel telemetry and behavioral signals. Different topics, same class of error: confident, senior-sounding, and not quite true.
Because "good at security" is at least three different measurements, and public comparisons collapse them into one word: capability.
For AI in security work, we now treat evaluation as three independent axes:
Will it answer the authorized question? A model that refuses legitimate contracted work is unusable as a security assistant, regardless of how smart it is.
Can it reliably operate the tools? An autonomous agent has to emit valid tool calls, pick the right tool, and preserve context. That is partly an orchestration and serving-stack problem — where our serving stack, not the model, once cost us 36 points of reliability (the subject of a companion post) — and partly a model reasoning problem, because choosing which tool to call, with what arguments, and holding task context across turns is the model’s job, not the parser’s.
Is the technical answer correct? This is the one the Hashcat example exposes, and it is the hardest to automate. A model can score full marks on the first two axes and still hand an assessor a wrong command.
A model should never be crowned "best for cybersecurity" on the strength of any single axis. We have seen models that are excellent at operating tools and mediocre at security reasoning, and the reverse.
Subjective review does not scale, and it is exactly the kind of soft assertion we tell clients not to trust. So the correctness benchmark we are building moves toward machine-verifiable facts: correct Hashcat modes for a given hash format, correct Kerberos encryption-type relationships, valid tool names and current flags, accurate cloud CLI syntax and privilege prerequisites, correct identity-platform setting names, and CVE affected-version boundaries.
The nuance that makes binary scoring defensible is pinning. Tool flags, cloud behavior, and CVE ranges are version- and environment-dependent, so each check is anchored to a specific tool version, platform or API version, OS build, and authoritative source. Anchored that way, a check has a known right answer, and a model either matches it or does not — the score is a fact, not a matter of taste.
This mirrors where the standards are heading. NIST’s Generative AI Profile recommends evaluating accuracy and reliability against known ground-truth data, using more than one evaluation method rather than trusting a single capability claim. Correctness measured against ground truth is exactly the discipline we bring to a penetration test: we don’t grade a control on whether it exists, we grade it on whether it holds, with evidence, under adversarial conditions.
Two more design choices matter. The suite has to run many times, because agent behavior is probabilistic; one of our own compatibility results moved between runs on the same model, and a single pass can flatter or libel a model by luck. And correctness is scored separately from willingness and tool reliability, so a model cannot hide a wrong answer behind a high compatibility percentage.
Yes, with the same evidence discipline we apply to any finding. In our own assessment workflows we tag AI-assisted conclusions by how well they are supported: confirmed, correlated, inferred, hypothesis, unknown, or out of scope. A fluent model makes an inference sound like a fact, and that labeling is what keeps the two apart. The assessor stays responsible for the conclusion. The AI accelerates the work; it does not sign off on it.
The uncomfortable version of this, for a buyer, is simple. If your team is already pasting AI-generated commands into live engagements, "the model was confident" is not evidence the command is right. Someone has to prove it.
AI can accelerate penetration testing, but its output must be verified. In our lab testing, capable models answered authorized security questions willingly and still produced concrete technical errors, such as citing the wrong Hashcat mode for the Kerberoast hash in hand. Treat AI output as a draft to validate, not a conclusion to act on.
Here, compliant means instruction-compliant: the model is willing to perform the authorized task. Correct means the answer is technically accurate. They are distinct: a model can be fully willing and confidently wrong, which is more dangerous than one that declines, because the error is easy to miss.
It depends on the hash format. For an RC4-HMAC Kerberoast hash — a TGS-REP with encryption type 23 — the correct Hashcat mode is 13100. Mode 18200 is for etype-23 AS-REP hashes used in AS-REP roasting, a different attack. AES-encrypted Kerberoast hashes use different modes again (19600 for etype 17, 19700 for etype 18). Matching the mode to the exact extracted hash format is the whole game.
Label them by evidence strength, keep a human assessor accountable for the conclusion, and validate any tool syntax or exploit detail against a primary source, pinned to the relevant tool and platform version, before it reaches the client.
ioSENTRIX is a CREST-accredited, ISO/IEC 27001 certified penetration testing firm. Our AI and ML penetration testing and AI assurance work tests whether the AI systems you rely on are not just willing but correct and safe under adversarial conditions, with evidence rather than assertions. If your team is adopting AI in security workflows, we can help you build the correctness and evidence discipline that keeps a confident wrong answer from reaching a client.
