October 23, 2026
What AI Evidence Does a SOC 2 Auditor Want?
What AI Evidence Does a SOC 2 Auditor Want?
Omair
New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 14+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
Ask a vendor whether AI can replace your penetration test and you will get a pitch. Ask the engineer who would otherwise do the work and you get a more useful answer: it depends entirely on which half of the job you mean.
Here is a question worth sitting with before you decide. Your application lets a customer apply a discount code at checkout. Can that customer apply it twice by replaying the request, or apply someone else's code by changing an ID in the URL? Now ask which tool you own that would tell you.
Nothing in that question is exotic. It is the kind of flaw that empties a balance sheet rather than filling a report, and scanners do not find it on their own — because finding it requires knowing what a discount code is for.
This argument is older than AI, and the industry has already lived through it once.
The choice in front of a buyer used to be: run Burp Suite yourself, or hire a firm that would also run Burp Suite. Anyone could download it. If the scanner were the thing you were paying for, the penetration testing industry would have had no reason to exist past the week that tool became widely available.
It is still here, and our view is that the scanner was never the differentiator — the person driving it was. The firm you hired brought judgment about which of a scanner's hundreds of findings mattered, the patience to chain three of them into something that did, and the context to know that an exposed admin endpoint on a staging host is a shrug while the same endpoint on a payments service is a Friday night.
AI has changed the speed of the first half of that work, dramatically. Our argument is that it has not changed which half was hard.
Vague AI skepticism is as useless as vague AI enthusiasm, so be precise here.
Autonomous tooling is genuinely strong at breadth. It enumerates attack surface faster than a person can, and recognizes known vulnerability patterns across an API surface at a scale no human would attempt by hand. It does not get bored on hour six of the same parameter across hundreds of endpoints, which is exactly where human attention fails. Writing in May 2024 — before most of today's agentic tooling existed — NCC Group's Steven van der Baan put the data-handling half plainly: "AI excels in handling immense data volumes while recognizing patterns."
And autonomous tools find real, serious bugs. That is not in dispute and it is worth stating clearly. One autonomous testing vendor has submitted 1,060 vulnerability reports to public bug bounty programs over roughly two years. In one ninety-day window, program owners classified its submissions as 54 critical, 242 high, 524 medium and 65 low severity. Systems like it have at times outranked human researchers on public leaderboards. Anyone telling you these tools only produce noise has not looked at the output.
So we use them. On every engagement. Handing routine coverage to a machine is how our engineers spend their days on the part that needs a person, and it is why AI-driven penetration testing is now part of how modern testing gets done rather than a separate product category.
The same vendor's status breakdown is worth reading in full. Of 1,044 categorized reports: 130 resolved, 303 triaged, 208 duplicates, 209 informative, and the rest new or pending.
Be careful what you conclude from that. A duplicate on a bug bounty platform means the finding was real and somebody else got there first; an informative usually means out of scope or accepted risk. Neither is a false negative. What the shape shows is a system doing what breadth does well — covering well-trodden ground, fast and repeatedly.
The flaws that put companies into a breach notification tend to sit off that ground: business logic that can be abused, authorization that can be sidestepped, and unremarkable weaknesses chained into real impact.
OWASP's own data makes the point better than we could. In the OWASP Top 10:2025, "A01:2025 - Broken Access Control maintains its position at #1 as the most serious application security risk; the contributed data indicates that on average, 3.73% of applications tested had one or more of the 40 Common Weakness Enumerations (CWEs) in this category." And in the same document, OWASP states a limitation of its own dataset that every buyer evaluating an AI tool should read twice: "The results in the data are largely limited to what the industry can test for in an automated fashion."
Read those two sentences together. The category OWASP ranks as the most serious application security risk is one whose industry measurement is constrained by what automation can see.
The reason is not that the models are not clever enough yet. It is that an authorization flaw is only a flaw relative to an intention. Changing a customer ID in a URL and getting a 200 response is a bug only if that customer was never meant to be readable — and nothing in the request or the response says so. The code says it only where someone wrote the rule down, which is precisely the case where the rule tends to work. NCC Group's May 2024 framing still describes the gap, even allowing for its age: AI "typically lacks the contextual understanding that human testers possess," and "Certain categories of vulnerabilities, such as logical flaws or business logic errors, often require human intervention to identify accurately."
Buyer behavior has moved in the same direction. In a 2026 industry survey of more than 450 security professionals, the share of organizations relying entirely on AI automation for their testing fell from 29% to 9% in a single year, while 47% said they now prefer a hybrid model. In the same survey, 78% of security teams reported hitting critical false negatives from purely automated scanning — not false alarms, which are merely expensive, but real vulnerabilities reported as absent.
The following is illustrative, assembled from patterns we see rather than from any single client engagement.
A logistics platform runs a scan before an annual audit. It comes back clean. An engineer then spends two days on the quoting workflow and finds that a freight quote can be advanced from draft to approved by replaying the approval request with a different quote ID — one belonging to another tenant. Each piece is unremarkable on its own: the quote ID is sequential, the approval endpoint does not re-check tenancy, and the status machine accepts an out-of-order transition. Chained, a customer can approve a competitor's freight pricing at a figure of their choosing.
No finding in that chain would have been rated above medium. The scan reported none of them, because every request it sent came back with a valid response.
| Autonomous tooling | How a tester settles it | |
|---|---|---|
| Known vulnerability patterns | Strong — fast, broad, tireless | Not where the value is |
| Attack surface enumeration | Strong — faster than any person | Scoping decisions only |
| Misconfiguration, outdated components | Strong | Confirming real reachability |
| Business logic abuse | Weak — needs to know intent | Map the intended state machine, then replay steps out of order and with skipped preconditions |
| Broken authorization, privilege escalation | Weak — a 200 response looks like success | Build a role-by-object permission matrix; replay each role's requests against every other role's objects |
| Authentication weaknesses | Partial | Test reset, session fixation and MFA-bypass paths by hand against the documented flow |
| Chaining low findings into real impact | Weak — scores each finding alone | Write the attack narrative end to end and execute it as one path |
| Proof of exploitability | Partial — a reproducible request, not its business cost | Attach the request, the response, the artifact and the conditions |
In our engagements, manual testing finds roughly 20% more vulnerabilities than scanning alone, mostly in authentication, authorization and privilege escalation.
A tool produces a list. A penetration test produces an argument: this is reachable, this is exploitable, this is what it costs you, and this is the line of code that fixes it.
That difference is procedural, not mystical. Our engineers are OSCP-certified and our testing is CREST-accredited, which means the methodology is externally examined rather than self-declared. Every finding we hand back carries a proof of concept — the request, the response, the artifact, the conditions — and a code-level fix, because a finding a developer cannot reproduce is a finding that does not get fixed. We run it as continuous PTaaS where the application changes faster than an annual cycle, and under rules of engagement agreed before anyone touches anything.
None of that is a claim only we can make. Independent adversarial testing is the position that produces this kind of evidence, whoever performs it, and plenty of good firms do it well. What we would push back on is the idea that the evidence is optional — or that a tool with no model of what your application is for can tell you what a finding is worth to your business.
For a cheap self-check before you talk to anyone, take your last automated scan and pick the three findings you would least like to be wrong about. For each one, see whether you have the attempt, the observed result, the artifact, and the conditions. All four is evidence. A severity rating and a CWE number is a claim — a normal place to be, and a reasonable place to stop being satisfied.
AI makes us faster. Skill is what makes it a pentest.
We are a boutique, CREST-accredited, ISO/IEC 27001 certified offensive security firm, and we use AI on every engagement — for coverage, not for judgment. Our penetration testing practice puts OSCP-certified engineers on the work in the table above: permission matrices replayed role against role, state machines driven out of order, chains executed end to end rather than scored finding by finding. Our AI/ML and LLM penetration testing does the same for the AI systems you are shipping now. Every finding arrives with a proof of concept and a code-level fix.
If you are weighing an AI tool against a scoped engagement, tell us what you want tested and we will come back with a scope and a price in writing, in one business day.
No, though it can replace part of the work inside one. AI is strong at breadth — enumerating attack surface and matching known vulnerability patterns at a speed no person can match. It is weak at business logic abuse, broken authorization and chaining several low-severity findings into real impact, because those require a model of what an application is intended to do. In a 2026 survey of more than 450 security professionals, the share of organizations relying entirely on AI automation for testing fell from 29% to 9% in a year, while 47% said they prefer a hybrid of automated coverage and human testing.
A tool returns a list of findings matched against known patterns. A manual penetration test returns proof: an engineer exploits and chains weaknesses the way an attacker would, establishes the business impact, and hands back a reproducible proof of concept with a code-level fix. The tool establishes whether a pattern appears. The test establishes whether someone can actually do something to you with it.
Mainly business logic flaws, broken access control and privilege escalation, and attack chains assembled from individually low-severity issues. OWASP notes that its own Top 10 data is "largely limited to what the industry can test for in an automated fashion," while Broken Access Control is ranked the number one category in the OWASP Top 10:2025. In our engagements, manual testing finds roughly 20% more vulnerabilities than scanning alone, mostly in authentication, authorization and privilege escalation.
Yes, and that is the more dangerous error. In a 2026 survey of more than 450 security professionals, 78% of security teams reported experiencing critical false negatives from purely automated scanning — real vulnerabilities reported as absent. A false positive costs an engineer review time. A false negative sends you to an audit, or to a customer, believing something was tested when it was not.
Yes, on every engagement. We use automation and AI for coverage, enumeration and routine pattern matching, which frees our OSCP-certified engineers to spend their time on business logic, authorization and exploit chaining. We do not use it to decide what matters, or to replace the proof of concept behind a finding.
It tends to change where the hours go rather than removing them. Automation compresses the coverage phase, which can shorten a timeline, but the specialist days spent proving exploitability are the cost in any serious engagement. A quote that is dramatically cheaper is usually buying fewer tester-days rather than more efficiency, which is worth asking about directly.
