September 8, 2026
Compliant Is Not Correct: The AI Gave Us the Wrong Hashcat Mode
Compliant Is Not Correct: The AI Gave Us the Wrong Hashcat Mode
Omair
New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 14+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
Someone asks for the AI inventory. You have one: fourteen systems, each with a named owner, a data classification, and a risk rating.
Then comes the second question. *How do you know that's all of it?*
Most inventories do not survive it, and not because anyone was lazy. The list was built from what people asked permission for. It is a record of requests, not a measurement. A policy tells you what is allowed; only discovery tells you what is running. Those are different claims, and only one is testable.
This post is about the testable one: which signals you collect, how they corroborate each other, where each is blind, and how to rate the result so it survives being pushed on.
Three things changed at once. AI stopped being a purchase — it arrives switched on inside software you already bought, the subject of the AI hiding inside the SaaS you already approved. Inference became reachable by anything that can make an HTTPS request, so a developer with an API key is now an AI deployment. And it stopped being tied to a person at a browser: jobs, service accounts, and agents call models on their own, which breaks any method assuming a human session to observe.
Meanwhile the inventory became load-bearing. The NIST AI Risk Management Framework and the CSA AI Controls Matrix assume you can enumerate your AI systems before their controls mean anything, exactly as inventory has been the first CIS control for as long as there have been controls.
Each has a test you can run and a blind spot worth writing down.
Identity. Export OAuth grants and app assignments from your identity provider, filter to clients first authorized in the last eighteen months, and classify them by scope granted rather than by name. Blind to personal-email signups, free tiers that never requested a grant, and keys minted inside a tenant you already approved.
Network egress. Pull ninety days of DNS and proxy logs, extract first-seen hostnames, and match them against a model-provider list you actively maintain, because that list ages weekly. Blind to AI inside an approved SaaS host, since the traffic goes where it always went; to unmanaged and mobile devices; and to payload under TLS.
Endpoint. Query your endpoint agent for local model runtimes, AI desktop applications, CLI tools, model-provider SDK installs, and MCP server configuration files. The only signal that sees inference happening entirely on a laptop. Blind to mobile and BYOD, and noisy on engineering fleets without ownership metadata.
Tenant admin state. For each product on your SaaS list, query the admin API — or screenshot the console where there is none — for whether AI features are on, who enabled them, and what corpus they retrieve from. The only signal answering "is it on" for embedded AI, and blind to every product not already on that list.
Money. Search expenses, card transactions, and invoices for AI vendors, and read the sub-processor notices vendors send — the one contractual artifact that changes when an approved product starts routing your data to a model provider. Ask procurement for that file, the same discipline as any vendor security assessment. Blind to free tiers and personal expensing.
Code and build. Grep repositories and CI configuration for model-provider SDKs, base URLs, prompt files, and keys, and check your secrets scanner for provider key formats. This is the developer path, reaching a model with no browser, no login, and no grant to catch. Blind to repositories you do not scan.
No signal is authoritative alone, and the method is not to concatenate six lists. Require every entry to carry three facts — an owner, a data path, and a capability class — each from a signal, not an assumption.
Capability class is the one people skip. Summarization is one risk class. A feature that can send a message, change a record, or call another system is another, and it inherits every property we argued matters for an autonomous action worth trusting: genuine, authorized, untampered, reconstructable. The cheapest test separating the classes is whether the thing obeys instructions embedded in content it processes — prompt injection testing against techniques catalogued in MITRE ATLAS, aimed at a ticket body rather than a chat window.
Then read the intersections, and pay most attention to disagreements. One signal is a lead. Two independent signals is a row. A contradiction is usually the best finding in the report: money says four paid seats, identity says nobody has signed in for months, egress says steady traffic all night. That is not a user. That is a key nobody rotated.
*A synthetic composite, assembled from patterns across engagements. Not a client, and not any single organization.*
A 900-person logistics company runs a discovery pass. The approved list has fourteen entries. Identity returns forty AI-related grants. Egress returns thirty distinct AI destinations. Twelve appear in both.
The overlap is the useful number. If the signals were independent and everything equally visible to both, forty times thirty divided by twelve would put the population near a hundred. Neither assumption holds — the signals partly share a cause, and some AI use is invisible to both — so it is not an estimate of the truth. It is a floor on what you are missing, and it makes the unknown discussable rather than rhetorical.
Its two most interesting entries came from tenant admin state and a money-and-identity contradiction.
State confidence at two levels, never as one number.
Per entry, rate the evidence rather than your certainty, using the four attributes that separate a claim from evidence in any AI security assessment: the attempt, the observed result, the artifact, the conditions. An entry is evidenced when you hold an artifact — a log line, an export, a screenshot of the setting. Reported when a person or vendor document said so. Inferred when one signal implied it and nothing confirmed it.
Per inventory, publish four things instead of a completeness claim. *Coverage denominators*: what share of endpoints ran the agent, what share of applications had an admin API queried, how many days of proxy logs existed. *Corroboration rate*: the share of entries confirmed by two or more independent signals — if most came from exactly one, your signals are not corroborating each other and you should assume a large unseen population. *Named blind spots*: personal accounts, unmanaged devices, mobile, on-device inference, every product whose AI setting is not queryable. *A date and an observation window*, because discovery decays, which is why a continuous exposure management rhythm suits this better than an annual sweep.
One further step separates a rating from a guess, and almost nobody runs it: test the discovery itself. In an authorized exercise, have a tester use AI through a path you believe is covered and a path you believe is not, seeding unique synthetic records — the canary technique — then check whether your pipeline flagged either. Detection you assumed is not detection you measured.
None of this yields a complete inventory. No discovery method is complete, ours included. It yields one whose gaps are named and sized — a different object to defend in front of an auditor or a board, and what ISO/IEC 42001 and the OWASP Top 10 for LLM Applications quietly assume. Alignment is not certification: ISO/IEC 42001 is certifiable, but it certifies a management system, not that your inventory found everything.
Three questions, in scoping-call order.
1. Which of the six signals did your last inventory use, and how many entries came from exactly one of them?
2. What are your coverage denominators — the share of endpoints, applications, and days genuinely observed rather than assumed?
3. When did you last test the discovery itself, by running a known AI interaction through a path you believe is covered and checking whether anything caught it?
"We would have to go look" is a normal answer, and a better one than false confidence. An AI risk register built on an unmeasured inventory inherits every gap in it silently.
ioSENTRIX is a CREST-accredited offensive security firm. We assure the stack you already chose rather than selling one that competes with it. For AI inventory discovery that means collecting across all six signals, corroborating entries rather than concatenating lists, hands-on validation of what each discovered system can retrieve and do under real credentials, and a rated inventory stating its coverage and blind spots in writing. What comes back is evidence — the attempt, the observed result, the artifact, the conditions — not a spreadsheet of assertions. Independent adversarial testing produces that evidence whoever performs it, because the tester has no stake in a reassuring answer.
To see what a rated discovery pass turns up in your environment, start a conversation with us. Estates that keep changing shape usually find a continuous testing model fits better than a snapshot.
