October 26, 2026
An AI Tool Isn't a Pentest. It's a Faster Scanner.
An AI Tool Isn't a Pentest. It's a Faster Scanner.
Omair
New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 20+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
A scoping call, near the end. The AI feature is a retrieval assistant over internal documents, and we have spent forty minutes on the application: the authentication, the retrieval boundary, the tool calls, the output handling. Then someone asks the question that changes the quote.
"And the model itself?"
The honest answer is another question. Which model, served how, and what access will we have to it? Because "test the model" describes four or five different engagements with different access requirements, different tooling and different deliverables, and the phrase on its own does not tell you which one anybody is buying.
AI model security testing is adversarial testing of the model layer itself — the weights, the training data, and the inference interface — rather than of the application that calls it. It targets failure modes that exist whether or not the surrounding code is written correctly: extraction, model inversion, membership inference, poisoning and evasion.
That distinction is the whole point. Most work sold as "AI security testing" is application security testing with a model somewhere behind it. That work is necessary and we do a lot of it. It is also not what we are describing here, and conflating the two is how a buyer ends up with a report that never touched the thing they were worried about.
Because the two kinds of testing find different bugs, and only one of them needs anything unusual from you.
Application-layer testing of an AI feature looks much like testing any other application. Can a user reach another tenant's documents through the retrieval layer? Does the agent's tool call carry the caller's authorization or the service's? Is model output handled as untrusted input before it reaches a shell, a browser or a database? Those are the questions behind most of the OWASP Top 10 for LLM Applications 2026, whose first-ranked entry is LLM01:2026 Prompt Injection and whose LLM10:2026 Improper Output Handling is a classic injection problem wearing new clothes. We reach them with a normal engagement and normal access.
Model-layer testing asks whether the model, considered as an artifact, leaks what it was trained on, can be copied, can be made to misclassify on demand, or has been tampered with upstream. Those questions need access the application tester never asks for: query volume, sometimes confidence scores, sometimes the weights, sometimes a view of the training pipeline. Scope that access wrong and the engagement produces nothing.
Five, and the precision matters because three of them are routinely overstated in vendor material.
Model extraction recovers a usable approximation of a model by querying it. Not the model. NIST AI 100-2e2025, the current edition of NIST's adversarial machine learning taxonomy, published March 2025, is blunt about this: exact extraction has been shown to be impossible, and what an attacker obtains is "a functionally equivalent model" that "achieves similar performance at the prediction task." NIST goes further and notes that even obtaining functional equivalence is computationally prohibitive. Be precise about what that does and does not rule out: NIST also describes direct algebraic extraction, which can compute model weights, so weight recovery is not impossible in general. What NIST rules out is exact extraction by querying, and it prices functional equivalence as expensive. "Query the API and walk away with the weights" is the version to distrust.
Model inversion generates inputs characteristic of a training class by exploiting model outputs, usually confidence scores. It does not hand back your training set. NIST again, directly: model inversion "cannot directly reconstruct the training data of the model." It produces class representatives. Faithful record-level reconstruction appears in the literature only under much stronger assumptions — in one case, access to the model parameters.
Membership inference answers a narrower and often more damaging question: was this specific record in the training data? NIST's wording is "to determine whether a particular record or data sample was part of the training dataset." For a model trained on clinical, legal or HR data, that single bit is the privacy finding. It is a distinct attack from inversion — cataloged separately in MITRE ATLAS and in the OWASP Machine Learning Security Top 10, while NIST folds inversion into data reconstruction and gives membership inference its own identifier — and the two are constantly conflated.
Poisoning manipulates the model through the training pipeline rather than the inference interface. NIST separates data poisoning, where the adversary controls part of the training data, from model poisoning, which "operates through model control," and treats backdoor poisoning as a sub-case that "causes a model to perform an adversary-selected behaviour in response to inputs that follow a particular backdoor pattern."
Evasion crafts inputs that cause misclassification at deployment time — NIST's "adversarial example" is "a modified testing sample that induces misclassification or misbehavior of a machine learning model at deployment time."
All five appear in the published catalogs, but not where most people look.
The one top-ten-style list that names them as numbered entries is the OWASP Machine Learning Security Top 10 — ML03:2023 Model Inversion Attack, ML04:2023 Membership Inference Attack, ML05:2023 Model Theft, ML02:2023 Data Poisoning Attack, ML01:2023 Input Manipulation Attack. Worth knowing before you cite it: OWASP's own project page describes the work as in draft and modified frequently, it is classified as a Lab project rather than a flagship, and its content is still the 2023 set. It is the most useful map here and it is not a settled standard. Say so when you use it.
MITRE ATLAS — Adversarial Threat Landscape for AI Systems — carries the same ground as a threat matrix. Its current content release, 2026.09, holds 16 tactics and 120 techniques with 88 sub-techniques. The relevant entries sit under Exfiltration: AML.T0024.000 Infer Training Data Membership, AML.T0024.001 Invert AI Model, AML.T0024.002 Extract AI Model. A note for anyone searching it: no ATLAS technique name contains the word "inversion" — the technique is "Invert AI Model" — and in the 2026.09 release the word appears just once in the whole dataset, inside the description of mitigation AML.M0035. Searching for "inversion" will very nearly return nothing and would wrongly suggest the concept is missing.
One widely repeated claim about ATLAS deserves correcting, because it affects how much you should trust an ATT&CK-shaped reading of it. ATLAS adopts ATT&CK's tactic framing and inherits all but two of its sixteen tactics by explicit reference; the two exceptions are precisely the ones it adds, AI Model Access and AI Attack Adaptation. At technique level the overlap is much thinner: 44 of its 208 techniques and sub-techniques carry an ATT&CK reference, so roughly four-fifths are AI-specific. "ATLAS extends ATT&CK" is true of the tactics and misleading about the techniques.
This is the question that decides whether an engagement is possible, and it has two axes rather than one.
NIST's taxonomy separates attacker knowledge — white-box, black-box and what NIST spells gray-box — from attacker capabilities, which in its predictive-AI section run to six: training data control, model control, testing data control, label limit, source code control and query access. Both axes apply to every attack class, and NIST warns they interact. But they are not interchangeable, and that is the scoping point: knowledge is what the attacker knows, capability is what the attacker controls. Poisoning turns on capability. It needs training-data control or model control, and no amount of query access substitutes for either.
Scoping an engagement on the knowledge axis alone is therefore a mistake, and it is a common one. We have read proposals offering "white-box model testing including data poisoning" where nobody involved had any route into the training data or the model artifact — white-box knowledge granted, the capability poisoning actually needs never mentioned.
ATLAS makes the access question explicit in a way that is useful at scoping time. Its AI Model Access tactic holds four techniques rather than a single ladder — AML.T0040 AI Model Inference API Access, AML.T0041 Physical Environment Access, AML.T0047 AI-Enabled Product or Service, and AML.T0044 Full AI Model Access. The first is query-only, through the interface normal users get. The last ATLAS defines as "full 'white-box' access" with "complete knowledge of the model architecture, its parameters, and class ontology," and notes this lets an adversary work offline "where it is hard to detect their behavior."
In practice, for a commissioned engagement:
Black-box, query-only access supports evasion testing and membership inference — NIST notes most membership inference attacks were developed for exactly this setting — and it supports extraction, bounded by the limits above. It is the access almost every buyer can grant, and it is the access an attacker is most likely to have.
White-box access, meaning the weights, makes evasion testing far stronger and record-level reconstruction plausible where query-only access makes it marginal. It also makes extraction largely moot: with the weights there is nothing left to extract.
Poisoning and backdoors need control rather than knowledge — training-data control, or model control via something like a fine-tuning API — and backdoors need more than that. NIST records that backdoor attacks are the only attacks in the literature requiring both training and testing data control — you need to influence the training set and then deliver the trigger at inference. That is a two-sided requirement and it is the single biggest reason a poisoning test gets cut from a scope.
One deployment fact changes the whole calculation. OWASP's LLM02:2026 entry makes the point plainly: open-weights deployments cannot rely on rate limits, because extraction, membership inference and inversion run offline at unbounded rates. If you ship weights, the query-budget defense is gone.
No, and this is worth being exact about rather than dismissive.
ISO/IEC 42001:2023 — registered title Information technology — Artificial intelligence — Management system, published December 2023 — is a genuine, genuinely certifiable standard, and the requirements for the bodies that audit and certify against it are set in ISO/IEC 42006:2025, which supplements ISO/IEC 17021-1 with AI-specific requirements. A certificate is real evidence.
It is evidence about a management system. It attests that an organization's governance, roles, risk and impact assessment processes, lifecycle controls and documentation conform to the standard's requirements, as judged by an accredited third party. What it does not do is make a claim about any particular model: the scheme certifies the management system, not a model, so a certificate does not say that a given model was tested, that extraction was attempted, or that any specific vulnerability class was assessed.
That is not a criticism of the standard, which does not claim otherwise. It is a warning about how the badge gets read. We keep meeting the assumption that a 42001 certificate in a vendor questionnaire answers the question "has this model been security-tested," and it does not. Alignment with a framework is not the same thing as a control holding under attack.
A different document from an application pentest report, and the difference is mostly about what gets stated as a limit.
A synthetic composite, of the kind we see often enough to describe: a mid-market lender deploys a credit-decisioning model behind an internal API. The provider supplies a model card and a 42001 certificate. Under query-only testing at realistic rate limits, membership inference distinguishes training records from held-out records at a measurable rate on a subset of the population, reported as a true-positive rate at a stated low false-positive rate. That is a measured leak of training-set membership through the API, not a per-applicant determination — and the distinction belongs in the report, because the first is a finding and the second would be a claim the test does not support. No weights were touched. No application vulnerability was involved. The application pentest from the previous quarter was accurate and had no reason to look.
A report worth paying for states which attack classes were in scope and which were excluded, with the exclusion reasoned rather than silent. It states the access granted, because a black-box result and a white-box result are different claims. It gives query budgets and rate limits, because that is the defense being tested. It scopes every finding to the configuration tested, not to the model family. And where a result is negative, it says what was tried and under what budget, because "extraction did not succeed within 50,000 queries at the production rate limit" is a finding and "the model is secure" is not.
Where a claim could not be established, it says so. That is the part most reports skip.
They overlap more than they used to, and the useful distinction is shape rather than subject. Red teaming is goal-driven and scope-wide — ATLAS's own AI Red Team mitigation now covers the complete AI-enabled system including models and data, and names membership inference, model inversion and functional extraction among the things to exercise. Model-layer security testing is the structured subset: named attack classes against the model artifact and its inference interface, with the access granted and the query budget stated, so a negative result means something. If you commission red teaming and expect the second, the difference matters when you decide which one to commission.
Partly. With query-only access through a vendor's API you can attempt evasion, membership inference and extraction, subject to the vendor's rate limits and terms — and the terms are usually the binding constraint, not the technique. Poisoning and backdoor testing are usually out of reach against a third-party hosted model, because you have no route to the training data — with one real exception worth checking before you write it off. NIST lists model control as a capability an attacker "might have ... such as through public fine-tuning APIs or openly accessible model weights." If the provider exposes fine-tuning, a route to model poisoning exists without any pipeline access at all. Establish that in the scope rather than discovering it mid-engagement.
It helps with some of this and not with the rest, and the boundary is worth knowing. NIST states that differential privacy by definition mitigates data reconstruction and membership inference. It also states that differential privacy "does not provide guarantees against model extraction attacks, as this method is designed to protect the training data, not the model," and records negative results against property inference. So it protects the training data, not the model artifact, and treating it as a general model-layer defense is a mistake with a primary-source citation against it.
Some of it. Your provider owns the weights and the training pipeline, so extraction and poisoning sit with them. What remains yours is everything about how your application uses the model — and that is where most real incidents live. Start with the application layer, and scope model-layer work when you host weights or when a regulator asks what you tested.
Enough to be realistic, which is the point of measuring it. The useful test is run at your production rate limit, so the result describes your deployment rather than a laboratory. If a test needs a million queries and your limit allows ten thousand a day, the finding is about the rate limit as much as the model.
ioSENTRIX is a CREST-accredited, ISO/IEC 27001 certified penetration testing firm with a SOC 2 Type 2 attestation. Our AI and ML penetration testing engagements test the model layer and the application around it as separate questions with separate access requirements, and the report says which was which. If you are being asked whether your model has been security-tested and are not sure what would count as an answer, that is a conversation worth having before the next questionnaire arrives.
