What Does It Mean to Security-Test a Model, Not the App Around It?

A crosswalk is not evidence. What artifact actually satisfies each NIST AI RMF function, which subcategories a penetration test answers, and why aligned, certified and tested are three different states.

Policy says inventory your AI, but few define the artifact. What the G7 minimum elements actually require, which fields you can re-derive and which are only prose, and five ways to validate an AI-BOM instead of filing it.

There is no shortage of AI vendor question lists. What is scarce is a view of which answers break when somebody tests them. Six claims that survive a questionnaire and not an engagement, the test that settles each, and what to ask instead.

An MCP server is an API with an unusually trusting client and an unusually powerful backend. Testing it means exposure and transport, three authorization attempts against a stated MUST NOT, the advertised tool surface, and what the server sends back — almost all of it without a model in the loop.

Forty prompt-injection payloads, none successful, risk low — and six weeks later the agent refunds the same customer twice. The unit of an agent test is the action, not the answer. Four phases, five channels, and why findings carry a rate instead of a checkmark.

“Test the customer-facing AI assistant” is one sentence, and the whole engagement will be argued from it. The seven surfaces an AI penetration test scope must name, what belongs in the rules of engagement, and what a finding must carry.

Two firms quote the same AI system and the numbers differ fourfold. Both are honest. The seven scope decisions that set the price of an AI penetration test, and the three questions that separate a scoped proposal from a priced guess.

Enterprise buyers have moved from attestation to evidence on AI. The premium does not show up as price — it shows up as sales cycle length, contract terms, and the deals you are never invited into. Four decisions for a board.

Your agent telemetry is better than it has ever been, and it still cannot certify that your controls held under attack. Three limits of the signal layer, each with a test attached.

Three proposals, all called an AI security assessment, priced five-fold apart. What actually drives the number, the four conditions that justify adversarial testing, what each deliverable contains, and the five questions to ask before you sign.

Every workflow you are automating had a person doing integrity work nobody logged as a control. Here is what has to replace it — and how to test that it holds.

Most AI risk registers collapse “untested” and “tested and failing” into the same amber, which inverts the priority a defender should have. Why a proven failure is a bounded, demonstrated path, why an unknown still deserves respect, and how a state field fixes the ranking.