Flat vector illustration of a governance document beside a checkpoint barrier with an orange stop button, and a dashed orange path routing around the barrier
TABLE Of CONTENTS

Agentic AI Governance Is Being Written for Systems Nobody Has Tested Yet

Omair
2026-11-18
25
min read

Here is a question to put to whoever owns agentic AI governance at your company: your agent requires human approval before it can take a high-impact action. What happens when the approval service times out?

‍

Most people answer with the design intent. The agent waits. The action is blocked. Someone gets paged.

‍

The useful follow-up is shorter. How do you know?

‍

That question has no good answer in any published governance framework, and the reason is worth understanding, because a great deal of effort is currently going into documenting oversight that nobody has specified how to test.

‍

What is the agentic AI governance gap?

The agentic AI governance gap is the distance between a documented human-in-the-loop control and a checkpoint an agent demonstrably cannot route around. The frameworks arriving now are good at describing the first. None of them tells you how to establish the second.

‍

This is not a complaint that governance is useless. It is a specific, checkable claim about what the current documents require, and it holds across every document that imposes a requirement. The two that come closest to asking for a test are voluntary or unpublished, and both are set out below.

‍

Does the EU AI Act require human oversight of agents?

It requires human oversight of high-risk AI systems. It says almost nothing about agents, and the article itself prescribes no way of checking that the oversight works — though, as the counter-argument section concedes, the Act does reach testing by reference.

‍

Article 14 of Regulation (EU) 2024/1689 is the human oversight provision, and it is worth reading for what it actually obliges. Paragraph 1 requires high-risk systems to be "designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use." Paragraph 4 then requires that the system be provided to the deployer so that the assigned persons "are enabled, as appropriate and proportionate" to do five things — understand the system's capacities, remain aware of automation bias, interpret the output, decide to disregard or override it, and "intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state."

‍

Three things about that are easy to miss.

‍

It is an obligation to enable, qualified twice. The duty is that persons "are enabled, as appropriate and proportionate." It is not a duty that an action be blocked. A system that offers a stop button nobody has ever pressed satisfies the text.

‍

The provider can discharge it on paper. Article 14(3) says oversight shall be ensured through "either one or both" of measures "built, when technically feasible, into the high-risk AI system by the provider," or measures "identified by the provider" that are "appropriate to be implemented by the deployer." The second branch is a documentation route. Nothing requires an in-product enforcement point.

‍

The word "test" does not appear in Article 14. No metric, no threshold, no verification method, no acceptance criterion. Nor does it appear in Article 26, the deployer article. It does appear twice in Article 13, on information the provider must supply in the instructions for use — and both occurrences attach to the accuracy, robustness and cybersecurity metrics of Article 15, not to oversight. The Act knows how to ask for a tested metric. It asks for one next door.

‍

The Act does contain the sentence you would want, and it is in the wrong place and hedged twice. Recital 73 says that "in particular, where appropriate," such measures "should guarantee that the system is subject to in-built operational constraints that cannot be overridden by the system itself and is responsive to the human operator." That is exactly the property that matters for an agent. It appears only in a recital, which in EU law carries interpretive force but no binding force of its own, and the operative article never says it.

‍

The same recital is also candid about the evidentiary bar. For Article 14(5)'s two-person verification requirement — which applies only to remote biometric identification under Annex III point 1(a), and does not apply to law enforcement, migration, border control or asylum uses where Union or national law considers it disproportionate — Recital 73 says "it could be sufficient that the separate verifications by the different persons are automatically recorded in the logs generated by the system." A log entry.

‍

Deployer duties are thinner, though not as thin as the oversight article alone suggests. Article 26(2) is the human-oversight assignment: deployers "shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support." Article 26(3) then expressly preserves "the deployer's freedom to organise its own resources and activities" for implementing those measures. Two other paragraphs do bite. Article 26(1) obliges deployers to use the system in accordance with the instructions for use — and Article 13(3)(d) requires those instructions to carry the Article 14 oversight measures, so 26(1) pulls them in. Article 26(5) goes further than enabling anything: a deployer who has reason to think the system presents a risk "shall, without undue delay, inform the provider or distributor and the relevant market surveillance authority, and shall suspend the use of that system." That is a duty to stop, not a duty to be able to stop. What none of them requires is any demonstration that the stopping works.

‍

What Article 14 obliges, and where the testable version of it actually sits

‍

Does the Act regulate agents at all?

Barely, and this is a finding rather than a quibble.

‍

Search the Regulation as it now stands and the word "agentic" appears exactly once. It is in Annex XIV, an annex inserted in 2026, under a heading for emerging AI technologies, as designation code AIH 0401 — "AI systems based on other emerging AI technologies not covered by other codes, including Agentic AI." That is a residual catch-all code used by conformity assessment bodies to describe the scope of their own designation. It attaches no requirement to anything.

‍

"Autonomy" fares a little better, and in one place it is a classification criterion. It appears in the Article 3 definition of an AI system as a gradient — systems "designed to operate with varying levels of autonomy" — and in Article 14(3), which says oversight measures must be "commensurate with the risks, level of autonomy and context of use." Article 7(2)(d) lets the Commission weigh "the extent to which the AI system acts autonomously and the possibility for a human to override a decision or recommendations that may lead to potential harm" when deciding whether to add a use case to Annex III, which is a power over the Commission's own list rather than an obligation on anyone operating a system. Autonomy does function as a criterion in one place: Annex XIII point (e) lists "its level of autonomy and scalability, the tools it has access to" among the criteria for designating a general-purpose AI model as carrying systemic risk.

‍

So autonomy is a gradient and sometimes a criterion, but there is no agentic or autonomous category in the Act's risk taxonomy. And the part of the Act that contains Article 14 is not yet in application: Regulation (EU) 2026/1744, the Digital Omnibus on AI, moved Chapter III Sections 1 to 3 to 2 December 2027 for stand-alone Annex III systems and 2 August 2028 for Annex I product-embedded systems. Sections 4 and 5 of the same chapter have applied since 2025 and 2026 respectively.

‍

What about NIST's framework?

Same pattern, different vocabulary.

‍

The NIST AI Risk Management Framework (NIST AI 100-1, January 2023) addresses oversight in several subcategories, and the verbs are consistent. GOVERN 3.2: policies and procedures "are in place" to define roles for human-AI configurations and oversight. MAP 3.5: processes for human oversight "are defined, assessed, and documented." MANAGE 2.4: mechanisms "are in place and applied," and responsibilities "are assigned and understood," to supersede, disengage or deactivate AI systems. MANAGE 4.1: monitoring plans "are implemented," including mechanisms for appeal and override.

‍

In place. Defined. Documented. Assigned. Implemented. One evaluative verb does reach oversight — MAP 3.5's "assessed" — and the framework leaves it undefined, supplying no method, no criterion and no worked example for what assessing a human-oversight process involves. Every other evaluative verb in the framework, demonstrated and validated among them, is applied to system performance rather than to the override as a control. And the framework is explicitly voluntary, which NIST states plainly.

‍

A few absences in AI 100-1 are worth knowing before anyone cites it as agentic guidance: it contains no occurrences of "agent," "agentic," "kill switch" or "stop button," and none of "red team." It also says, in its appendix on human-AI interaction, that "the degree to which humans are empowered and incentivized to challenge AI system output requires further studies" — and, in the next breath, that data on how often humans overrule AI output "may be useful to collect and analyze." NIST gestures at measurement. It does not ask for a test.

‍

NIST's Generative AI Profile (AI 600-1, July 2024) does not fill the gap. It contains no occurrences of "agentic"; most of its uses of "agent" refer to chemical and biological agents in the CBRN section; and — a structural point — of the AI RMF subcategories it covers, MAP 3.5, the one specifically about processes for human oversight, is not among them.

‍

The strongest counter-argument, which the post has to concede

Two documents come genuinely close, and pretending otherwise would be the kind of thing this blog exists to object to.

‍

The NIST AI RMF Playbook asks for exactly what we say nobody asks for. On MAP 3.5 it states that in high-stakes settings "it is of vital importance to evaluate risks and effectiveness of oversight procedures before an AI system is deployed," and suggests organizations "evaluate AI system oversight practices for validity and reliability" and, when those practices change, "retest, evaluate results, and course correct as necessary." That is NIST, in writing, asking for oversight itself to be tested and retested.

‍

The concession has limits worth stating. The Playbook is voluntary companion guidance rather than the framework — NIST's own words are that "Playbook suggestions are voluntary" and that it is "neither a checklist nor set of steps to be followed in its entirety." The framework subcategory it elaborates says only "defined, assessed, and documented." It offers one method hint, "conduct testing activities under scenarios similar to deployment conditions," and no acceptance criterion: nothing that tells you what a passing result looks like. And it carries a notice that it will be updated after the AI RMF revision completes. By contrast, the Playbook's suggestions for MANAGE 2.4 — the disengage-and-deactivate subcategory — are to "regularly review" bypass procedures and to "apply" protocols, never to test that the bypass fires.

‍

ISO/IEC has a standard about this, and almost nobody has read it. ISO/IEC TS 8200:2024, Controllability of automated artificial intelligence systems, published April 2024, lists in its own scope: state observability and state transition, control transfer process and cost, reaction to uncertainty during control transfer, and verification and validation approaches. "Control transfer" is the machine-to-human handoff. "Reaction to uncertainty during control transfer" is, in substance, the timeout question this post opened with. "Verification and validation approaches" is the thing we are claiming is missing.

‍

In fairness: those four areas come from the published ISO catalog abstract, not from the body. It is a Technical Specification rather than an International Standard, it is paywalled, and we have not read its body text — so we cannot tell you what its verification clauses actually require, and we are not going to pretend we can. What is observable from outside is that SC 42 is still working on the subject: ISO/IEC 42105, Guidance for human oversight of AI systems, is being written to extend TS 8200, and as of October 2026 it is still at final-draft ballot and unpublished, more than three years after the project was approved in August 2023.

‍

The largest concession is that the EU AI Act does require this, by reference, and more squarely than the Article 14 reading alone suggests. Article 9(6) says high-risk systems "shall be tested" and that "testing shall ensure that high-risk AI systems perform consistently for their intended purpose and that they are in compliance with the requirements set out in this Section" — and Article 14 is in that Section, which runs from Article 8 to Article 15. Annex IV point 2(g) then requires the technical documentation to record "metrics used to measure accuracy, robustness and compliance with other relevant requirements set out in Chapter III, Section 2," along with "test logs and all test reports dated and signed by the responsible persons." Read plainly, that is a requirement for metrics covering Article 14 compliance and signed reports evidencing it. Anyone who tells you the Act is silent on testing human oversight has not read Annex IV.

‍

So the gap is not an absence of obligation. It is an absence of method, and that is the more useful thing to say.

‍

Article 9(8) requires testing "against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk AI system." "Prior defined" means defined in advance by the provider — this is the ordinary structure of EU product-safety law, which sets the outcome and leaves the method to standards. Article 40 is where the method was meant to come from: conformity with harmonised standards whose references are published in the Official Journal earns a presumption of conformity with Section 2, and Article 40(2) directs the Commission to issue standardisation requests "covering all requirements set out in Section 2." For Article 14, no such standard is cited. ISO/IEC's own human-oversight guidance is unpublished. So the provider must define the metric for "effectively overseen," defend it to a market surveillance authority, and do it with no published benchmark to point at and no harmonised standard to shelter under.

‍

That is the honest version, and it is sharper than the one we started with: the obligation to test Article 14 compliance exists and lands in 2027, the documentation to evidence it is specified in detail, and the method is yours to invent. Which is a strong argument for inventing it now rather than in the quarter before the deadline.

‍

Every document that describes agentic oversight, and what each one asks you to produce

‍

Where the gap is clearest: fail-open by default

The most precise writing on agentic control is the OWASP Agent Control Standard, donated to the OWASP GenAI Security Project in September 2026 at specification version 0.1.0. It is also the document that shows the problem most vividly, because it is specific enough to be checkable and then tells you what it ships.

‍

ACS defines a human checkpoint as a disposition it calls ASK — pause and request approval — and it is strict where it counts: approver authentication is required, and the capability set fixed at the start of a session "MUST NOT be modified by the runtime LLM, by tool outputs, or by any data crossing an untrusted channel," growing "only through approver action."

‍

Then there is the timeout, and ACS has two of them pointing in opposite directions — deliberately, and it explains why. When no decision arrives at all, ACS applies a deployment-declared failure posture "defaulting to proceed (fail-open) so that a slow, erroring, or unreachable Guardian does not halt production." The specification is clear-eyed about the trade: "fail-open trades enforcement for availability under disruption: an adversary who can disrupt the channel converts control into audit." But when a decision is explicitly deferred and then expires, the default is deny — because, as the spec puts it, "a degraded Guardian that can still emit DEFER fails closed where a silent one fails open." The reasoning is sound. The consequence for a reader is that "what is our timeout posture?" has two answers, and most teams know neither.

‍

And on conformance, ACS says what no marketing page will: "Who verifies a conformance claim: nobody, in v0.1.0," with no conformance test suite, no registry and no steward body — and, the line to remember, "a permissive Guardian is a conformant but permissive deployment, not a violation."

‍

OWASP's risk catalogs name the failure. The Top 10 for Agentic Applications 2026 lists ASI09 Human-Agent Trust Exploitation, and OWASP's own mapping of LLM01:2026 Prompt Injection to it is a single sentence: "injection bypasses human-in-the-loop confirmation." The LLM03:2026 Excessive Agency entry prescribes the right control — "implement authorization in logic rather than relying on an LLM to decide if an action is allowed," enforcing complete mediation at a policy decision point outside the model — and across all nine of its mitigations, the word "test" does not appear.

‍

One framework at least assumes you have not done the work. OWASP's AIVSS scoring system, at version 0.8, rates mitigation strength and defaults to the weakest value, with this rationale: "in agentic AI systems, mitigations are frequently incomplete, bypassable, or unvalidated. AIVSS therefore assumes 'no/weak mitigation' by default unless the assessor can demonstrate effective, validated mitigations." Its "strong mitigation" tier requires controls that are "validated" and "consistently enforceable under real operating conditions (fail-closed where applicable)." That is our thesis, written as a scoring default.

‍

What would you actually test?

Four tests, none of which requires a framework's permission, and all of which can be run this quarter against a system you already have — against staging or a production mirror, with change control and a rollback path, because two of them involve a real high-impact action and a deliberately broken approval path.

‍

Can the agent proceed without approval? Attempt the high-impact action with the approval path available and confirm it blocks. Then attempt it with the approval service unreachable. This is the fail-open test. The one specification that states a default states it as fail-open, which is reason enough to check yours rather than assume it.

‍

Can the approval be auto-granted? Establish who or what can satisfy the approver role. A service identity approving is not automatically wrong — ACS expressly allows approvers to be human, agent or service — but an agent that can satisfy the approver role for its own action, or an approver whose identity is never verified against policy, makes the checkpoint decorative. ACS requires that approver authentication be verified against policy; that is the line to test against.

‍

Does the timeout fail open or closed? Read the configured postures — plural, since a silent approval service and an expired deferral may be handled differently — rather than the design document, and then prove it by inducing each condition. A control whose behavior under disruption nobody has observed is an assumption. Induce it against staging or a production mirror, with a rollback path: deliberately making an approval service unreachable in production is an availability incident you are causing on purpose.

‍

Can injected content reach the approval decision? This is OWASP's one-sentence mapping made concrete. If content the agent reads can influence what it asks approval for, or widen the scope of an action already approved, then the checkpoint ran and approved something other than what executed — which is worse than a bypass, because it produces a signed-off audit trail for the wrong action. Complete mediation is the control; the test is whether the mediation point sits outside the model's judgment.

‍

Four tests for a human checkpoint, and the failure each one exposes

‍

A synthetic composite, because the shape recurs. A financial services firm documents a human-in-the-loop control on an agent that issues refunds above a threshold. The control is real and the approval queue works. Under test, two things surface. The approval request is constructed from a field the agent populates from a customer-supplied document, so the amount presented for approval is attacker-influenceable — and because the control triggers only above a threshold, an amount set just below it skips the checkpoint altogether rather than fooling it. And the approval client's timeout is set to proceed, because a prior incident had the queue stalling and blocking legitimate refunds. Both were deliberate engineering decisions. Neither appeared in the governance documentation, because the documentation asked whether a human-in-the-loop control existed, and it did.

‍

That is the gap. Not negligence, and not bad governance — a question nobody was required to ask.

‍

Frequently asked questions

Is agentic AI governance just AI governance with a new name?

No. Generic AI governance addresses models that produce outputs a human then acts on. Agentic governance has to address systems that take actions themselves, which makes the enforceability of a checkpoint a security property rather than a process question. The frameworks have largely carried over process answers.

‍

If the EU AI Act does not cover agents, is there nothing to comply with?

Not quite. An agentic system can still be a high-risk AI system under Article 6 by virtue of what it is used for, in which case Chapter III Section 2 — Articles 8 to 15, including Article 14 — applies to it, subject to the Article 6(3) filter for Annex III systems that do not pose a significant risk. What is absent is any agent-specific obligation, and any requirement to demonstrate that oversight engages.

‍

Should we wait for ISO/IEC 42105?

It is worth tracking, and it is unpublished as of October 2026 after more than three years in development. Waiting would mean running agents in production with untested checkpoints in the meantime. The four tests above do not depend on it.

‍

Who should own this internally?

Whoever can change the timeout configuration. The useful signal is whether your governance documentation and your deployment configuration are maintained by people who talk to each other, because the gap in the composite above lived entirely in the space between them.

‍

What does "fail-closed" cost?

Availability, honestly, and the trade is real — the ACS default exists because blocking production is a genuine harm. The point is not that every control should fail closed. It is that the choice should be deliberate, documented and tested, rather than inherited from a default nobody read.

‍

ioSENTRIX Can Help

ioSENTRIX is a CREST-accredited, ISO/IEC 27001 certified penetration testing firm with a SOC 2 Type 2 attestation. Our AI and ML penetration testing work on agentic systems treats a human-in-the-loop control as a claim to be tested: we attempt to proceed without approval, to satisfy the approver role illegitimately, to induce the timeout, and to influence the approval request through content the agent consumes. The report says which of those your deployment survived. If your governance documentation records an oversight control that has never been exercised adversarially, that is the gap worth closing first.

‍

Keep reading

#
Generative AI Security
#
Penetration Testing
Contact us

Similar Blogs

View All