New Update: Our new blog is updated.
Free download
Omair Manzoor is the Founder, CEO, and Chief Hacker of ioSENTRIX, a CREST-accredited cybersecurity firm. With 20+ years in offensive security, he has led penetration tests for Fortune 500 companies including engagements for Amazon, Bank of America, VISA, and the London Underground. His exploits are part of the Metasploit Framework and Immunity Canvas. Omair is a published researcher, conference keynote speaker (DOD, CISA, BSides), and technical reviewer for academic security textbooks.
Your incident response plan has three assumptions baked into it so deeply that they are not written down anywhere.
You can isolate the thing that acted. You can preserve its state. You can replay the logs and establish what happened.
Now run the plan against this: an autonomous agent processed a support ticket, made eleven tool calls, and issued a refund it should not have issued. It ran under a service identity shared by the whole triage fleet. The instruction that caused it came from the ticket body, which a customer wrote. The model is stateless. The agent framework is managed. By the time anyone notices, the conversation has rolled out of the context window.
Which host do you isolate? What do you image? And when the regulator asks what the root cause was, what do you show them?
That is a synthetic composite, not a client. The assumptions it breaks are real.
It is claimed ground, but it is immature ground — the published guidance exists, almost none of it is final, and none of it tells you whether your own plan survives contact.
It would be convenient to claim nobody has written this down. That is not true, and it is worth being precise about who has:
So the honest position is this: there is no final government standard for AI incident response procedure. There are two solid community frameworks, one preliminary NIST draft, and one ISO standard still in committee. The frameworks are arriving, and faster than most plans are being updated. What none of them can do is tell you whether the plan you already have works, and that is the gap this post is about.
The general IR standard has also moved, and plenty of plans still cite the old one. NIST SP 800-61r3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management, is final as of April 2025, and it abandoned the four-phase life cycle that Revision 2 carried forward in 2012. NIST's reasoning, in its own words: "incidents occur frequently and cause far more damage. Recovering from them often takes weeks or months due to their breadth, complexity, and dynamic nature." Rev 3 restructures incident response around the six CSF 2.0 Functions instead, and says "all six Functions have vital roles in incident response."
If your plan is built on Preparation / Detection & Analysis / Containment-Eradication-Recovery / Post-Incident Activity, it is built on a superseded structure. That is a separate problem from the AI one, and it is cheaper to fix.
Containment of an AI incident means revoking the agent's ability to act, not isolating the machine it ran on — because in most deployments there is no machine that is meaningfully "it."
Walk the usual options and watch them fail.
Isolate the host. The model is behind an API. The agent framework may be managed. Isolating your own compute stops new requests and tells you nothing about what already happened.
Disable the account. Which account? If the agent runs under a shared service principal, disabling it stops the whole fleet. If you do not have per-agent credentials, containment and outage are the same action. OWASP's agentic top ten names this directly, and it is the same gap we have written about as non-human identity security: its identity and privilege entry describes "the architectural mismatch between user-centric identity systems and agentic design," and states that "without a distinct, governed identity of its own, an agent operates in an attribution gap that makes enforcing true least privilege impossible."
Roll back. To what? The model is stateless, but the agent's memory, its vector store, and the records it already wrote are not. NIST's draft AI profile anticipates this: for incident management it notes "recovery may include retraining or disabling AI modules" and asks organizations to "define triggers for when to roll back an AI component."
Kill the prompt. Removing the injected instruction from one ticket does nothing if the same payload sits in fifty retrievable documents, or in the agent's memory, or in a tool's cached response.
The containment primitives that actually work are the ones you have to build in advance: a per-agent identity you can revoke alone, a capability kill switch per tool rather than per agent, a hard stop on the tool that moves money or data, and a documented blast radius. If the tool chain itself is reached through a model context protocol server, the trust questions in MCP security come first — you cannot contain a capability you have not enumerated. The NIST AI RMF asks for precisely this capability under MANAGE 2.4 — "mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use."
Here is the test that tells you whether you have it: name the single action that stops your agent acting, without stopping anything else, and have someone perform it in a drill. If nobody can name it, containment is theoretical.
The evidence for an AI incident is the decision chain — prompt, retrieved context, tool calls, arguments, returns, and the identity behind each — and almost none of it is in the places your IR plan looks.
OWASP's incident response guide is specific about the difference. It observes that "AI incidents may require different types of evidence than traditional incident response: e.g., in the case of a data poisoning incident, the team will need to secure and evaluate training data, or if the model is continuously learning, then they need to secure a snapshot." Its forensic-preservation section, in the chapter on attacks via third-party model providers, goes beyond host artifacts: "compromised model files and configurations; inference logs, access logs, and system traces; build, deployment, and CI/CD pipeline metadata; associated training data or datasets."
NIST's draft AI profile says the same thing in control language. Against the CSF subcategory for preserving incident data and metadata, it notes that "AI may result in capturing new types of metadata (e.g., data versions, inputs, hyperparameters)" and asks defenders to "preserve logs, inputs, outputs, and decision chains of AI systems to ensure provenance…"
The properties that record needs are not new. They are the AU family of NIST SP 800-53 Rev 5, and the three that matter most are the three agent deployments routinely fail:
Add AU-9 Protection of Audit Information — protection "from unauthorized access, modification, and deletion" — and you reach the point. A log the agent can write to is not evidence about the agent.
OWASP's agentic material points at the same gap, and says it better than we would. Its cascading-failures entry cross-references a threat taxonomy entry on repudiation and untraceability, and its own mitigation reads: "Logging and non-repudiation. Record all inter-agent messages, policy decisions, and execution outcomes in tamper-evident, time-stamped logs bound to cryptographic agent identities. Maintain lineage metadata for every propagated action to support forensic traceability, rollback validation, and accountability during cascades."
Tamper-evident, time-stamped, and bound to cryptographic agent identities. That is the whole specification in one clause, and it is the clause most agent deployments cannot currently satisfy on any of the three counts.
Because the thing that made the decision cannot be interrogated, and the reasoning that produced it was never stored.
OWASP states it without hedging: "Many advanced AI models, particularly deep neural networks, are opaque 'black boxes.' ... This makes root cause analysis of an AI incident significantly more challenging than debugging traditional cyberattacks." NIST's draft profile is similarly direct on the tooling gap: "new analysis tools may be needed to capture the entire picture of AI system attacks and AI system defenses," and asks teams to "analyze AI-specific artifacts (e.g., model logs, inference tables, provenance data) during an incident to establish the root cause."
Practically, root cause for an agent incident means reconstructing a chain and attributing each link: which input carried the instruction, which retrieval introduced it, which tool call acted on it, under what authority, and why no checkpoint stopped it — and whether there was a human checkpoint to stop it at all. You can only do that if you stored enough at the time. Nobody reconstructs an agent decision from an API gateway log, which is the argument we made at length about telemetry that sees everything and proves nothing.
And a non-obvious consequence: reproduction is not guaranteed. The same input to the same system can produce a different decision; OWASP's incident response guide names the cause, noting that "AI models (especially generative ones) produce outputs that are not strictly deterministic," which "can lead to unpredictable or unintended behavior without any attacker present." An IR process whose root-cause step depends on reproducing the fault will stall. The substitute is the stored chain, which is why the logging decisions are the ones that pay off.
It requires providers of high-risk AI systems to report serious incidents on a statutory clock, and it forbids altering the system in a way that would compromise a later investigation before the authority has been told. The second half is the one that breaks standard practice.
The EU AI Act, Regulation (EU) 2024/1689, Article 73 requires providers of high-risk AI systems to report serious incidents to market surveillance authorities. The deadlines are in the text: not later than 15 days after becoming aware, not later than two days for a widespread infringement or a serious and irreversible disruption of critical infrastructure, and not later than 10 days in the event of a person's death. A "serious incident" is defined in Article 3(49) and includes death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of Union-law obligations protecting fundamental rights, and serious harm to property or the environment.
Then there is Article 73(6), which is the provision that should be pinned to the wall of every IR room running an AI system. The provider must investigate without delay, and "shall not perform any investigation which involves altering the AI system concerned in a way which may affect any subsequent evaluation of the causes of the incident, prior to informing the competent authorities of such action."
Read that against your muscle memory. Standard practice under pressure is to patch the prompt, swap the model version, tighten the filter and get the system safe. The text says an investigation that alters the system in a way which may affect a later evaluation of causes has to wait until the authority has been told. That is a sequencing requirement, and sequencing is a design decision you make before an incident, not during one.
Now the timing, because this is where a lot of writing on the subject gets it wrong. The Act's general application date is 2 August 2026, and Chapter IX, which contains Article 73, is not among the exceptions listed in Article 113. Read alone, that makes the reporting duty look live today for every high-risk system. It is not that simple.
Article 111(2), as replaced by the amending regulation adopted in July 2026, provides that the Regulation applies to operators of high-risk systems placed on the market before the date of application of Chapter III "only if, as from that date, those systems are subject to significant changes in their designs." The carve-out names "this Regulation," not Chapter III, and excepts only Article 5. So for a high-risk system already on the market, Article 73 waits for the Chapter III dates — which that same amendment moved to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems — or for a significant design change, whichever comes first.
The reporting duty and the record-keeping duties in Articles 12 and 19 therefore arrive together, rather than reporting arriving first. The duty that genuinely is in force today is a different one: Article 55 requires providers of general-purpose AI models with systemic risk to report serious incidents, and that has applied since 2 August 2025.
The planning consequence is the part that should drive a decision. The evidence you need to explain an incident has to exist before the incident, and a significant design change can pull your date forward without warning. Build the logging on your own schedule. This describes what the Regulation says and is not legal advice; which limb applies to a given system is a question for counsel.
One clarification while we are here, because it gets muddled in procurement. ISO/IEC 42001:2023 is the international standard for AI management systems, and ISO notes that certification is voluntary, and that organizations choose it when they want "independent confirmation that their AI management system meets the requirements of ISO/IEC 42001:2023." That is a certification of a management system against a standard's requirements. It is not evidence that any given control held when something attacked it, and it is not compliance with the AI Act — a distinction we have drawn before, at more length, in compliant is not correct and across AI governance and compliance standards.
An IR plan that has been exercised against an AI actor, with the gaps written down.
Five things, in order of how cheap they are:
1. Name the containment action per agent, per tool. One sentence each. Then drill it. The drill is the test; the sentence is just the hypothesis. 2. Decide your evidence set before the incident. Prompt, retrieved context, tool calls with arguments and returns, the identity on each call, model and prompt versions, and timestamps from a synchronized source. Write it against AU-3, AU-10(1) and AU-12(1) so the requirement has a citation, and store it where the agent cannot write. 3. Give every agent its own identity. This is the expensive one and the one that unblocks everything else. Without it, containment is a fleet outage and attribution is a guess. 4. Sequence containment against the reporting duty. For anything in AI Act scope, decide now who authorizes a change to the system during an incident, and what gets told to whom first. 5. Run the tabletop with the model as the suspect. Not the exercise in digital forensics and incident response you already run. Not "an attacker used AI." The agent did something wrong, nobody told it to, and you have to explain why.
The pattern across all five is the one this firm argues everywhere: the plan existing is not the same as the plan working, and the only way to tell them apart is to run it adversarially and see what breaks, which is the whole of our AI integrity argument. An IR plan is a control. Controls get tested.
Not a final one. OWASP's GenAI Incident Response Guide v1.0 (July 2025) is the most complete published guidance; NIST's Cybersecurity Framework Profile for AI is an initial preliminary draft; the ISO standard on AI incident reporting data elements is still a committee draft. General incident response is covered by NIST SP 800-61r3, final April 2025.
For providers of high-risk systems in scope, Article 73 sets 15 days from awareness as the default, two days for a widespread infringement or serious and irreversible critical-infrastructure disruption, and 10 days where a person has died.
Usually yes, and usually at the cost of every other agent sharing its identity. Whether you can stop one agent without stopping the fleet is a design question you answer before the incident, not during it.
The tool call: the tool invoked, the arguments, the return, and the identity it ran under, time-correlated with the input that triggered it. Model output alone tells you what was said; the tool call tells you what was done.
No. A model that acts wrongly on its own, or on content nobody planted deliberately, produces the same containment, evidence and reporting problems. The EU AI Act's definition of a serious incident turns on harm and malfunction, not on an adversary.
ioSENTRIX is a CREST-accredited, ISO/IEC 27001 certified penetration testing firm. Our AI and ML penetration testing and AI assurance work includes driving an agent into the state your IR plan has to handle, then testing whether it does — whether containment stops one agent or all of them, whether the decision chain can be reconstructed afterwards, and whether the evidence would survive a question from someone who did not take your word for it. We assure the stack you have chosen and sell no competing platform. If your plan has never been run with the model as the suspect, that is the exercise.
