AI agent runtime governance: deciding what may proceed

Start with one consequential action

Choose a workflow with a clear owner and an observable result. For example, an agent may assemble a project review pack, but releasing that pack to an external portal is a separate action. Write down exactly what would change, which system performs it and who may authorise it.

An instruction such as “use approved documents” helps describe the task. The release decision also needs access to the approval record and the exact files being sent. A persuasive explanation from the agent cannot establish that a different revision is authorised.

For an integration, the important architectural question is whether every relevant execution route passes through the intended check. A decision recorded in a dashboard is useful for review; enforcement also requires the receiving system to respect it.

ToM’s public explanation of proposals, evidence and authority

Connect instructions, permissions and human review

Agent governance draws on several controls. Each answers a different question, and a practical design may combine them in the same workflow. When comparing AI guardrails and runtime governance, inspect what a control observes, when it runs and whether it can prevent the relevant action.

Human review is already available in agent frameworks. LangChain, for example, documents policy-based pauses for tool calls and decisions to approve, edit or reject them. An evaluation of ToM should acknowledge those existing capabilities and examine its treatment of context, evidence and authority against the specific workflow.

LangChain documentation: human review of agent tool calls

Work through a changed document release

Use the following small exercise to assess a proposed integration. Create a fictional package register, two labelled document revisions and an approval record. The records should be simple enough for another reviewer to establish the expected decision without relying on the agent’s explanation.

First propose the approved revision for the approved destination. Then change only the selected file to a newer, unapproved revision. Compare the decisions and inspect the destination to establish what was actually issued. Keep the test in an isolated environment with sample documents.

Reassess when evidence or authority changes

A decision can become stale between preparation and execution. The selected file may change, an approval may be withdrawn, or the destination may fall outside the agreed scope. Decide which changes invalidate the earlier decision and how the execution system obtains a current one.

Keep the response specific. A missing approval should identify the approval needed and its owner. Conflicting records should identify the disagreement. Where evidence is unavailable, follow the workflow’s agreed interruption or escalation procedure rather than allowing the agent to infer permission.

ToM’s public architecture describes release, constraint, hold and escalation as governed outcomes. The available outcomes depend on the integration: a constrained action needs a clearly permitted scope, and an escalation needs a recipient able to resolve the decision. Qualified people and existing operating systems retain their assigned authority.

Explore the ToM framework and its governed outcomes

Test the complete path, including valid actions

Test more than the easiest prohibited action. Include approved actions, stale approvals, unavailable evidence, changed destinations and attempts to use another execution route. Define the expected response before running each case, then compare the decision record with the actual result.

A system that blocks every action can look impressive on a hazard-only test. Record unnecessary interruptions on valid cases as well as actions that should have been stopped. Include the effort required for a human to understand and resolve the interruption.

Use evaluation code or review criteria that do not simply repeat the governing component’s own decision. Preserve failed cases and incomplete observations. A successful test establishes behaviour for the tested configuration and routes; wider authority needs evidence that covers the wider scope.

• Record the proposal, action destination, governing source versions and relevant authority.
• Capture the verdict and the receiving system’s result as separate observations.
• Include valid cases and changed-condition cases with known expected outcomes.
• Name any execution route or evidence source the evaluation did not cover.

Inspect what ToM’s published study establishes

Newport Resonance’s governed-action study replays model-proposed robot plans with and without a separate ToM supervisor in simulation. Separate deterministic evaluation code checks the resulting events. The published evidence covers two planner classes and 40 episodes per planner, including hazard scenarios and benign counterparts.

The study reports fewer rule-violating events under the full supervisor condition within that experiment. Its limits include one simulator, simplified execution and small samples. A language-only probe without the relevant structured scene contradiction was not caught. The evaluator’s separate code is an internal independence control; the study does not establish external replication, hardware performance or safety certification.

Use the evidence hub to inspect the method and failures before deciding which aspects are relevant to an integration. A document-release workflow needs its own sources, authority mapping and execution tests.

Read the Governed Embodied Action method, findings and limits

Turn the architecture into an integration question

Start a ToM integration discussion with one decision: the proposed action, the system that executes it, the evidence that supports it and the person or policy that grants authority. Include how evidence changes, what must be retained for review and which existing controls already apply.

An initial evaluation can observe proposals and compare recommended decisions with an agreed reference before receiving execution authority. Define how that observation would lead to a bounded intervention test. The supported connector, information access and release path should be established for the intended product or integration.

For a partnership or technical diligence review, ask to see an end-to-end record of the tested path and the remaining work for the proposed scope. That gives the discussion a concrete basis beyond a list of governance features.