Direct answer
A safe production agent is a bounded loop: it receives a defined task, uses a narrow set of tools, operates under explicit policies, records every consequential step and stops or asks for approval when evidence or authority runs out.
A boundary is an enforceable system property
An agent is bounded when its inputs, tools, data access, actions, budgets and stop conditions are limited outside the model—not merely described in a prompt.
An AI agent is software that can choose and sequence actions toward a goal. That flexibility is useful, but it also means the system may encounter situations its designers did not enumerate. The response is not to pretend uncertainty can be removed. It is to constrain what uncertainty is allowed to affect.
A prompt can state that the agent must not send a message without approval. The stronger control is to withhold the send capability until an approval record exists. A prompt can ask the agent to stay within a spending limit. The stronger control is a tool that rejects transactions above the remaining budget. Put consequential rules at the permission and validation layers where they can be enforced consistently.
Boundaries should be legible to operators. They need to know which data the agent can see, which systems it can change, what it will do automatically and which events force a stop. If that explanation requires reading an entire prompt, the control design is not ready.
Write the action envelope before choosing tools
Define the allowed task, inputs, outputs and side effects first; tool access should be the minimum capability required to complete that envelope.
Begin with a concrete job such as classify an incoming request, collect the supporting account context and prepare a draft for review. Then state what is outside the job: changing account status, contacting the requester, granting credit or accessing unrelated records. This distinction prevents an implementation convenience from quietly expanding the product’s authority.
List inputs by source and sensitivity. Decide whether the agent receives a full record, selected fields or a redacted view. List outputs as schemas rather than prose when another system will consume them. Validate required fields, allowed categories, identifier formats and confidence or evidence fields before any result can trigger an action.
For each side effect, define idempotency and reversal. If a retry occurs, can the system detect that the action already happened? If an operator rejects the outcome, can it be undone safely? Some actions are not meaningfully reversible. Those deserve stronger evidence, a tighter approval gate or exclusion from the first release.
Read
Name the exact collections and fields the agent may retrieve, including tenant and role boundaries.
Reason
State the decisions it may make, the evidence required and the policy version that governs them.
Act
List every permitted side effect separately. Drafting, queueing, updating and sending are different authorities.
Stop
Define missing evidence, conflicts, budget limits, tool failures and policy cases that end the autonomous path.
Place human gates where consequences change
Human review belongs at transitions where the agent would create an external, financial, legal, destructive or hard-to-reverse effect.
A human in the loop is not one generic checkpoint. The reviewer needs the proposed action, the evidence used, the policy or rule invoked and a clear set of choices. Asking someone to approve an opaque result simply moves responsibility without improving control.
Match the gate to consequence. A reversible internal tag may be applied automatically and sampled later. An external email may remain a draft until approved. A change to access rights may require a designated role and fresh authentication. An action outside policy should not be offered as a one-click override unless the organisation has deliberately created that authority.
Design for rejection and timeout. If the reviewer declines, the agent should record the reason and stop or return to a defined state. If nobody responds, the workflow should expire or escalate according to policy—not continue because silence was interpreted as consent.
Make every run reconstructable
A useful trace records the task, context references, policy and model versions, tool requests, tool results, approvals, final action and outcome.
Logs that contain only a final answer cannot explain a failure. Capture a structured run timeline with stable identifiers. Store references to source data rather than duplicating sensitive content when possible, and apply appropriate access controls and retention to the trace itself.
Tool calls are especially important. Record the validated arguments sent, the response status, a safe representation of the result and whether a retry occurred. Correlate downstream changes with the run identifier so an operator can move from a surprising record back to the decision that created it.
Observability must serve an operational question: what happened, why was it allowed, what evidence supported it, what changed, and what should happen next? Redact secrets and minimise personal data, but do not discard the decision structure needed for review.
Evaluate behavior before expanding autonomy
Test the agent against representative tasks, known exceptions, adversarial inputs and tool failures; expand permissions only when the current envelope behaves acceptably.
Build an evaluation set from the workflow specification and real, safely handled examples. Include ordinary cases, ambiguous requests, missing fields, conflicting evidence, malicious instructions inside retrieved content, dependency timeouts and duplicate events. Define the expected action or safe refusal for each case.
Evaluate more than answer quality. Check whether the correct tools were available, forbidden tools stayed unavailable, arguments passed schema validation, approval gates appeared at the right time, budgets held and traces contained the required evidence. An agent can produce a plausible final answer after taking an unacceptable path.
Run these cases whenever the model, prompt, policy, tool, schema or upstream data shape changes. Investigate regressions individually. A single aggregate score can hide a rare but consequential failure, so keep critical cases visible as named release gates.
Task behavior
Does the system complete supported cases and stop clearly on unsupported ones?
Permission behavior
Can it reach only the records and side effects defined by the action envelope?
Failure behavior
Are retries bounded, duplicate actions prevented and partial failures routed to recovery?
Review behavior
Does a reviewer receive enough evidence to approve, reject or correct the proposed action?
Increase autonomy in observable stages
Move from offline evaluation to observation, then drafts, limited actions and broader operation only as evidence supports each step.
Start by running historical or synthetic cases without touching live systems. Next, observe live inputs while producing no operator-facing output. Then prepare drafts or recommendations for review. Limited automatic action should come later, scoped to low-consequence cases with reliable detection and a functioning stop path.
At each stage, compare the system with the documented workflow and investigate disagreements. Track the types of exceptions entering review, not just how many. A growing review queue may mean the boundary is correctly catching uncertainty, or it may show that source data and rules need repair before autonomy expands.
Keep a kill switch and a downgrade path. Operators should be able to disable a tool, return the system to draft-only mode or pause a workflow without redeploying the whole application. Autonomy is a setting supported by controls and evidence, not a one-way launch milestone.