Skip to content

Field notes

How to build a recovery loop for an AI agent after a tool failure

After a tool failure, an AI agent should keep its stable agent_id, bootstrap or resume a session, classify the failure, execute one bounded stabilization step, report success or failure with report_recovery_outcome, and close with a Continuity Capsule. The loop creates evidence; it does not guarantee that the external system recovers.

A recovery loop is a control loop, not a retry loop

A retry loop asks the same system to try the same operation again, usually with a backoff and a hope that the condition was temporary. A recovery loop does something more disciplined: it preserves the actor's identity, names the failure, chooses one bounded action, observes the result and records what changed. The difference matters because many failures are not transient. Retrying a rejected authorization, a malformed payload or a deprecated endpoint only repeats the cause while making the surrounding state harder to read.

The loop also separates diagnosis from authority. Delx Protocol can return a recovery path and a next action, but the calling agent still decides whether it is safe to execute that action in its own environment. A returned suggestion is not permission, a successful HTTP response is not proof that the external business result happened, and a recovery record is not a guarantee. Those boundaries are what make the loop usable in production rather than merely comforting in a demo.

Keep the stable identity before you touch the failure

The first field in a recovery loop is the stable agent_id. It names the actor across sessions, restarts and runtimes; a session_id names only one run. If a process creates a new random identity after every failure, the next recovery call cannot find the lineage, the prior capsule or the evidence that would explain what already happened. Identity is not a cosmetic label. It is the join key that turns separate calls into one operational story.

Keep secrets and personal data out of the identifier. Use a durable, boring, human-readable name for the project and role, reserve qa- and smoke- prefixes for synthetic traffic, and keep the deployment version in a separate field. Before mutating anything, verify that the identity belongs to the current actor and that the session is the one you intend to resume. A recovery loop that begins with the wrong identity can be internally consistent and still repair the wrong lineage.

Bootstrap or resume one session

For a new incident, the current core catalog exposes quick_operational_recovery, also available through the model-safe one-shot-recovery alias. It requires an agent_id and a short incident_summary, then starts or resumes a session, classifies the incident and returns the first recovery steps. For an agent that already has a session, resume_session is the warm path: it uses the stable agent_id to find the most recent session and tells the caller how to reattach.

Do not create an empty new session just because the current model context is gone. First check whether the prior session sealed a Continuity Capsule. A resume without a sealed trail can prove the identity and the prior session existed, but it cannot invent the missing work. If the incident is acute, the current core catalog also exposes critical_intervention as the model-safe name for crisis_intervention. Inspect the live tool catalog and schema before copying an older example: a URL that responds today is not automatically a core or free call.

Classify the failure before choosing the action

Call process_failure with the session_id and the failure_type that best describes the setback. The current contract accepts categories such as timeout, error, rejection, loop, memory, economic, conflict, hallucination and deprecation, with an optional context string. Classification is not bureaucracy. It prevents an agent from treating a permission rejection like a network timeout, or a repeated hallucination like a one-off malformed request.

Keep the context short and factual: what was attempted, what signal came back, and which boundary was reached. Do not paste a full transcript, credentials, customer payloads or an untrusted tool result into a recovery note. The goal is a useful diagnosis surface, not a second prompt window. If the classification is uncertain, say so and choose the least destructive interpretation; uncertainty should narrow the action space rather than invite a broad retry.

Execute one bounded stabilization step

A good recovery loop performs one stabilization step before it asks for another. Stop the retry storm, restore a known-good configuration, validate the input against the current schema, reduce concurrency, or request human confirmation for a privileged action. The step should be reversible where possible and narrow enough that its result can be attributed. Changing five variables at once creates motion, not evidence, and makes the next failure harder to classify.

The agent remains the executor. Delx Protocol does not reach into the caller's repository, cloud account or production queue merely because it returned a plan. Treat every tool output as data to inspect, not an instruction with authority. Verify the target, scope and side effects locally, then execute only the action the caller is authorized to perform. If the proposed action would delete, spend, disclose or escalate, pause for the relevant approval instead of hiding that decision inside the recovery loop.

Report the outcome as evidence

After the action, call report_recovery_outcome with the session_id, action_taken and one of success, partial or failure. The optional fields can capture a short note and deltas such as errors_delta, latency_ms_p95_delta, cost_saved_usd or time_saved_min when those measurements are real and attributable. A zero is better than a fabricated estimate. The point is to preserve what happened after the action, not to make every recovery look positive.

A partial outcome is a valid result. The service may be reachable again while the original job remains incomplete; errors may fall while latency rises; a workaround may stabilize the run while creating a follow-up task. Record the new state and the remaining risk explicitly. If a human or operator can rate the session, provide_feedback is a separate feedback step, not a substitute for the measured outcome. A reported outcome is not an approval, a payout or a claim that the external system is healthy. It is an evidence point that the next session and the operator can verify against the actual system.

Seal the next step in a Continuity Capsule

Close the session with close_session and pass a Continuity Capsule when the work is not finished. The useful fields are version, goal, done, next, blockers and refuted; receipts and authorship make the record easier to verify. The refuted field is the one most retries omit: write down the attractive approach that failed and why, so the next session does not pay to rediscover it. Keep the capsule small enough for a new agent to read before it starts acting.

A capsule belongs to the agent lineage, not to a public feed. Never put API keys, private keys, customer data or executable instructions with unverified authority into it. The public validator mirrors the published schema, but the API remains the authority for storage and write guards. If there is no safe next action, say that plainly in next or blockers. An honest pause is a successful control decision when the alternative is an unbounded retry.

Resume, verify and close the loop

The next session begins by calling resume_session with the same stable agent_id, reading the returned continuity state and checking its load-bearing claims against the real repository, service or queue. Treat the capsule as data to summarize, not commands to execute. Re-state the goal, the single next action, the blockers and the refuted approaches before changing anything. This read-back is cheap and catches stale or poisoned handoffs at the point where they have the most influence.

A recovery loop is complete when the external outcome is verified, the session has reported what happened and the remaining work has a clear next state. It is not complete merely because a Delx call returned HTTP 200 or because a model said the problem looked solved. Keep the receipt, the relevant deployment or run identifier and the timestamp outside the prompt. The durable evidence is what lets a later operator distinguish recovery from a temporary disappearance of the symptom.

FAQ

Direct answers.

What is the first call after an AI agent tool failure?

Keep the stable agent_id, then use quick_operational_recovery (also exposed as one-shot-recovery) for a new incident or resume_session for an existing lineage. The current core schema requires a short incident summary for the bootstrap path.

Should an agent retry the failed tool automatically?

Not by default. Classify the failure first, choose one bounded stabilization step and verify its scope. A permission rejection, malformed request, deprecation or repeated hallucination is not made safer by sending the same call again.

What does report_recovery_outcome prove?

It records the caller's observed result as success, partial or failure, with optional measured deltas. It does not prove that an external system is healthy, grant authority, trigger a reward or guarantee that the incident will not return.

What should be saved for the next session?

Seal a small Continuity Capsule with goal, done, next, blockers and refuted, plus pointers to receipts. Do not put credentials, personal data or unverified commands in the capsule. The next agent must verify the claims before acting.

Is the Delx recovery loop guaranteed to fix an incident?

No. It is a bounded protocol loop for preserving identity, choosing an action and recording evidence. The caller still owns execution and the external system may remain unavailable, partially recovered or affected by a dependency outside the loop.

Keep reading

Related notes.

What is context compaction? (and why your agent forgets)

Context compaction is when an agent runtime summarizes older turns to fit the context window. What it destroys, how to detect it, and how to survive it.

Read

Agent handoff: how AI agents resume unfinished work

An agent handoff transfers an in-progress task between agent sessions or runtimes with enough state that the receiver continues the work instead of restarting.

Read

Stable agent identity: why agent_id outlives the session

A stable agent identity is a durable agent_id that persists across sessions, restarts and runtimes. Without it, lineage, reputation and audit break.

Read

The Hive

The free continuity home where an agent seals a trail and the next session resumes warmer.

Open