Loop Engineering Patterns: Retry, Recovery, and Escalation

A loop that runs unattended will, given enough runs, do three things. It will crash in the middle of an action. It will retry something that already happened. It will wedge on a bad input and grind until a budget runs out or someone notices.

The engineering world solved most of these problems for long-running workflows years ago under names like durable execution and idempotency. The agent world is re-learning the same lessons with one new wrinkle: the model can sometimes fix its own mistakes, and sometimes makes them worse when it tries.

The 529 Pattern

Status code 529 is Anthropic's overload signal. It means the upstream is full and you should back off. The distinction from 429 (rate limit - you specifically are over quota), 503 (service is down), and 502 (transit failure) matters because the retry strategy differs for each.

The pattern that works combines four mechanisms:

Exponential backoff with full jitter randomizes the retry interval to disperse the retry wave. Without jitter, all clients retry at predictable intervals - seconds 1, 3, 7 - concentrating load at each spike. With full jitter (delay = random(0, base * 2^attempt)), retries spread across the interval and the API sees a smooth recovery distribution instead of crushing waves.

backoff.tstypescript
function backoffDelay(attempt: number, baseMs = 1000): number {
  const max = baseMs * 2 ** attempt; // 1000, 2000, 4000...
  return Math.random() * max;         // full jitter: random(0, max)
}

async function retryWithJitter<T>(
  fn: () => Promise<T>,
  maxAttempts = 5,
): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt >= maxAttempts - 1) throw err;
      await sleep(backoffDelay(attempt));
    }
  }
}

A circuit breaker stops retrying entirely when the failure rate is too high. After five consecutive failures, the breaker opens. It holds open for 30 seconds, then allows one probe request. If the probe succeeds, the breaker closes. If it fails, the breaker stays open. This prevents the retry storm pattern where many clients all retry simultaneously and make the overload worse.

circuit-breaker.tstypescript
class CircuitBreaker {
  private failures = 0;
  private openedAt = 0;

  constructor(
    private threshold = 5,
    private cooldownMs = 30_000,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.openedAt && Date.now() - this.openedAt < this.cooldownMs) {
      throw new Error("circuit open - not retrying");
    }
    this.openedAt = 0; // allow one probe request through
    try {
      const result = await fn();
      this.failures = 0; // success closes the breaker
      return result;
    } catch (err) {
      if (++this.failures >= this.threshold) this.openedAt = Date.now();
      throw err;
    }
  }
}

A fallback model routes to a different model when the primary is overloaded. Opus and Sonnet share infrastructure but have different load characteristics. Opus is more expensive per call and used by fewer concurrent jobs, meaning those jobs are heavier and saturate it more easily. When Opus returns 529, switching to Sonnet for the next attempt often succeeds immediately.

fallback.tstypescript
type Model = "opus" | "sonnet" | "haiku";

async function callWithFallback(prompt: string): Promise<string> {
  try {
    return await callModel("opus", prompt);
  } catch (err) {
    // 529: upstream overloaded. Sonnet has different load characteristics.
    if (isOverloaded(err)) return await callModel("sonnet", prompt);
    throw err;
  }
}

Hedged abandonment switches to an alternative path entirely. If there is a cached result, a different provider, or a precomputed value, use that instead of retrying the congested path.

Classifying Failures

Not all failures are the same. A production retry strategy classifies errors before deciding the recovery path:

Transient: network timeouts, rate limits, 529 overload signals. Always retry with exponential backoff and jitter.
Permanent: authentication errors, not found, invalid parameters. Never retry - escalate immediately.
Budget: hitting a cost ceiling. Pause the task, notify the orchestrator, await budget top-up.
Capability: the agent requested a tool it does not have. Escalate to the parent agent or human.
Semantic: output failed validation. Retry with explicit format correction in the next turn.
Fatal: unrecoverable state. Mark the task failed, return partial results and an error receipt.

Idempotency Before Retryability

The most load-bearing pattern in loop recovery: tag every operation with an idempotency key so the downstream side recognizes a repeat and returns the original result instead of executing again. Every crash-and-resume is a retry of what was in flight. Make the action idempotent before you make it retryable. A retry on a non-idempotent action is a double-execution generator with good intentions.

Checkpointing

For long-running loops, the pattern is session-based checkpoints. Before every major operation, save the current state. If the loop crashes, it resumes from the last checkpoint instead of starting over.

In practice this means the agent writes its state to a shared memory file after each completed step. A crashed run reads the memory file at startup, finds the last completed step, and continues from there. The Claude Agent SDK's Memory tool is built for exactly this pattern.

The Verifier Gate

Do not let the model grade its own retry. The model's own judgment is the wrong tool for deciding whether a retry succeeded. External verification grounded in real tool results - a code interpreter that runs the code, a search that checks the claim, a test suite that validates the output - corrects reliably where introspection does not.

The rule: the verifier, not the model, decides whether a retry is kept. And every loop runs against caps on iterations, time, and spend, so the retry budget is bounded by construction. If the verifier never passes, the caps end the run.

The Dead Letter Pattern

After an operation fails N times, move it aside so it stops blocking the queue and can be inspected later. A stuck loop does not spin until the spend cap kills it. It hits its attempt limit, routes to a dead-letter path or to a human with the audit trail attached, and the next operation proceeds.

A capped exit is a routine signal that the problem was harder than the loop was provisioned for. It is not a failure state. It is a planned exit path, the same as convergence, just with a different report.

Read the full Context Engineering series: orioverse.com/blog/what-is-context-engineering

A Note for Vibe Coders and Non-Technical Users

Loops that run unattended crash, retry and wedge. You see the same thing in daily prompting: the assistant fails, tries again in the same way, fails again, and you both go in circles. These prompts teach it the retry and recovery patterns from this post - in your own plain text.

title
If an action fails, do not silently keep trying. Tell me that it failed and pause. Briefly say what you think went wrong, and whether it looks temporary or permanent. Then wait for me to say retry.

What it does: turns a retry storm into a controlled stop-and-ask, the circuit breaker behavior.

title
If a service or tool is overloaded, do not hammer it at the same speed. Wait a little longer between each retry, and spread the tries out. After two or three tries, if it is still failing, stop and tell me instead of pushing on.

What it does: prompts exponential backoff with jitter in plain language, so it stops the crushing retry wave.

title
Do not deliver work you only checked in your own head. If the work involves code, a page, a file, or a number, check the real thing before you say it is done. If I ask, show me the evidence, not just your say-so.

What it does: the verifier gate - the model does not grade its own output; it grounds the result in real tool results.

title
If you see an overload signal, do not try harder. Stop, wait, and tell me what is congested. If a waiting period makes sense before retrying, propose it in one sentence.

What it does: teaches the 529/overload stop-and-back-off behavior, not push-through.

Loop Engineering Patterns: Retry, Recovery, and Escalation