LLM failover design guide: recover requests without hiding unsafe retries

A useful failover policy is a closed ladder of approved routes, retryable errors and deadlines. It records every attempt and stops before a reliability feature becomes duplicate tool execution, runaway cost or a silent model substitution. Design the application side and gateway side together because neither layer can infer the other's side effects.

Published 2026-08-15 · Updated 2026-08-15 · KeepRouter Editorial · 10 minute read

LLM request state machine showing safe retry, fallback, cancellation, and side-effect boundaries — Failover policy must preserve operation identity and stop before an unsafe duplicate side effect.

Short answer: failover should recover only the failures your application has classified as safe to repeat. Start with an approved route set, then define retryable errors, per-attempt timeouts, a total deadline, streaming rules, idempotency and cost evidence. Do not let a generic retry policy silently change models or execute a tool twice.

Separate the recovery actions

Recovery action	Route change	Appropriate when	Main danger
Retry same route	No	A transient network or selected server error may clear	Duplicate tokens and longer deadline
Try another provider route	Provider or deployment changes	Equivalent route has passed the same acceptance suite	Region, revision, data policy or cache can differ
Fall back to another model	Public model or family changes	Product has explicitly approved different behavior	Quality, tools, safety and price can change
Stop and surface error	No further attempt	Failure is permanent, deadline is spent or side effect is uncertain	Lower completion rate, but preserves correctness

The last option is part of a sound policy. A system that always returns something can be less reliable than one that stops before corrupting state.

Build an error taxonomy from real responses

Collect errors from the exact endpoints and providers you use. Authentication failures, unknown model IDs, unsupported fields and malformed input normally require a configuration or code change. Repeating the request does not alter those facts.

Rate limits, connection resets, selected 5xx responses and timeouts may be retryable. Even then, inspect provider guidance, Retry-After headers and the remaining product deadline. A coding-agent turn can tolerate a different delay from autocomplete or voice interaction.

Store the original error class and every later attempt. Returning only the final 200 makes an unstable route look healthy and hides the cost of recovery.

Budget time across the whole ladder

Set one logical-operation deadline and divide it among attempts. If the client allows 12 seconds, two independent 10-second upstream timeouts do not fit. Reserve time to stream a response and to return a useful error.

Use exponential backoff only where waiting can improve the condition. Backoff after a permanent schema error wastes the entire deadline. Apply jitter when many workers could retry together after a shared outage.

The gateway latency guide shows how to report first attempts and rescued calls separately.

Streaming needs a state machine

Before the first output event, another attempt may be invisible to the user. After partial output, replay can duplicate text or produce a contradictory answer. Track whether the client has received headers, content, a tool call, usage and a terminal event.

Choose an explicit policy for partial streams: stop and return a marked incomplete response, resume only if the protocol and provider support it, or restart as a new user-visible attempt. Do not concatenate output from different models as though it were one response.

Protect side effects at the application boundary

The gateway sees a model request. It usually cannot know whether a generated tool call has already sent an email, charged a card or changed a database row. The application must assign a stable operation ID, store tool state durably and reject duplicate execution.

Require the same tool schema and authorization tests for every fallback model. A weaker model that formats arguments differently is not an equivalent route merely because it speaks the same compatible API.

Reconcile the cost of all attempts

Cost source	Why it can repeat
Input tokens	Every provider receives the prompt again
Output tokens	A failed stream may already have generated content
Cached input	Cache keys and availability can differ by provider route
Gateway request fees	Some products charge by request or plan usage
Tool operations	External systems may perform paid or irreversible work

Attach all attempts to one logical operation, but retain per-attempt model, usage, status and charge. Product teams need the total; platform teams need the route detail. The cost-control article explains this attribution model.

Run the failure matrix before production

Reject invalid auth and confirm there is no retry.
Send an unsupported parameter and confirm the policy stops.
Simulate a rate limit with and without Retry-After.
Delay the first byte past the attempt timeout.
Interrupt a stream before and after partial content.
Fail after a tool call is proposed and after a tool result is returned.
Exhaust the total deadline and inspect the client error.
Compare request records and billing evidence for every attempt.
Disable the primary route and prove the rollback control.

Start with what LLM API failover means, then document the final policy beside the routing configuration. The policy is a production contract, not a hidden gateway default.

Frequently asked questions

Should every 5xx response be retried?

No. Use provider guidance, endpoint behavior, remaining deadline and side-effect state. Some server errors are persistent.

Can failover switch models?

Only when the product explicitly allows it and the fallback passed the same task, tool, safety and cost checks.

How many attempts are safe?

There is no universal count. Bound attempts by the user deadline, retryable condition, side-effect risk and total cost ceiling.

What should be logged for failover?

Record one logical operation ID plus every attempt's route, model, error, timestamps, usage, charge and final state.

Sources reviewed

Article last reviewed 2026-08-15