# What is LLM API failover?

> LLM API failover is a controlled attempt to complete a request through another approved serving route after the first route fails, times out or becomes unavailable. It is not the same as an unlimited retry, and it should not silently choose an unevaluated model. Safe failover defines eligible routes, retryable conditions, timeout budgets, side-effect protections and evidence for every attempt.

_Last reviewed 2026-08-15 · [Editorial review](https://keeprouter.com/editorial-policy#editorial-team)_

## Retry, provider failover and model fallback are different

| Action | What changes | Main risk |
|---|---|---|
| Retry the same route | Nothing except the attempt | Duplicate work and longer latency |
| Fail over provider | Serving provider or deployment | Different revision, region, policy or behavior |
| Fall back model | Public model ID or family | Different quality, tools, safety and cost |

Use the narrowest action that can recover the failure. A transient connection error may justify a retry. A provider outage may justify another approved provider for the same task. A model fallback needs product evaluation because it can change what the user receives.

## Decide which failures are retryable

Authentication errors, malformed payloads, unsupported fields and invalid model IDs normally require a configuration or code fix. Retrying them consumes time and may add cost without changing the result. Rate limits, selected server errors, connection resets and timeouts can be retryable, but only within the provider contract and your deadline.

Streaming makes the decision harder. If the client already received partial text, replaying the whole request can duplicate content. Tool calls are more serious: an upstream may have produced or executed an action before the connection failed. The application needs idempotency keys, durable tool state and a rule for when human review replaces automation.

## Build an explicit failover ladder

1. List the exact endpoint and capabilities the request requires.
2. Approve each route with the same streaming, tool and output tests.
3. Order routes and set per-attempt plus total deadlines.
4. Classify errors by retryability and stop on permanent failures.
5. Prevent duplicate side effects with stable operation identifiers.
6. Record every attempt, route, error, usage and final result.
7. Alert when failover frequency changes, even if users still receive 200 responses.

The [LLM routing answer](/answers/what-is-llm-routing) explains the broader policy. The [failover design guide](/blog/llm-failover-design-guide) provides a production test plan.

## Account for cost and cache behavior

A failed attempt may still consume input or output tokens. A second provider may not share the first provider's prompt cache. Model fallback can change tokenization and output length. Reconcile each attempt separately and calculate the cost of the whole logical operation, not only the final successful response.

## Keep the user-visible contract honest

If the product promises a specific model, do not silently return another family. If a fallback is allowed, disclose the policy at the right product level and retain the actual model in request evidence. Availability is useful only when the recovered output still satisfies the task.

## Frequently asked questions

### Is failover the same as retry?

No. A retry can repeat the same route, while failover changes to another approved provider, deployment or model route.

### Which errors should not be retried?

Invalid authentication, unsupported parameters, malformed input and unknown model IDs normally need a fix rather than another attempt.

### Can streaming requests fail over?

They can, but partial output and terminal-event state must be handled explicitly to avoid duplication or corrupted client state.

### Does a failed request cost money?

It can. Providers may bill consumed input or generated output before the failure. Reconcile every attempt, not only the last one.

### Should a fallback use a cheaper model?

Only if that model passed the task's acceptance tests and the product contract allows the behavior change.

## Sources reviewed

1. [Cloudflare AI Gateway request handling](https://developers.cloudflare.com/ai-gateway/configuration/request-handling/)
2. [Cloudflare AI Gateway dynamic routing](https://developers.cloudflare.com/ai-gateway/features/dynamic-routing/)
3. [Vercel AI Gateway provider options](https://vercel.com/docs/ai-gateway/models-and-providers/provider-options)
4. [OpenRouter provider routing](https://openrouter.ai/docs/guides/routing/provider-selection)

## Related guides

- [Model routing](https://keeprouter.com/features/model-routing.md)
- [What is LLM routing?](https://keeprouter.com/answers/what-is-llm-routing.md)
- [LLM failover design guide: recover requests without hiding unsafe retries](https://keeprouter.com/blog/llm-failover-design-guide.md)
- [Does an AI gateway add latency?](https://keeprouter.com/answers/does-ai-gateway-add-latency.md)
- [How to control multi-model API costs with evidence, limits, and ownership](https://keeprouter.com/blog/control-multi-model-api-costs.md)
- [agent builders](https://keeprouter.com/built-for/agent-builders.md)

## Test the contract with a real model

Create a narrowly scoped key, select a model from the live catalog, and run the exact request shape your application depends on.

[Create a free key](https://keeprouter.com/login?returnTo=%2Fconsole%2Fkeys%3Fmodel%3Dfree) · [Live models and pricing](https://keeprouter.com/models.md)