LLM routing vs load balancing: four policies teams often confuse

Routing chooses a target because of task or policy; load balancing distributes work among targets already considered equivalent. Failover and retry are separate failure responses.

Published 2026-08-15 · Updated 2026-08-15 · KeepRouter Editorial · 8 minute read

Decision diagram separating model routing from endpoint load balancing — Routing chooses an eligible model path; load balancing distributes traffic inside an equivalent pool.

Short answer: LLM routing chooses a target because of the request, task, tenant, or policy. Load balancing distributes requests among targets that have already been declared interchangeable for that workload. Failover moves to another eligible target after a failure. A retry repeats an attempt. Combining these words into “smart routing” hides decisions that need different safety rules.

The four-policy map

Policy	Trigger	Necessary precondition	Main risk	Evidence to retain
Routing	A request or policy attribute	A deterministic rule and an eligible target set	Silent quality or capability mismatch	Rule version, matched condition, selected target
Load balancing	Capacity or distribution objective	Targets are equivalent for this contract	Different outputs despite assumed equivalence	Candidate set, chosen target, distribution state
Failover	A classified failure	A second target can safely continue the operation	Duplicate side effects or semantic drift	First failure, eligibility reason, fallback target
Retry	A retryable failure or transport interruption	The operation is idempotent or has a deduplication key	Duplicate charge, tool action, or latency amplification	Attempt count, error class, backoff, final outcome

Cloudflare's dynamic routing documentation describes rules based on request properties and percentage rollouts. Portkey documents conditional routing and fallback behavior. OpenRouter's provider routing documentation lets callers influence provider order, fallbacks, parameter support, and data policies. LiteLLM documents reliability policies across deployments. Those are useful primitives, but none of them proves that two different models are interchangeable for your prompt, tool schema, or safety boundary.

Define eligibility before algorithms

Start with a capability predicate, not a vendor list. An eligible target might need to support a specific endpoint, tool calls, a structured-output schema, a minimum context requirement, a data policy, a region, and an approved model snapshot. Only after the predicate is satisfied should you apply a cost, latency, capacity, or rollout policy.

A compact policy can be reviewed like this:

eligible = endpoint_ok
        && tool_schema_ok
        && data_policy_ok
        && model_is_approved

target = route_by_task(eligible)
target = balance_within_equivalent_deployments(target)
result = retry_if_idempotent(target)
result = fail_over_only_to_preapproved_targets(result)

For explicit model selection through KeepRouter, the model routing feature and model catalog describe the public surface. Check API observability for the evidence needed to reconstruct a decision, and use error documentation to decide which failures are retryable. Do not infer that a catalog entry is an automatic fallback candidate.

An implementation checklist

[ ] Give every routing rule an owner, version, test corpus, and rollback path.
[ ] Separate capability filters from optimization scores.
[ ] Pin model IDs where output stability matters; test aliases as mutable configuration.
[ ] Make tool calls and other side effects idempotent before retrying.
[ ] Cap total attempts across retries and fallbacks, not only per target.
[ ] Preserve the original error and every attempted target in the request record.
[ ] Evaluate success quality as well as HTTP status.
[ ] Test brownouts, partial streams, cancellations, and an exhausted fallback chain.

Boundary: a successful response is not equivalence

Two targets can both return valid JSON and still differ in instruction following, tool arguments, safety behavior, token accounting, or response time. Route only across differences your application is designed to tolerate. For broader architecture, read the AI gateway guide; for a proof harness, use how to evaluate an AI gateway.

Frequently asked questions

Is failover a form of load balancing?

No. Load balancing distributes normal traffic among eligible equivalents; failover changes target because an attempt failed or became ineligible.

Can different model families be automatic fallbacks?

Only after application-specific evaluation shows that the fallback satisfies the same endpoint, tool, data, and quality contract. HTTP success alone is insufficient.

Sources reviewed

Article last reviewed 2026-08-15