Engineering answer

What is LLM routing?

LLM routing is the policy that chooses which approved model and serving path receives a request. A route can be fixed by model ID, selected by task rules, or chosen from several eligible providers. Good routing is bounded: it defines allowed candidates, retry conditions, cost limits, and evidence for the final choice instead of sending traffic to any available model.

Last reviewed 2026-08-15 · Editorial review: KeepRouter Editorial

Four decisions that are often called routing

Decision	What changes	What must stay controlled
Model selection	The public model ID changes	Task quality, tools, output format and cost ceiling
Provider selection	The serving provider changes for the same model family	Data policy, region, model revision and response behavior
Load balancing	Traffic is distributed across equivalent eligible routes	Health criteria, weights and capacity
Failover	A failed or unavailable route is replaced	Retry budget, idempotency and approved fallback set

These decisions belong to different owners. Product teams usually own acceptable model behavior. Platform teams own route health and credentials. Finance or product operations may own spend limits. Combining them in one opaque score makes incidents hard to explain.

A routing policy needs a closed candidate set

Start with the exact endpoint your application calls, such as Chat Completions, Responses, Messages, embeddings, or image generation. Build a candidate set from models that support that route and have passed the same evaluation. Then state which dimensions may change automatically.

A policy might allow two providers for one approved model while forbidding a silent switch to a cheaper model family. Another policy might permit a fallback model for summarization but not for code execution. The model routing feature explains KeepRouter's public boundary, while routing versus load balancing covers the operational distinction in more depth.

The request needs an evidence trail

Record the requested model, final model, endpoint, request identifier, status, token usage, charge, latency, and whether a retry or fallback occurred. If provider identity is part of your compliance decision, choose a product that exposes and lets you control it. KeepRouter exposes public model IDs while its upstream mappings remain operator-managed and private.

A practical rollout sequence

Freeze a representative test set with tools, long inputs, malformed requests, streaming and expected failures.
Approve candidate IDs against the same pass criteria.
Define retryable errors, timeout budgets and the maximum number of attempts.
Stage the policy with a narrow key and a small traffic cohort.
Compare task outcome, failure rate, measured usage and total request cost.
Keep the previous route as a tested rollback until the new policy is routine.

Read how to switch models without changing code before moving the selected ID into deployment configuration. For gateway selection, use the AI gateway evaluation guide rather than a feature count.

Where routing should stop

Do not route a side-effecting agent call through automatic retries unless the action is idempotent or protected by a stable operation key. Do not substitute a model that has not passed tool and safety tests. Do not infer that similarly named models from different providers are the same hosted service. Routing reduces integration work, but it does not remove model evaluation or incident ownership.

Frequently asked questions

Is LLM routing the same as model switching?

Model switching is one routing decision. Routing can also choose providers, balance equivalent routes, or apply failover within an approved set.

Can a router choose any available model?

It should not. Production routing should use a closed set of models that passed the endpoint, tool, quality and cost checks for the task.

Does routing always lower cost?

No. A policy can prefer cost, but retries, longer outputs, cache behavior and quality failures can increase total cost.

Should provider identity be logged?

Yes when provider choice affects compliance, region, debugging or procurement. If the gateway hides that identity, document the resulting boundary.

Can agent calls fail over safely?

Only when side effects are idempotent or protected from duplication, and every fallback model has passed the same tool and behavior tests.