Engineering answer

What is LLM routing?

LLM routing is the policy that chooses which approved model and serving path receives a request. A route can be fixed by model ID, selected by task rules, or chosen from several eligible providers. Good routing is bounded: it defines allowed candidates, retry conditions, cost limits, and evidence for the final choice instead of sending traffic to any available model.

Last reviewed 2026-08-26 · Editorial review: KeepRouter Editorial

Four decisions that are often called routing

Decision	What changes	What must stay controlled
Model selection	The public model ID changes	Task quality, tools, output format and cost ceiling
Provider selection	The serving provider changes for the same model family	Data policy, region, model revision and response behavior
Load balancing	Traffic is distributed across equivalent eligible routes	Health criteria, weights and capacity
Failover	A failed or unavailable route is replaced	Retry budget, idempotency and approved fallback set

These decisions belong to different owners. Product teams usually own acceptable model behavior. Platform teams own route health and credentials. Finance or product operations may own spend limits. Combining them in one opaque score makes incidents hard to explain.

Eligibility comes before optimization

Request context -> Capability gate -> Policy limits -> Ranked routes -> Selected route -> Request evidence

A route must pass the capability gate before cost, quality, or latency can rank it. Check the endpoint, tool support, input and output modalities, context size, data policy, region, and evaluation status first. An ineligible model should never become the winner simply because it is cheap or fast.

This example is policy pseudoconfiguration, not a KeepRouter API schema:

{
  "endpoint": "chat.completions",
  "eligibleModels": ["approved-model-a", "approved-model-b"],
  "requirements": {
    "tools": true,
    "region": "approved-region",
    "maxInputTokens": 32000
  },
  "objective": "quality",
  "limits": {
    "maxCostPerRequestUsd": 0.08,
    "maxAttempts": 2
  }
}

Policy objective	Eligibility gate	Ranking signal	Evidence to retain
Quality first	Passed task evaluation and required capabilities	Evaluation score for the same task set	Evaluation version, requested model and final model
Cost first	Passed the minimum quality and policy threshold	Estimated cost under the same token assumptions	Token usage, cache status and final charge
Latency first	Passed quality, region and capability checks	Recent comparable p95 latency	Client region, time to first token, retry and provider route

Use measured task outcomes for quality, not a general leaderboard. Use the latency measurement answer for comparable timing and the cost control guide for total request cost.

A routing policy needs a closed candidate set

Start with the exact endpoint your application calls, such as Chat Completions, Responses, Messages, embeddings, or image generation. Build a candidate set from models that support that route and have passed the same evaluation. Then state which dimensions may change automatically.

A policy might allow two providers for one approved model while forbidding a silent switch to a cheaper model family. Another policy might permit a fallback model for summarization but not for code execution. The model routing feature explains KeepRouter's public boundary, while routing versus load balancing covers the operational distinction in more depth.

The request needs an evidence trail

Record the requested model, final model, endpoint, request identifier, status, token usage, charge, latency, and whether a retry or fallback occurred. If provider identity is part of your compliance decision, choose a product that exposes and lets you control it. KeepRouter exposes public model IDs while its upstream mappings remain operator-managed and private.

Before a policy goes live, verify the failure and evidence contract:

No eligible route: return a distinct policy error instead of picking an unapproved model.
Capacity or rate limit: record the rejected route, retry decision, delay, and next route.
Timeout: preserve the attempt number and elapsed time so a slow provider is not mistaken for gateway overhead.
Partial stream: record whether output reached the caller before retrying; do not duplicate side effects.
Final success: retain the requested and selected model, endpoint, usage, charge, latency, fallback flag, and correlation ID.

The API observability feature describes the evidence KeepRouter exposes. The LLM failover answer covers retry boundaries when the first eligible route fails.

A practical rollout sequence

Freeze a representative test set with tools, long inputs, malformed requests, streaming and expected failures.
Approve candidate IDs against the same pass criteria.
Define retryable errors, timeout budgets and the maximum number of attempts.
Stage the policy with a narrow key and a small traffic cohort.
Compare task outcome, failure rate, measured usage and total request cost.
Keep the previous route as a tested rollback until the new policy is routine.

Read how to switch models without changing code before moving the selected ID into deployment configuration. For gateway selection, use the AI gateway evaluation guide rather than a feature count.

Where routing should stop

Do not route a side-effecting agent call through automatic retries unless the action is idempotent or protected by a stable operation key. Do not substitute a model that has not passed tool and safety tests. Do not infer that similarly named models from different providers are the same hosted service. Routing reduces integration work, but it does not remove model evaluation or incident ownership.

Frequently asked questions

Is LLM routing the same as model switching?

Model switching is one routing decision. Routing can also choose providers, balance equivalent routes, or apply failover within an approved set.

Can a router choose any available model?

It should not. Production routing should use a closed set of models that passed the endpoint, tool, quality and cost checks for the task.

Does routing always lower cost?

No. A policy can prefer cost, but retries, longer outputs, cache behavior and quality failures can increase total cost.

Should provider identity be logged?

Yes when provider choice affects compliance, region, debugging or procurement. If the gateway hides that identity, document the resulting boundary.

Can agent calls fail over safely?

Only when side effects are idempotent or protected from duplication, and every fallback model has passed the same tool and behavior tests.