Engineering answer
What is the difference between an AI gateway and an API gateway?
An API gateway manages general service traffic through authentication, routing, rate limits, policy and observability. An AI gateway applies similar controls to model calls, then adds model-aware concerns such as canonical model IDs, provider selection, token usage, streaming events, tool-call continuity, multimodal payloads and AI-specific failover. Some products combine both layers; others place a focused AI gateway behind an existing API gateway.
Last reviewed 2026-08-26 · Editorial review: KeepRouter Editorial
The overlap is real, but the traffic is different
| Concern | General API gateway | AI gateway |
|---|---|---|
| Caller authentication | API keys, OAuth, mTLS or signed requests | Gateway keys plus model and endpoint scope |
| Routing target | Service, version, region or upstream | Model ID, provider route, endpoint family or fallback |
| Usage unit | Requests, bytes and duration | Requests plus input, output, cached tokens or generated assets |
| Streaming | Generic HTTP or event streams | Model-specific SSE events, partial output and terminal usage |
| Policy | Rate, identity, network and schema rules | Model allowlists, spend limits, prompt or response controls |
| Failure handling | Retry service instances | Retry or fail over only within model and side-effect boundaries |
A standard gateway can proxy model HTTP traffic, but it does not automatically understand token accounting, tool-call state, model capability, provider-specific errors or the difference between Chat Completions and Responses. An AI gateway can implement those concerns, although the exact feature set varies by product.
Three common architectures
- Focused AI gateway only. A small team points its SDK at a managed model gateway and lets that service own model access and billing.
- API gateway in front of an AI gateway. The outer layer handles company identity, network policy and tenant admission. The inner layer handles models, tokens and provider routes.
- AI plugins inside the existing gateway. Platform teams extend Kong, Envoy or another gateway with model-aware plugins and keep the data plane inside their infrastructure.
The right arrangement depends on who already owns traffic policy. If an API platform team has mature identity and network controls, replacing them with a separate public entrypoint may create duplicate policy. If a small product team has no gateway platform, operating one solely for model calls can be unnecessary work. See when an AI gateway is useful before adding another service.
Follow the request through both layers
Client -> API Gateway -> Application -> AI Gateway -> Model provider
| Hop | Primary responsibility | Identifier that should continue |
|---|---|---|
| Client to API Gateway | User session, tenant admission and public rate policy | Trace ID or request ID |
| API Gateway to application | Service routing, identity claims and network policy | Trace ID plus tenant ID |
| Application to AI Gateway | Task context, approved model and product limits | Trace ID plus AI request ID |
| AI Gateway to provider | Model route, provider credential, usage and fallback | AI request ID plus provider request ID |
The application remains a useful boundary between the two gateways. It can translate an authenticated user request into a narrowly scoped model request without exposing the model gateway key to the client.
Deployment choice matrix
| Starting point | Usually simplest | Check before shipping |
|---|---|---|
| No existing gateway platform | Use one focused AI gateway for server-side model calls | Key scope, endpoint support, logs, spend limits and rollback |
| Mature API gateway and platform team | Keep the API gateway at the edge and place the AI gateway behind the application | Timeout, streaming, body limits, identity propagation and duplicate limits |
| Required private data plane or gateway standard | Add model-aware plugins or self-host the AI layer | Provider compatibility, operational ownership, upgrades and evidence export |
| One provider and one stable model | Start with the provider SDK, then add a gateway when a specific control is needed | Avoid an extra layer without a measured requirement |
A short decision tree is enough: if enterprise identity and network policy already live at the edge, keep them there. If model routing, usage, or failover needs a separate owner, add the AI layer behind the application. If neither condition applies, document the requirement before adding another hop. The managed versus self-hosted comparison helps resolve the data-plane choice.
Questions to settle before combining the layers
Write down which layer authenticates the end user, which key reaches the model gateway, where tenant and feature identifiers are attached, and which system is authoritative for rate and spend limits. Decide where prompt content may be logged and how request IDs cross both layers. Two dashboards with different request IDs will slow incident response.
Also verify timeouts. A general API gateway may close a connection before a long model response or streaming session finishes. Request and response size limits can reject image blocks or long contexts even when the model route accepts them. The AI gateway security answer covers trust boundaries, while the observability feature describes the KeepRouter evidence surface.
Correlate one request, not two dashboards
| Layer | Example record |
|---|---|
| API Gateway | trace_id=req_7f2, tenant_id=team_42, status=200 |
| Application | trace_id=req_7f2, ai_request_id=ai_91c, feature=answer |
| AI Gateway | ai_request_id=ai_91c, model=approved-model-a, fallback=false, usage_total_tokens=1842 |
| Provider adapter | ai_request_id=ai_91c, provider_request_id=provider_abc, attempt=1 |
Propagate identifiers in headers or structured context, then record them as fields rather than embedding them only in free-form messages. Do not put prompts, secrets, or raw authorization headers into correlation fields. During a test, begin with the client trace ID and confirm that you can find the final model, usage, charge, latency, and provider result. Use the gateway latency answer to separate the two gateway hops from model generation time.
Keep the contract narrow
Do not advertise a single universal AI endpoint if the implementation supports only one operation. Document Chat Completions, Responses, Messages, embeddings, images and audio separately. A useful gateway makes those boundaries easier to inspect. It should not hide them behind a generic proxy label.
Frequently asked questions
Can an ordinary API gateway proxy LLM calls?
Yes, but generic proxying does not add model catalogs, token accounting, provider-aware routing or model-specific stream handling by itself.
Do I need both gateways?
Only if the layers have distinct owners and policies. Many small teams need one focused layer; larger platforms may keep enterprise traffic controls in front.
Which layer should enforce spend limits?
Choose one authoritative layer and reconcile its model usage evidence. Duplicate independent limits can produce confusing failures.
Can I put an AI gateway behind Kong or Cloudflare?
Often yes, subject to timeout, streaming, body-size, identity and logging configuration on both layers.
Is an AI gateway only for LLM text?
No. Some gateways support images, embeddings, audio and other modalities, but each operation needs an explicit route and capability check.