Engineering answer
What is the difference between an AI gateway and an API gateway?
An API gateway manages general service traffic through authentication, routing, rate limits, policy and observability. An AI gateway applies similar controls to model calls, then adds model-aware concerns such as canonical model IDs, provider selection, token usage, streaming events, tool-call continuity, multimodal payloads and AI-specific failover. Some products combine both layers; others place a focused AI gateway behind an existing API gateway.
Last reviewed 2026-08-15 · Editorial review: KeepRouter Editorial
The overlap is real, but the traffic is different
| Concern | General API gateway | AI gateway |
|---|---|---|
| Caller authentication | API keys, OAuth, mTLS or signed requests | Gateway keys plus model and endpoint scope |
| Routing target | Service, version, region or upstream | Model ID, provider route, endpoint family or fallback |
| Usage unit | Requests, bytes and duration | Requests plus input, output, cached tokens or generated assets |
| Streaming | Generic HTTP or event streams | Model-specific SSE events, partial output and terminal usage |
| Policy | Rate, identity, network and schema rules | Model allowlists, spend limits, prompt or response controls |
| Failure handling | Retry service instances | Retry or fail over only within model and side-effect boundaries |
A standard gateway can proxy model HTTP traffic, but it does not automatically understand token accounting, tool-call state, model capability, provider-specific errors or the difference between Chat Completions and Responses. An AI gateway can implement those concerns, although the exact feature set varies by product.
Three common architectures
- Focused AI gateway only. A small team points its SDK at a managed model gateway and lets that service own model access and billing.
- API gateway in front of an AI gateway. The outer layer handles company identity, network policy and tenant admission. The inner layer handles models, tokens and provider routes.
- AI plugins inside the existing gateway. Platform teams extend Kong, Envoy or another gateway with model-aware plugins and keep the data plane inside their infrastructure.
The right arrangement depends on who already owns traffic policy. If an API platform team has mature identity and network controls, replacing them with a separate public entrypoint may create duplicate policy. If a small product team has no gateway platform, operating one solely for model calls can be unnecessary work. See when an AI gateway is useful before adding another service.
Questions to settle before combining the layers
Write down which layer authenticates the end user, which key reaches the model gateway, where tenant and feature identifiers are attached, and which system is authoritative for rate and spend limits. Decide where prompt content may be logged and how request IDs cross both layers. Two dashboards with different request IDs will slow incident response.
Also verify timeouts. A general API gateway may close a connection before a long model response or streaming session finishes. Request and response size limits can reject image blocks or long contexts even when the model route accepts them. The AI gateway security answer covers trust boundaries, while the observability feature describes the KeepRouter evidence surface.
Keep the contract narrow
Do not advertise a single universal AI endpoint if the implementation supports only one operation. Document Chat Completions, Responses, Messages, embeddings, images and audio separately. A useful gateway makes those boundaries easier to inspect. It should not hide them behind a generic proxy label.
Frequently asked questions
Can an ordinary API gateway proxy LLM calls?
Yes, but generic proxying does not add model catalogs, token accounting, provider-aware routing or model-specific stream handling by itself.
Do I need both gateways?
Only if the layers have distinct owners and policies. Many small teams need one focused layer; larger platforms may keep enterprise traffic controls in front.
Which layer should enforce spend limits?
Choose one authoritative layer and reconcile its model usage evidence. Duplicate independent limits can produce confusing failures.
Can I put an AI gateway behind Kong or Cloudflare?
Often yes, subject to timeout, streaming, body-size, identity and logging configuration on both layers.
Is an AI gateway only for LLM text?
No. Some gateways support images, embeddings, audio and other modalities, but each operation needs an explicit route and capability check.