# Does an AI gateway add latency?

> Yes. An AI gateway adds at least a network and processing hop, so its own overhead cannot be zero. Whether users notice it depends on geography, connection reuse, request size, gateway work, provider time to first token, output length, retries and streaming. Measure the full request path with the same payload and report gateway overhead separately from provider generation time.

_Last reviewed 2026-08-15 · [Editorial review](https://keeprouter.com/editorial-policy#editorial-team)_

## Latency has several clocks

| Measurement | Starts | Stops | What it reveals |
|---|---|---|---|
| Client round trip | Before the client sends | After the client receives the terminal response | Total user-visible time |
| Time to first byte | Before the request | First response byte | Network, admission and upstream start |
| Time to first token | Before the request | First usable model output | Routing plus provider queue and generation start |
| Stream duration | First token | Terminal event | Output generation and delivery |
| Gateway processing | Gateway receive | Upstream send, then upstream receive to client send | Work attributable to the gateway |

Do not compare one product's internal processing metric with another product's public round trip. They measure different intervals. Vendor latency claims may also use different regions, payloads, connection states and percentile windows.

## A fair measurement design

Run direct and gateway paths from the same client region with the same model revision, provider, prompt, output cap and streaming mode. Reuse connections for warm tests, but also record cold starts if your application creates new connections. Capture at least p50, p95 and p99 over enough requests to see queueing and rate-limit behavior.

Separate successful first attempts from retries. A fallback that rescues a failed call can improve task completion while making that individual request slower. Blending rescued calls into a single average hides the trade-off. The [failover answer](/answers/what-is-llm-api-failover) explains how to bound those attempts.

## Gateway work that can add time

Authentication, quota checks, request transformation, policy evaluation, logging, cache lookup, provider selection and response normalization all consume time. Large synchronous log writes or a remote policy database can become part of the critical path. Streaming transformations can also delay the first usable event if the gateway buffers too much data.

Some gateway features can reduce total time. A nearby edge entrypoint can improve connection setup. A valid cache hit can avoid generation. Provider health routing can bypass a degraded path. These are workload-dependent outcomes, not reasons to describe gateway overhead as zero.

## Set a product budget

Define an acceptable increase for time to first token and total completion time based on the user interaction. A coding agent, autocomplete field and background batch have different budgets. Monitor the same metrics after deployment because route selection and upstream behavior change.

Keep the test script, raw timestamps, model ID, endpoint and gateway configuration with the result. Re-run after changing routing, logging or retry policy. The [observability feature](/features/api-observability) lists the request evidence available in KeepRouter; use [the evaluation framework](/blog/evaluate-ai-gateway) to compare it with another product.

## Frequently asked questions

### How much latency does a gateway add?

There is no universal number. Region, connection reuse, policy work, logging, payload size and upstream routing all affect the result.

### Should I measure time to first token or total time?

Measure both. Time to first token describes responsiveness, while total time includes output generation and delivery.

### Can failover improve latency?

It can avoid a degraded route, but a failed first attempt usually makes the rescued request slower. Report those cases separately.

### Does streaming remove gateway overhead?

No. Streaming changes when output becomes visible, but routing and processing still happen before and during the stream.

### Can caching make the gateway faster than direct generation?

A valid cache hit can, but only for cacheable requests under the configured policy. Compare hit and miss paths separately.

## Sources reviewed

1. [Cloudflare AI Gateway analytics](https://developers.cloudflare.com/ai-gateway/observability/analytics/)
2. [Vercel AI Gateway observability](https://vercel.com/docs/ai-gateway/observability)
3. [OpenTelemetry trace semantic conventions](https://opentelemetry.io/docs/concepts/signals/traces/)

## Related guides

- [API observability](https://keeprouter.com/features/api-observability.md)
- [What is LLM API failover?](https://keeprouter.com/answers/what-is-llm-api-failover.md)
- [How to evaluate an AI gateway with a proof-based scorecard](https://keeprouter.com/blog/evaluate-ai-gateway.md)
- [AI gateway latency guide: measure the hop, the provider and the rescued request](https://keeprouter.com/blog/ai-gateway-latency-guide.md)
- [How do I choose an AI gateway?](https://keeprouter.com/answers/how-to-choose-an-ai-gateway.md)
- [Best AI gateways by use case](https://keeprouter.com/compare/best-ai-gateways.md)

## Test the contract with a real model

Create a narrowly scoped key, select a model from the live catalog, and run the exact request shape your application depends on.

[Create a free key](https://keeprouter.com/login?returnTo=%2Fconsole%2Fkeys%3Fmodel%3Dfree) · [Live models and pricing](https://keeprouter.com/models.md)