Engineering answer

Does an AI gateway add latency?

Yes. An AI gateway adds at least a network and processing hop, so its own overhead cannot be zero. Whether users notice it depends on geography, connection reuse, request size, gateway work, provider time to first token, output length, retries and streaming. Measure the full request path with the same payload and report gateway overhead separately from provider generation time.

Last reviewed 2026-08-15 · Editorial review: KeepRouter Editorial

Latency has several clocks

Measurement	Starts	Stops	What it reveals
Client round trip	Before the client sends	After the client receives the terminal response	Total user-visible time
Time to first byte	Before the request	First response byte	Network, admission and upstream start
Time to first token	Before the request	First usable model output	Routing plus provider queue and generation start
Stream duration	First token	Terminal event	Output generation and delivery
Gateway processing	Gateway receive	Upstream send, then upstream receive to client send	Work attributable to the gateway

Do not compare one product's internal processing metric with another product's public round trip. They measure different intervals. Vendor latency claims may also use different regions, payloads, connection states and percentile windows.

A fair measurement design

Run direct and gateway paths from the same client region with the same model revision, provider, prompt, output cap and streaming mode. Reuse connections for warm tests, but also record cold starts if your application creates new connections. Capture at least p50, p95 and p99 over enough requests to see queueing and rate-limit behavior.

Separate successful first attempts from retries. A fallback that rescues a failed call can improve task completion while making that individual request slower. Blending rescued calls into a single average hides the trade-off. The failover answer explains how to bound those attempts.

Gateway work that can add time

Authentication, quota checks, request transformation, policy evaluation, logging, cache lookup, provider selection and response normalization all consume time. Large synchronous log writes or a remote policy database can become part of the critical path. Streaming transformations can also delay the first usable event if the gateway buffers too much data.

Some gateway features can reduce total time. A nearby edge entrypoint can improve connection setup. A valid cache hit can avoid generation. Provider health routing can bypass a degraded path. These are workload-dependent outcomes, not reasons to describe gateway overhead as zero.

Set a product budget

Define an acceptable increase for time to first token and total completion time based on the user interaction. A coding agent, autocomplete field and background batch have different budgets. Monitor the same metrics after deployment because route selection and upstream behavior change.

Keep the test script, raw timestamps, model ID, endpoint and gateway configuration with the result. Re-run after changing routing, logging or retry policy. The observability feature lists the request evidence available in KeepRouter; use the evaluation framework to compare it with another product.

Frequently asked questions

How much latency does a gateway add?

There is no universal number. Region, connection reuse, policy work, logging, payload size and upstream routing all affect the result.

Should I measure time to first token or total time?

Measure both. Time to first token describes responsiveness, while total time includes output generation and delivery.

Can failover improve latency?

It can avoid a degraded route, but a failed first attempt usually makes the rescued request slower. Report those cases separately.

Does streaming remove gateway overhead?

No. Streaming changes when output becomes visible, but routing and processing still happen before and during the stream.

Can caching make the gateway faster than direct generation?

A valid cache hit can, but only for cacheable requests under the configured policy. Compare hit and miss paths separately.