# How to evaluate an AI gateway with a proof-based scorecard

> The best gateway evaluation is a reproducible harness, not a vendor feature count. Score the exact protocols, models, failures, evidence, data controls, and operating duties your system needs.

_Published 2026-08-15 · Updated 2026-08-15 · [KeepRouter Editorial](https://keeprouter.com/editorial-policy#editorial-team) · 10 minute read_

![AI gateway evaluation scorecard covering contract, evidence, failure, security, and operations](https://keeprouter.com/editorial/blog/evaluate-ai-gateway.png)

_A useful evaluation closes each required workload with evidence, an owner, and a rollback condition._

**Short answer:** evaluate an AI gateway by sending a fixed corpus of your own production-shaped requests through it and requiring pass/fail artifacts for protocol fidelity, model eligibility, reliability, observability, cost attribution, data controls, and operations. A public feature matrix is useful for discovery, but it is not acceptance evidence.

Start by choosing one bounded workflow: for example, a server-side text endpoint with streaming and one idempotent tool. Remove secrets from captured requests, define structural and quality assertions, and keep the previous route available. Expand only after this slice passes.

## Evidence scorecard

| Dimension | Required artifact | Example failure condition |
| --- | --- | --- |
| Protocol | Request/response fixtures for every consumed field | A field is dropped, renamed, or silently coerced |
| Model eligibility | Current catalog or API result plus scoped smoke evidence | Route exists but selected model rejects the endpoint |
| Streaming | Ordered event log including cancellation and partial failure | UI hangs or treats an error as successful completion |
| Tools | Full tool-call and tool-result transcript | Arguments change shape or a side effect is duplicated |
| Reliability | Retry/fallback policy and injected-failure results | Non-retryable errors repeat or fallback crosses an unapproved model |
| Observability | Correlated app, gateway, and billing records | A request cannot be traced to owner, model, outcome, and usage |
| Cost control | Key scope, output bound, attempt limit, and alert test | One job can create unbounded attempts or output |
| Data controls | Logging, retention, region, and redaction configuration | Sensitive content appears in logs contrary to policy |
| Operations | Deployment, upgrade, incident, and rollback runbooks | Only one person can restore service |

## Compare operating models, not only APIs

A managed gateway, a BYOK control plane, and a self-hosted proxy can expose similar request shapes while assigning responsibility differently. Cloudflare documents a gateway layer with analytics and traffic controls. Portkey documents gateway configuration and governance capabilities. LiteLLM documents a proxy server that your team can deploy and operate. OpenRouter documents a hosted unified API and provider-routing controls. KeepRouter exposes its public routes through [OpenAPI](/api/openapi.json) and current model eligibility through the [model catalog](/models).

Ask who owns provider accounts, credentials, billing reconciliation, gateway uptime, upgrades, security patches, data retention, model onboarding, and incident response. A deployment that looks inexpensive on a rate card may be expensive to operate; a managed service may reduce operations but require different trust and data review. Record these as responsibilities, not marketing adjectives.

## Run the harness in four stages

1. **Static inspection.** Read OpenAPI, model details, status, security, limits, and error documentation. Record unknowns as unknowns.
2. **Local contract tests.** Point a test client at each candidate using sanitized fixtures. Assert fields, events, tools, usage, and error classes.
3. **Injected failure tests.** Exercise invalid credentials, unsupported fields, rate limits, timeout, cancellation, retry exhaustion, and eligible fallback. Never trigger destructive tools.
4. **Bounded canary.** Route a small, reversible slice with explicit quality, latency, error, and cost gates. Compare against the same baseline workload.

Use [AI gateway fundamentals](/blog/ai-gateway-guide) to define scope, [routing versus load balancing](/blog/llm-routing-vs-load-balancing) to review policy, and the [OpenAI-compatible migration checklist](/blog/openai-compatible-api-migration-checklist) to test client behavior. KeepRouter's [features hub](/features), [status page](/status), and [security page](/security) cover separate layers; none substitutes for a generation-path result.

## Boundary: verification is route-, model-, account-, and time-specific

A passing GET, health page, or OpenAPI path proves structure or availability at that moment. It does not prove that your key can call a selected model, that a tool round trip works, or that a fallback preserves quality. Conversely, one failed request may indicate account scope or payload error rather than global outage. Label every artifact with timestamp, route, model, key scope, build when available, and test case so reviewers know exactly what it proves.

## Frequently asked questions

### Can a feature matrix choose an AI gateway?

It can shortlist candidates. Acceptance should come from reproducible tests of your exact contract, workload, data policy, and operating responsibilities.

### Does a health check prove the generation route works?

No. It proves only the health surface at that time. A scoped generation test is needed for model eligibility, authentication, protocol, and usage evidence.

## Sources reviewed

_Article last reviewed 2026-08-15_

1. [Cloudflare AI Gateway documentation](https://developers.cloudflare.com/ai-gateway/)
2. [Portkey AI Gateway documentation](https://portkey.ai/docs/product/ai-gateway)
3. [LiteLLM documentation](https://docs.litellm.ai/)
4. [OpenRouter quickstart](https://openrouter.ai/docs/quickstart)
5. [KeepRouter OpenAPI](https://keeprouter.com/api/openapi.json)

## Related guides

- [A practical multi-model API, from first call to production](https://keeprouter.com/features.md)
- [Compare AI gateway operating models, not slogans](https://keeprouter.com/compare.md)
- [AI gateway guide: what it controls, when to use one, and how to adopt it](https://keeprouter.com/blog/ai-gateway-guide.md)
- [OpenAI-compatible API migration checklist](https://keeprouter.com/blog/openai-compatible-api-migration-checklist.md)
- [Kimi K3 on KeepRouter: working with a 1M-token context window](https://keeprouter.com/blog/kimi-k3-1m-context.md)

[All posts](https://keeprouter.com/blog.md) · [Models & pricing](https://keeprouter.com/models.md)