Performance
What the gateway does on each request, what it keeps off that path, and how to measure it.
2 min readUpdated for v2.4.0Edit on GitHub
obench score grades gateway overhead against a GPU-free backend.What runs on every request
Keys, models and MCP servers resolve from an in-process cache first and Redis second. The gateway warms Redis from Postgres at start, re-pushes it on a timer and after every Redis reconnect, and invalidates the in-process copies over Redis pub/sub, so a change made in the dashboard reaches every replica without a restart.
Admission is the fairshare scheduler in the same process. With several gateway replicas, the limits are cluster-wide slots in Redis: one script call when a request is admitted and one when it finishes. With a single replica there is no Redis call on this path at all. See Fairshare engine.
What stays off it
- Telemetry. Usage rows and spans are written in the background. While ClickHouse is unreachable they spill to a local write-ahead file and replay when it returns. See Datastores.
- Tracing. Spans export to an OTLP collector through a background batch processor, so a slow collector never slows a request.
- Per-tenant metrics. Prometheus labels stay low-cardinality (admission class, status class). Per-tenant and per-key breakdowns come from ClickHouse instead.
Streaming
Response chunks go to the client as they arrive. The fairshare slot is held until the stream ends, so the gateway's count of requests in flight matches what the backend is actually serving, not how long a connection happened to be open.
Cached responses
A request that hits the response cache is answered by the gateway and never reaches the backend. Caching is set per model.
Measure it yourself
obench score runs capacity ramps, gateway overhead, streaming quality, overload, resilience and fairshare sections against the benchmark backend in the Compose stack, and grades each one. Scores are kept as baselines so later runs show regressions. See Benchmarks.
Sizing a pool from a load probe
For a self-hosted model, the Models page can run a short load probe against the upstream and recommend a concurrency cap from the latency curve. Nothing changes until you apply it. See Capacity auto-tune. For a model on Kubernetes, discovered capacity sizes the pool from its ready replicas instead.