Architecture
The processes obleth runs, the three listeners of the gateway, where state lives, and how reads and writes move between them.
2 min readUpdated for v2.4.0Edit on GitHub
The gateway is one Rust binary, obleth. It opens three listeners and connects to Postgres, Redis and ClickHouse at start. The dashboard is a separate Next.js app that talks only to the gateway's management API.
The pieces
The data plane forwards requests to your model servers or an inference gateway.
The three listeners
| Port | Serves | Authenticated with |
|---|---|---|
| 8080 | The data plane: OpenAI routes, the Responses and Anthropic Messages APIs (translated to chat), Videos, Verdicts, the MCP gateway, and model discovery (/v1/models, /model/info) | Tenant keys (sk_…, as Authorization: Bearer or x-api-key), or identity-provider JWTs when OBLETH_JWT_ISSUERS is set |
| 9180 | The management API under /api/v1: tenants, keys, models, settings, usage, the live fairshare state | The admin token |
| 9091 | Prometheus /metrics, with low-cardinality labels only | None |
Per-tenant and per-key breakdowns come from ClickHouse, not Prometheus labels. Routes are listed in Data-plane routes and Management API.
Other processes
- Dashboard (
control-plane): reads and writes gateway state through the management API only. Its own sign-in tables live in the same Postgres database. - Slurm provisioner (
obleth-provisioner, optional): a single reconciler that launches and watches models on Slurm through the management API. See Slurm provisioning. - Compression sidecar (optional): a CPU-only scorer the compression boon calls. The gateway falls back to heuristics when it is absent.
- Benchmark backend (optional): a simulated model server for demos and load tests.
Where state lives
| Store | Holds | On the request path |
|---|---|---|
| Postgres | The source of truth for configuration and audit | Never |
| Redis | A cache of keys, models and MCP servers; token buckets and term budgets; response cache; shared fairshare slots | Yes |
| ClickHouse | One row per request, and trace spans | Written in the background |
Everything in Redis can be rebuilt from Postgres except live bucket and reservation state. See Datastores.
Reads: two caches, then 401
Models and MCP servers resolve the same way. A Redis error during key lookup returns 503 authentication backend unavailable, never 401, so an outage doesn't look like a revoked key. Recently resolved keys keep working from the in-process cache while Redis is down.
The gateway fills Redis from Postgres at start, re-pushes it every OBLETH_REDIS_REWARM_SECS (default 300) and after every Redis reconnect, and prunes entries that no longer exist in Postgres. POST /api/v1/resync (Settings › Data and backups › Reconcile cache) does the same on demand.
Writes: one path
Every configuration change takes this path; nothing writes to Redis directly. The invalidation message makes every gateway replica drop its in-process copy, so a new weight or a disabled key applies on the next request without a restart. If removing a key, tenant, model or MCP server from Redis fails, the write returns 502 rather than leaving the old entry serving.
Replicas
Gateway replicas are stateless apart from their in-process caches and admission queues, so you can run several behind a load balancer. Each replica runs its own fairshare scheduler; with more than one live, they share pool slots and caps through Redis so the configured limits hold for the fleet. See Fairshare engine.
The steps one request goes through are in Request lifecycle.