Architecture
Admission before inference
obleth is the policy layer between your load balancer and model backends. Clients hit an OpenAI-compatible proxy on :8080; operators configure tenants and quotas through a Management API on :9180. Under the hood it is one Rust binary — but the split that matters is who sends work versus who executes it.
Admission
obleth owns
Who can send work, how much, and in what order — before requests reach your models
Routing
Upstream owns
Which GPU pod serves a request stays with Aibrix, vLLM, or your inference stack
Dashboard
How you run it
Create tenants, rotate keys, tune weights, and watch live slots from the control plane
Instant
Priority changes
Bump a tenant's weight during an incident — every gateway pod picks it up on the next request
System map
Control plane & data plane
- Management API — tenants, keys, routes
- Control-plane dashboard
- Postgres as configuration source of truth
- OpenAI-compatible proxy (in-cluster)
- Fairshare admission + token budgets
- Prometheus /metrics on each pod
Three datastores
Postgres
Config & audit
Redis
Hot path — keys, cache, budgets
ClickHouse
Usage ledger (optional OLAP)
Three datastores
Each chosen for one job
Postgres
Config source of truth
Tenants, keys, model routes, MCP registrations, quotas, audit log. Never on the request hot path.
Redis
Hot path only
Key/model resolution, Lua-atomic token budgets, response cache, and pub/sub invalidation across pods.
ClickHouse
Append-only ledger
Per-request usage rows with admission class, token counts, and queue wait. Daily rollups kept forever.
Data plane
The eight-step path
Key resolution hits moka (5 min TTL, 100k cap) then Redis before Postgres. Token budgets reserve at admission and reconcile when the stream completes.
hop
Clients
OpenAI-compatible API
hop
Load balancer
TLS · round-robin
gateway
obleth
data plane
hop
Backends
vLLM · Aibrix
Inside the data plane
Request lifecycle
Auth through telemetry on every request. Cache hits return before fairshare — no slot acquired, no upstream call.
Read moreModular deploy
Bundled Postgres/Redis or external URLs. Enable ClickHouse and observability profiles only when you need them.
Read moreCapacity autotune
Run a load probe from the dashboard, review the recommended concurrency cap, and apply it when ready.
Read moreAuto model routing
Send `model: "auto"` and obleth picks a backend by capacity, cost, and routing tags — with an optional intent classifier.
Read moreAccess windows & budgets
Time-box tenant access with weekly windows, and cap cumulative spend with lifetime or monthly term budgets.
Read moreNext · Performance
Caching, fairshare, metrics
Exact-match response cache, weighted admission queue, Moka + Redis resolution tiers, and Prometheus histograms on :9091.