Request Lifecycle

Every request that arrives at the obleth data plane runs through the steps below in order. Understanding this pipeline explains how fairshare and cost accounting work together.

Pipeline overview

① Auth
② Parse body + resolve model (auto-routing, allowlist, boons)
③ Response cache check  ──── hit ──→ return immediately (no permit, no budget)
④ Fairshare admission   ──── at capacity ──→ queue until a slot opens
⑤ Budget reserve + term gate (one atomic Redis call) ──→ 429 / 403
⑥ Proxy upstream
⑦ Reconcile actual cost
⑧ Emit telemetry

Step 1 — Auth

obleth extracts the bearer token from either the Authorization: Bearer <token> header or the x-api-key: <token> header.

The raw token is never stored. It is immediately hashed with SHA-256 and the hash is looked up:

1. moka in-process cache (TTL=5 min, cap=100k keys) — fastest, no network hop.
2. Redis (obleth:key:{hash}) — shared across all gateway pods.
3. If neither cache has it, the key doesn't exist → 401 invalid api key.

The cache returns a ResolvedKey containing everything admission needs: tenant_id, weight, tokens_per_minute, max_in_flight, fairshare_group, group_weight, disabled.

If disabled is true, the request is rejected with 403.

Step 2 — Parse body and resolve model

obleth reads and parses the request body (limit: 64 MiB). It extracts the model field and looks it up in the model registry (same cache chain: moka → Redis).

For paths that require a registered model (/v1/chat/completions, /v1/completions, etc.):

• If model is missing → 400 model is required
• If the model isn't registered → 404 model not registered
• If the model is enabled: false → 403 model is disabled

Models carry an admission_weight (default 100, a percentage) that scales the tenant's weight for this request: effective_weight = round(tenant.weight * admission_weight / 100), minimum 1.

Before the cache check, obleth also applies the tenant's model allowlist and any model boons granted to the route, and estimates the token cost — see Token-measured Fairness.

Step 3 — Response cache check

If the matched model has cache_enabled: true, obleth computes a cache key from sha256(model_name + request_body) and checks Redis (obleth:cache:{key}).

A cache hit returns the stored response immediately and exits the pipeline. No fairshare permit is acquired, no budget is consumed, and the upstream is never called. The usage record is written with cache_status = "hit".

A cache miss or cache off continues to step 4. A Redis failure here is treated as a miss.

The cache is skipped entirely for two request shapes, regardless of the model's setting: a gateway tool loop (a cached answer would replay stale tool results) and a request whose tenant has output guardrails armed (the cache is shared across tenants, so serving or populating it would bypass that tenant's own scanning).

Step 4 — Fairshare admission

obleth calls the fairshare scheduler with:

• tenant_id + weight (from the resolved key)
• group + group_weight (for hierarchical mode)
• cost (estimated token count)

The scheduler holds a global concurrency semaphore (OBLETH_GLOBAL_MAX_IN_FLIGHT, default 256). If a permit is available, the request is admitted immediately (Admission::Fast).

If the cluster is at capacity, the request joins a per-tenant queue. When a slot opens, the scheduler grants it to the tenant most behind its weighted fair share and admits it with Admission::Queued — see Fairshare Engine. A queued request keeps its place until a permit frees up; there is no timeout-based degradation. See Saturation Behavior for what happens when demand exceeds capacity.

Step 5 — Budget reserve and term gate

Before proxying, one atomic Redis Lua script both evaluates the tenant's cumulative term budget and reserves the estimated token cost from its per-minute token bucket. The term gate runs first inside the script, so a term-exhausted request never reserves per-minute tokens it would have no completion path to refund.

• Term budget (budget_tokens and/or budget_cost_usd over a lifetime, monthly, or term period, tracked in obleth:term_usage:{tenant}): if either cap is already met, the permit is released, the request is finalized with Admission::Rejected, and the client receives 403 tenant term budget exhausted.
• Token bucket (refills at tokens_per_minute / 60000 tokens per millisecond; 0 means no per-minute cap): if the bucket is short, the permit is released, the request is finalized Admission::Rejected, and the client receives 429 token budget exceeded.

An API key can carry its own term budget as well. That gate runs as a separate call ahead of the tenant's and returns 403 api key term budget exhausted.

If OBLETH_FAIL_OPEN=true (default) and the Redis call fails, obleth logs a warning, raises an alert, and continues — the budget check is skipped rather than rejecting the request. With fail-open disabled the request is rejected with 503. See Quotas & Rate Limits.

Step 6 — Proxy upstream

obleth forwards the request to the upstream (Aibrix, vLLM, or a registered model's api_base) using a pooled reqwest HTTP client. Streaming (SSE) responses are streamed through to the client byte-for-byte.

Each attempt is bounded by a timeout (request_timeout_secs, or the global OBLETH_UPSTREAM_TIMEOUT_SECS default). On a transient failure — a connection error, a timeout, or HTTP 408/429/5xx — obleth retries the same endpoint up to max_retries times with exponential backoff, then fails over to the model's next endpoint if one is configured. Retries and failover only happen before the first response byte reaches the client; client errors (4xx) are returned immediately and never retried. A single permit and budget reservation cover the whole attempt sequence. See Reliability & Failover.

The fairshare permit is held for the entire duration of the upstream call, including streaming time. This is important: concurrency accounting reflects real GPU occupancy, not just the time to the first byte.

On upstream success, if the model has cache_enabled and the response is ≤ 512 KiB and status 200, the response body is written to Redis with the configured TTL.

Step 7 — Reconcile cost

After the stream finishes, obleth reads the actual token counts from the upstream's usage field (or counts them from the SSE stream). It runs a second Lua script to reconcile:

reconcile = estimated_tokens - actual_tokens

If the estimate was high, tokens are refunded to the bucket. If the actual cost was higher, additional tokens are charged. This ensures billing accuracy regardless of estimation error.

Step 8 — Emit telemetry

obleth sends a UsageRecord to the telemetry channel (a non-blocking async mpsc::send). The hot path returns immediately. A background task batches records and inserts them into ClickHouse every second.

Each record carries a completion timestamp (ts_ms), token counts, latency (queue_wait_ms, ttft_ms, total_ms), status code, frozen cost_usd, frozen energy figures (energy_wh, energy_cost_usd, co2_g), the coarse request_type (derived from the API path), and an optional session_id (captured from client headers or body fields). Rejected requests (429 budget, 403 term budget, upstream errors) are logged too — not only successes, and a client that disconnects mid-stream is recorded with status 499.

Requests from a tenant flagged synthetic are stamped request_type = "benchmark" instead of the path-derived class, and gateway health probes are stamped health_probe. Usage and cost reads exclude both by default; pass include_internal=true to opt back in. Benchmark traffic is also excluded from the permanent usage_daily rollup.

If ClickHouse is unavailable and fail_open is set, records spill to a bounded local WAL and are replayed once ClickHouse recovers — see Reliability & Fail-open.

Individual rows are queryable via GET /api/v1/usage/logs and the dashboard Request Logs page. They are retained for the configured window (default 180 days) before day-partition pruning; the permanent usage_daily rollup is kept forever. See ClickHouse Usage.

When tracing_enabled is set on the key or tenant, the proxy also emits a set of span records via the same non-blocking sink — one per pipeline phase (auth_resolve, admission, cache_lookup, upstream, boon:tool_loop, etc.) plus a root proxy_request span. Spans land in a separate ClickHouse spans table and are surfaced in the dashboard's Request Detail panel. See Per-request span tracing.

The permit (fairshare slot) is dropped after reconciliation, freeing capacity for the next queued request.

Error summary