Request lifecycle

1. 1Authenticate
2. 2Read the body, resolve the model, apply boons and input guardrails
3. 3Response cache
   hitAnswer now: no slot, no budget, no upstream call
4. 4Fairshare admission
   pool fullWait, up to the admission timeout
5. 5Reserve budgets
   exhausted429 or 403
6. 6Call the upstream (retries, failover, streaming)
7. 7Settle cost and write the usage row

The Responses API (/v1/responses) and the Anthropic Messages API (/v1/messages) are translated to a chat request after step 1, run steps 2 to 7 as chat, and are translated back. A verdicts request runs the steps once for all its questions: one slot and one set of reservations cover the whole fan-out.

1. Authenticate

The credential comes from Authorization: Bearer or x-api-key. A JWT from a configured issuer is verified instead (Identity tokens). A key is hashed and looked up in the in-process cache, then Redis. Not found returns 401; a Redis error returns 503. The result carries the tenant and key ids, weights, group, limits, budgets, model allowlist and guardrails policy.

A disabled key returns 403, and so does a tenant that isn't active or is outside its access window, expiry or weekly schedule.

2. Resolve the model and enrich the request

The body is read (up to 64 MiB, else 413) and the model field is resolved through the same cache chain. An alias resolves to its model's canonical name, which is used from here on. model: "auto" goes to the auto router. Then:

• a missing model returns 400, an unregistered one 404, a disabled one 403, and one outside the tenant's allowlist 403;
• a path the gateway doesn't recognise, with no registered model, returns 404 instead of being forwarded;
• the route's boons rewrite the request (vision, knowledge retrieval, compression, tool injection, structured output, image generation, speculation);
• the tenant's input guardrails scan it, and a blocking match returns 400.

The token cost is estimated from the body the upstream will receive. The x-obleth-boons: off header skips boons for one request, never guardrails.

3. Response cache

When the model has caching on with a TTL above 0, the gateway looks up a hash of the tenant, the model and the body. A hit is returned with x-obleth-cache: hit and ends the request: no slot, no budget and no upstream call, and the usage row says cache_status = "hit". A Redis error counts as a miss. The cache is skipped for gateway tool loops and for tenants with output guardrails. See Response cache.

4. Fairshare admission

The request asks the scheduler for a slot in its model's pool, carrying its tenant, key and group weights and caps and the estimated cost. With a free slot it is admitted at once (fast). Otherwise it waits and is admitted (queued) when the scheduler picks it for a freed slot. After OBLETH_ADMISSION_TIMEOUT_SECS (default 60) it returns 503 timed out waiting for model capacity with Retry-After: 5. A waiter whose client has gone is dropped without charge. See Fairshare engine and How fairshare works.

5. Reserve budgets

One atomic Redis script per scope checks and reserves the estimate:

1. the key's term budget, else 403 api key term budget exhausted;
2. the tenant's term budget (tokens or USD, lifetime, monthly or term), else 403 tenant term budget exhausted;
3. the tenant's per-minute token bucket (tokens_per_minute, 0 for no cap), else 429 token budget exceeded.

A term reservation holds the estimate until the request settles, so concurrent requests can't spend the same headroom; one that never settles expires after 10 minutes. On any refusal the slot is released. If Redis fails here, OBLETH_FAIL_OPEN=true (the default) lets the request through without enforcement and raises an alert; false returns 503. See Quotas and rate limits.

6. Call the upstream

The request goes to a healthy endpoint of the model, or its api_base, or OBLETH_UPSTREAM_BASE_URL. The client's credentials are never forwarded; the model's own api_key is sent as Authorization: Bearer. Each attempt is bounded by the model's request_timeout_secs or OBLETH_UPSTREAM_TIMEOUT_SECS, and any silence from the upstream, the wait for headers included, by OBLETH_UPSTREAM_READ_TIMEOUT_SECS. Connection errors, timeouts, 408, 429 and 5xx are retried with backoff and then fail over to the next endpoint, but only before the first byte reaches the client; other 4xx are returned as they are. See Reliability and failover.

Streams are passed through as they arrive, and the slot is held until the upstream stream ends. A successful response of 512 KiB or less is cached when step 3 applied.

7. Settle cost and record usage

The real token counts come from the upstream's usage (the gateway always asks a streaming upstream for it and strips the extra chunk if the client didn't). The per-minute bucket is refunded or charged the difference from the estimate, and the term reservation is replaced by the actual tokens and cost, once. Failed requests are not billed: an error without usage records zero, and a 502 or 504 after retries refunds the reservations. A stream cut short after output reached the client is recorded as 502 and billed for what streamed: the reported usage, else the text at about 4 characters per token, never the max_tokens estimate. Helper calls made for the request (vision, tool loops, drafts, image generation, structured output) count against the term budgets. See How fairshare works.

A usage record then goes onto a bounded in-memory channel and a background task inserts batches into ClickHouse every OBLETH_TELEMETRY_FLUSH_INTERVAL_MS (5 seconds by default), or as soon as 500 rows are waiting; the request doesn't wait for it. Rejected and failed requests are recorded too, and a client that disconnects mid-stream is recorded as 499. While ClickHouse is unreachable, rows go to a local spill file and replay later. With tracing on for the tenant or key, one span per step lands in the spans table. See Datastores and ClickHouse usage.

Errors by step

Every error string is in Data-plane routes.

Guardrails on non-chat endpoints+

Input guardrails run for every request from a tenant with a policy, including x-obleth-boons: off, unrouted and non-chat requests: embedding, rerank and moderation inputs are scanned too. Output guardrails that block or redact read chat responses only, so under such a policy /v1/completions, audio transcription and translation, and unrecognised paths return 400 before the input scan; use /v1/chat/completions, /v1/responses or /v1/messages. Under an input policy, auto routing doesn't call the intent classifier. See Guardrails.