Data Plane Routes

The data plane listens on port 8080 and proxies OpenAI-compatible requests to the configured upstream. All routes require a valid API key in the Authorization header.

Route table

A request that names a registered model on any other path is still forwarded upstream as-is, so vendor extensions such as /v1/rerank or /v1/moderations work without extra configuration. A request that resolves to no registered model and whose path is not one of the recognized OpenAI endpoints is rejected with 404 unknown endpoint rather than forwarded, so stray probes and scanners never reach the upstream or the usage ledger.

Model discovery

A plain GET /v1/models with no model in the body is answered from obleth's own registry, so every registered model — including Slurm-provisioned ones on their own endpoints — is listed, not just what one upstream reports. The non-standard detail probe (GET /v1/models carrying {"model": …}) and GET /v1/models/{id} fall through and are forwarded upstream untouched.

Routes that require a registered model resolve by the request model field against the model registry: /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /v1/audio/transcriptions, /v1/audio/translations, /v1/audio/speech, and the /v1/images/* routes. On those paths a missing model returns 400, an unregistered one 404, and a disabled one 403. Only chat routes participate in auto model routing. See Multi-modal Models.

Multipart (speech-to-text)

Audio transcription/translation clients send multipart/form-data with the audio file. obleth parses the multipart body, rewrites the model field to the route's upstream_model, rebuilds the form, and forwards the file part unchanged.

Upstream URL construction

obleth appends the client's request path to the model's api_base. Configure api_base as the provider base ending in /v1 (not a full endpoint URL) to avoid a doubled path such as .../v1/embeddings/v1/embeddings. When a model defines multiple endpoints, the selected endpoint's api_base is used the same way. See Reliability & Failover.

Authentication

Every request (except /health) must include:

Authorization: Bearer sk_<48 hex chars>

The key is resolved as described in the Authentication reference.

Request handling

1. Auth key is resolved → tenant, weight, group, TPM quota looked up
2. Model name extracted from JSON body → matched against model registry
3. Token estimate computed from the request body
4. Response cache checked if enabled for the model
5. Admission: in-flight check → queue if at capacity (admitted as soon as a slot opens)
6. Per-minute token budget checked and reserved atomically via Redis Lua (429 if exceeded)
7. Term budget checked for tenants with a cumulative cap (403 if exhausted)
8. Request forwarded to upstream with headers:
   • Authorization replaced with the selected endpoint's (or model's) api_key (if set)
   • bounded by a per-request timeout, retried with backoff on transient failures, and failed over to the next endpoint if configured — see Reliability & Failover
9. Streaming response streamed back to client
10. Actual token counts extracted from the response tail when the upstream reports usage
11. Budget reconciled (reserve → actual delta refunded)
12. Usage record written to ClickHouse (or WAL)

For non-token modalities, cost is computed from the response shape instead of token usage: image routes bill n × cost_per_image, and text-to-speech bills input characters × cost_per_character. Embeddings bill prompt tokens via input_cost_per_token. See Multi-modal Models.

Header handling

obleth forwards client headers except hop-by-hop, auth, and encoding headers:

• host
• content-length
• authorization
• x-api-key
• accept-encoding
• connection
• x-obleth-boons — a gateway directive (see Model Boons), never passed upstream

Every response carries x-obleth-request-id with the request's UUID; it is the id to use with GET /api/v1/usage/logs.

Response cache

When cache is enabled for a model, the data plane:

1. Computes SHA-256(model || 0x00 || request_body) over the client body before forwarding
2. Checks obleth:cache:{hash} in Redis
3. On hit: returns cached response immediately (no upstream call, no fairshare permit, no budget reserve)
4. On miss: forwards request, then stores successful 200 responses up to 512 KiB with the model's cache_ttl_secs

Cache hits include x-obleth-cache: hit. Misses continue through the normal streaming path and are recorded in the usage ledger as cache_status = "miss".

MCP gateway

Registered MCP servers are reachable at /mcp/{server} (and any sub-path) using the same tenant API key as the data plane. obleth authenticates the key, resolves the server from its hot cache, replaces the Authorization header with the server's stored auth_header, and reverse-proxies the request and response bodies unchanged — it does not parse JSON-RPC, so both streamable-HTTP and SSE transports work. Request bodies are capped at 16 MiB.

Outcomes are counted in obleth_mcp_requests_total{server,status}. See MCP Gateway.