Introduction

obleth is a gateway you run in front of model servers that have a fixed capacity: vLLM, SGLang, an inference gateway, or any OpenAI-compatible server you run yourself, on-prem or in your own cloud account. Clients send requests to one endpoint. obleth identifies the caller, decides who goes next when a model's backend is busy, forwards the request, and records what it cost.

These docs describe release v2.4.0.

What it adds

• Fairshare admission per model. Every model has its own pool of slots. When a pool is full, requests wait in a queue and each freed slot goes to the fairshare group, tenant and key furthest behind its weighted share. A busy model never holds up one with free capacity. See Fairshare engine.
• Automatic routing. Send model: "auto" and the gateway picks a registered model from the request's needs, cost and live load. See Auto routing.
• Usage accounting. Tokens, cost and energy are recorded per request. Per-minute token rates and term budgets are reserved at admission and settled to real usage afterwards; failed requests are not billed. See Quotas and rate limits.
• Boons. Capabilities the gateway grants a model that lacks them: vision, structured output, compression, knowledge retrieval, image generation, tool use through MCP servers, and speculative drafting. See Boons.
• Guardrails. Per-tenant scanning of request and response text for prompt injection, PII, banned keywords or harm, with block, redact or log actions. See Guardrails.
• One endpoint for many APIs. The OpenAI API (chat, completions, embeddings, audio, images, rerank, moderations), the Responses API, the Anthropic Messages API for Claude Code and the Anthropic SDKs, the Videos API, typed verdicts, and an MCP gateway. Every surface runs through the same admission, budgets and billing.

Where it sits

• The data plane serves client traffic, authenticated with tenant API keys (sk_…) or identity-provider tokens.
• The management API creates tenants, keys and models, changes weights and reads usage. It takes a separate admin token.
• The dashboard is a web app that works only through the management API.

obleth decides who sends and in what order. It does not pick which replica serves a request: that stays with your inference gateway or load balancer. See Inference gateways and Architecture.

When it fits

• Several teams, courses or services share models whose capacity you can't grow on demand.
• You want requests to wait their turn by weight when a model is full, instead of failing with 429.
• You want usage and cost attributed per tenant and key, in tokens.
• You want OpenAI and Anthropic clients to reach self-hosted models through one endpoint.

It is not built for spreading traffic across many hosted cloud providers, and its response cache is exact-match only, not semantic.

Where to go next

• Quick start: run it locally with Docker Compose and send a first request.
• Install with Docker Compose or Install on Kubernetes for a real install.
• First tenant and key: the first thing to do after installing.
• Connect clients: point SDKs and tools at the gateway.
• Model registry: register the models you serve.

The source, Compose files and Helm chart are in obleth-gateway. See License for the terms.