v2.4.0Source-available · BSL 1.1

An AI gateway for backends with fixed capacity.

Unlike a cloud API, a model server you run yourself can take only so many requests at once. obleth sits between your clients and those servers. When a model is full, it queues requests and admits them by weighted fair share. It routes each request to a model with room, and records what every tenant used. Clients keep their OpenAI or Anthropic SDKs.

Quick start Read the docs

app.pythe only change your clients make

from openai import OpenAI client = OpenAI( - base_url="https://api.openai.com/v1",+ base_url="https://ai.example.edu/v1", api_key=os.environ["OBLETH_KEY"], ) client.chat.completions.create(model="auto", …)

The obleth Overview: three things need attention, a full model pool, and traffic for the last hour

Fairshare admission

Weighted fair queuing, per model.

Each model has a pool sized to what its backend can serve. When a pool is full, requests queue and free slots go to groups, then tenants, then keys, by weight. The limits hold across every gateway replica.

Automatic routing

model: auto, with every score shown.

Candidates are filtered by health, capacity and required features, then scored on free capacity, price and task fit. Optional difficulty tiering sends harder requests to stronger models. The Playground's Router tab shows each score.

Usage accounting

Tokens, cost, energy and CO₂ per request.

Budgets are reserved at admission and settled at completion. Cost is frozen when a request ends, so later price changes don't alter reports. Energy comes from your Prometheus power metrics.

Fairshare dashboard

Who holds each slot, and who is waiting.

Each pool lists every tenant's running requests against its fair share and what it has queued. A weight change previews which tenants give up slots before it is saved.

How fairshare works

A model pool with 24 of 24 slots in use and 56 requests waiting, split across three tenants

Docker, Kubernetes or Slurm

Install guides

Docker ComposeThe whole stack with one command, including a GPU-free test backend.docker compose up -d KubernetesA Helm chart. Pool sizes follow each model's ready replicas, so your autoscaler stays in charge.helm install obleth oci://ghcr.io/…SlurmLaunch models as jobs from a recipe, keep them at a replica count, and route to them once healthy.via slurmrestd

One endpoint for every kind of model

/v1/chat/completions/v1/responses/v1/messages/v1/embeddings/v1/rerank/v1/images/generations/v1/audio/transcriptions/v1/audio/speech/v1/videos/v1/verdicts/v1/moderations/mcp/<server>

Connecting clients

Run it locally

Docker Compose starts the gateway, the dashboard and a GPU-free test backend. Then create a key and send a request.

Quick start