OBLETHdocs

Inference gateways

Put obleth in front of an inference gateway or router that spreads requests over your model-server replicas.

3 min readUpdated for v2.4.0Edit on GitHub

You needA gateway or router with an OpenAI-compatible API
Connect itRegister each model with the gateway's URL as its API base
obleth addsKeys, tenants, fairshare, budgets and usage accounting

Who does what

An inference gateway sits between an API layer and several replicas of a model server and picks the replica for each request: the least loaded one, or the one with a matching prefix or KV cache. A Kubernetes Service in front of several model-server pods is the simplest form. obleth works in front of any of them, sending OpenAI-compatible requests to the gateway as the model's upstream.

ConcernoblethInference gateway
Who may send, and at what priority✓
Per-tenant and per-key budgets and limits✓
Usage, cost and energy accounting✓
Response cache, boons, MCP tools✓
Which replica serves a request✓
Prefix- and KV-cache affinity✓

obleth decides who sends and when; the gateway decides which replica serves it.

Connect a model

  1. 1

    Add the model

    On Models, choose Add model › Connect an endpoint and enter the gateway's base URL as the API base, for example http://my-inference-gateway.inference.svc:8080/v1. Check connection lists the models the gateway serves. See Register a model.

  2. 2

    Pick the upstream model

    The Upstream model is the ID the gateway routes on, sent in the request's model field. The model's Name is what clients call, so you can rename a model for clients without touching the gateway.

  3. 3

    Add the gateway's headers and key

    If the gateway reads a header to choose a pool, a routing strategy or a tenant, add it under Upstream headers; use the header names your gateway documents. If the gateway needs its own key, set Upstream API key. obleth never forwards a client's Authorization or x-api-key upstream; it sends the model's key as Authorization: Bearer <key>.

Upstream headers also go on obleth's own calls to the model, such as health checks and auto-tune, so a header that picks the backend picks the same one for them. See Upstream headers.

To register many models at once, use Import from a provider with the gateway's base URL. A format suffix in a gateway's model IDs, such as -FP8, moves into the model's quantization and the full name stays as an alias.

Size each model's pool

Set each model's pool to about the number of requests the replicas behind the gateway serve at once for that model. See Pool sizes, or measure it with Capacity auto-tune.

When an autoscaler changes the replicas behind the gateway, a fixed number goes stale. On Kubernetes, use the discovered mode with the kubernetes source, and point the model at a Service that selects the model-server pods, not the gateway's own pods. The endpoints source would count the gateway as a single replica.

Catalogs at /v1/models/

Some gateways serve their model catalog only at /v1/models/, with the trailing slash, and answer 404 at /v1/models. obleth's model validation and health checks retry with the slash, so such a gateway still validates and its catalog can be read. Check connection in Add model doesn't retry; type the upstream model instead. A gateway that refuses the catalog request outright leaves image and video models unknown rather than down; see Health checks. Clients calling obleth can use either spelling: both return obleth's own registry.

Without an inference gateway

An inference gateway is useful for replica-aware routing, not required. A model's API base can point straight at a single model server or at a hosted OpenAI-compatible provider. Several servers can be registered as the model's endpoints, and obleth fails over or balances between them itself.