OpenAI-compatible API

obleth's data plane is a transparent OpenAI-compatible proxy. Any client that speaks the OpenAI HTTP API works with obleth with only a base_url change.

Authentication

All data-plane requests must include a tenant API key. Either header is accepted:

Authorization: Bearer sk_<48 hex chars>
x-api-key: sk_<48 hex chars>

The admin token (OBLETH_ADMIN_TOKEN) is for the Management API only. Never send it to the data plane.

Supported routes

obleth proxies all standard OpenAI inference routes to the configured upstream:

A request that names a registered model on an unusual path is still forwarded to that model's upstream untouched, so provider-specific extensions keep working. A request that resolves to no registered model on a path obleth doesn't recognize is rejected with 404 unknown endpoint before any dispatch or telemetry — stray scans and probes (/props, favicons, vulnerability scanners) never become upstream round-trips, ledger rows, or a leak of the internal upstream URL. The /v1/models discovery endpoints are the only model-less paths the gateway serves.

GET /v1/models

obleth does not simply forward this to one upstream. It asks every distinct upstream that backs a registered model (the default base plus each model's endpoints) for its own catalog and unions the entries verbatim, de-duped by id and sorted — so Slurm-hosted models on their own endpoints appear alongside the gateway's. Lookups are concurrent and best-effort; a slow or down upstream is skipped rather than failing the listing.

Entries whose id matches a registered route are annotated with that route's modality, so clients don't have to guess it from the id:

• model_type — obleth's vocabulary (chat, embedding, audio_transcription, audio_speech, image).
• mode — the LiteLLM-convention alias many clients already read, identical except that image is spelled image_generation.

Entries that match no registered model (wildcard pass-throughs) are left verbatim, without those fields.

obleth serves more than chat. Embeddings, speech-to-text, text-to-speech, and image generation are all OpenAI-compatible and billed through the same ledger. Each non-chat route declares a model_type. See Multi-modal Models for the full guide.

Non-streaming request

curl -s http://localhost/v1/chat/completions \
  -H "Authorization: Bearer $SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-70b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is obleth?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Streaming (SSE)

curl -N http://localhost/v1/chat/completions \
  -H "Authorization: Bearer $SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-70b",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to 5"}],
    "max_tokens": 32
  }'

obleth streams the SSE response byte-for-byte from the upstream. The fairshare permit is held until the stream closes.

Models

obleth routes by the model field in the request body. For paths that require model resolution (/v1/chat/completions, /v1/completions), the model must be registered in obleth's model registry.

Register a model:

curl -X POST http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "llama-3-70b",
    "description": "Meta Llama 3 70B instruct route through Aibrix",
    "upstream_model": "meta-llama/Llama-3-70b-instruct",
    "api_base": "http://my-aibrix:8080",
    "input_cost_per_token": 0.0000005,
    "output_cost_per_token": 0.0000015,
    "context_window": 131072,
    "enabled": true
  }'

For the bundled benchmark fixture backend, register a model route named benchmark-endpoint with api_base: "http://benchmark-backend:8081", or let obench create and update its fixture routes for you.

api_base convention: set api_base to the provider base URL ending in /v1 (for example https://provider.example/v1), not a full endpoint URL. obleth preserves the client's request path and appends it to api_base, so a full endpoint like .../v1/embeddings would double the path. This applies to every model type.

Model admission weight

admission_weight scales the tenant's weight for the fairshare score on this model. It is expressed in percent and defaults to 100, meaning the tenant's own weight is used unchanged:

effective_weight = round(tenant.weight × model.admission_weight / 100)   # minimum 1

Set it above 100 to make a route count for more — see Admission weights for model cost for a worked table.

# Count this route as 4x the tenant's weight
curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{...existing fields..., "admission_weight": 400}'

SDK configuration

Python (openai package)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost/v1",   # or http://localhost:8088/v1 direct
    api_key="sk_...",
)

response = client.chat.completions.create(
    model="llama-3-70b",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=64,
)
print(response.choices[0].message.content)

LangChain

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost/v1",
    api_key="sk_...",
    model="llama-3-70b",
)

TypeScript / Node.js

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost/v1',
  apiKey: 'sk_...',
});

const response = await client.chat.completions.create({
  model: 'llama-3-70b',
  messages: [{ role: 'user', content: 'Hello' }],
  max_tokens: 64,
});

Request limits

Session tracking (request logs)

To group related requests in the dashboard Request Logs view, send a session identifier on each call. obleth records it on the ClickHouse usage row and does not forward it to the upstream unless you include it in the body yourself.

Accepted sources (first match wins):

Client-supplied values are trimmed and capped at 200 characters. When none is present, obleth derives a stable id by hashing the conversation seed. The OpenAI user field is deliberately not a session source — it identifies an end-user, not a conversation. See Conversations & Sessions and Control Plane — Request Logs.

Headers forwarded to upstream

obleth forwards the client's headers to the upstream, minus host, content-length, connection, accept-encoding, the client's Authorization / x-api-key credentials, and x-obleth-boons (a gateway directive, not an upstream header). If the model has an api_key configured, obleth injects that as the upstream's Authorization: Bearer <model_api_key> instead.