OpenAI-compatible API

Send a request

Set the client's base URL to the gateway and its API key to a gateway key. The data-plane port is 8088 on the Compose host, 80 through the HAProxy edge profile, and 8080 when you run the gateway natively.

curl -s http://localhost:8088/v1/chat/completions \
  -H "Authorization: Bearer $OBLETH_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-70b",
    "messages": [{"role": "user", "content": "What is obleth?"}],
    "max_tokens": 256
  }'

The same settings work for LangChain's ChatOpenAI, the TypeScript openai package and any other OpenAI-compatible client. For Claude Code and the Anthropic SDKs, see Claude Code and Anthropic SDKs.

Authentication

Every route except GET /health needs a credential, in either header:

Authorization: Bearer sk_…
x-api-key: sk_…

The credential is an API key or a short-lived identity token from an issuer you trust. The admin token is for the Management API only; never send it to the data plane. Error bodies are listed in Authentication.

What the gateway serves

Other surfaces have their own pages: images, audio and speech, video jobs, /v1/messages, /v1/verdicts, /v1/search and /mcp/<server>. Every route and error message is in Data-plane routes.

The model field picks the model. A model's name and any of its aliases both work, and usage is always recorded under the canonical name. A variant name reaches the model with extra boons on. model: "auto" hands the choice to the auto router.

Streaming

Send "stream": true to get server-sent events. The gateway passes chunks to the client as they arrive and sets x-accel-buffering: no so an nginx-style proxy in front does not buffer them. The request keeps its place in the model's pool until the stream ends.

The gateway always asks the backend for token usage on the final chunk, because billing needs it. If your client did not ask for stream_options.include_usage, that extra chunk is removed before it reaches you.

When the backend reports prompt tokens served from its prefix cache in usage.prompt_tokens_details.cached_tokens (vLLM does with --enable-prompt-tokens-details), chat responses pass it through and the usage ledger records it as cached_input_tokens.

A whole stream ends with a finish chunk, the usage chunk if you asked for it, and [DONE]. If the backend stops before that, you get one event in the error shape below, which OpenAI's SDKs raise as an APIError, and the stream ends without [DONE]. See When a stream is cut short.

Errors

Errors use the OpenAI envelope:

{"error": {"message": "model 'llama-3-7b' is not registered", "type": "obleth_gateway_error"}}

A 503 with Retry-After means the request waited in the model's queue for OBLETH_ADMISSION_TIMEOUT_SECS (default 60) without getting a slot. Retry after the delay, or see How fairshare works.

Headers

The gateway reads these request headers:

And adds these to responses:

Client headers are forwarded to the backend except host, content-length, connection, accept-encoding, authorization, x-api-key, x-obleth-boons and x-obleth-end-user. When the model has an upstream key, the gateway sends it as Authorization: Bearer.

Responses API

POST /v1/responses becomes a chat request, runs through the same routing, admission, boons and billing, and comes back as a Responses object, streamed or not. Usage is recorded as request type responses.

• instructions becomes a leading system message, and input (a string, or message, function_call and function_call_output items) becomes the conversation.
• max_output_tokens becomes max_tokens, text.format becomes response_format, and function tools carry over. Hosted tools such as web search are dropped.
• The gateway stores no conversations. store: true, previous_response_id and background: true return 400; send the full conversation in input each time.
• Usage carries input_tokens_details.cached_tokens when the backend reports cached prompt tokens; input_tokens stays the full count.
• The body limit is 32 MiB.

Model discovery

GET /v1/models lists every enabled model once, under the name clients call, straight from the gateway's registry, followed by each of its variants. It never calls a backend, so a model whose backend is down or has no catalog is still listed, and models a backend serves but the gateway has not registered are not. A tenant with a model allowlist sees only its own models.

Each entry has id, object: "model", owned_by: "obleth", model_type, mode (the LiteLLM spelling, where image is image_generation and video is video_generation), quantization, tags, aliases, the model's own supports_* flags, and its boons and boon_details (see below). A variant's entry adds variant_of and description, and its boons are its model's plus its own. Staged and retired models are left out, and a deprecated model's entry adds status, plus replacement, retire_at and note when set. Search tools are listed at GET /v1/search/tools instead.

GET /v1/models/{id} returns the same entry for a name, alias or variant, and still answers for a retired model, so a caller can find its replacement. A staged model, or one outside the tenant's allowlist, returns 404.

GET /model/info adds prices, capability flags, context window, boons with their boon_details, variants, lifecycle status and last observed health. It never includes a model's upstream URL or key. See Model registry.

The model's own abilities and its boons

The supports_* flags describe the model itself. A boon is an ability the gateway adds, often by calling another model, so a result from a model with a boon can't be reproduced with that model alone. boon_details says, for each boon the model holds, whether it runs and which model does the work:

{
  "id": "text-model",
  "supports_vision": false,
  "supports_function_calling": false,
  "boons": ["vision", "web_search"],
  "boon_details": {
    "vision": { "active": true, "native": false, "helper_model": "vision-model" },
    "web_search": { "active": false, "inactive_reason": "needs_function_calling" }
  }
}

This model can't see images. When a request has one, vision-model describes it and text-model answers from the description. Its web_search grant does nothing, because the model can't call tools.

active follows the gateway's settings. A tenant's own compression policy can still turn compression off for its keys. To record what acted on one request, keep its x-obleth-boons response header. To call the model with no boons, send x-obleth-boons: off.