Auto routing

Send a request

curl http://localhost:8088/v1/chat/completions \
  -H "Authorization: Bearer $OBLETH_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Write a Python function that parses ISO-8601 durations."}]}'

The gateway picks a registered chat model before admission. The request is then queued, billed and logged under the model it picked. Other request types (embeddings, audio, images) must name a model.

With difficulty tiering on, send x-obleth-effort: low, medium or high to set the request's difficulty yourself.

If no model can serve the request, it fails with 503 no model is available to satisfy the auto request.

How a model is chosen

The router works on the live model list, in three stages.

1. Filter. Models that cannot serve the request are dropped: disabled, staged, deprecated or retired, unhealthy, not eligible for auto, outside the tenant's model list, too small a context window, or missing a capability it needs (tools, a JSON schema, images). A matching boon can stand in for vision or schema support.
2. Tier (optional). With difficulty tiering on, models too weak for the request's difficulty on its topics are dropped. This stage narrows the field but never empties it.
3. Score. Each survivor is scored on spare capacity, this request's estimated cost, and tag fit, then multiplied by its route bias.

At temperature 0 the top score wins, ties broken by name. Above 0, the pick is sampled from the scores, spreading traffic across near-equal models.

Tags and difficulty

Each model can carry tags from a fixed list, set in the Routing section of its page:

coding · general · reasoning · math · vision · long-context · fast · creative · writing

creative is fiction and verse. writing is any prose to draft or edit: emails, letters, essays, posts, and rewrites for tone, length or grammar.

A tag can carry a strength from 1 to 3, such as coding:3, for difficulty tiering.

A request's tags and difficulty come from the classifier, when one is set, or else from keyword heuristics. The heuristics find only coding, math, vision, long-context and writing.

The classifier is a small, fast chat model that you register like any other. It answers through verdicts: one yes/no question per tag that the candidate models carry, plus a three-level difficulty score, each read from the first token's probabilities. The questions share a prompt prefix, so with prefix caching they cost about as much as one. A tag applies above p(yes) 0.5, at most three per request. coding and writing are never applied together: "write a SQL query" says yes to both, so only the stronger of the two is kept.

A classifier failure routes cheaper, never more expensive: unanswered tags are not applied, unanswered difficulty is 1, and with no answers the heuristics are used. A slow or missing classifier never blocks a request. Pick a model that answers directly, not a reasoning model. To build one, see Training a classifier.

Conversations

By default each request is routed afresh. The classifier reads the newest user message, so a conversation that opens with small talk and moves on to code is routed as code from that turn on.

A follow-up with no topic of its own, such as "make it shorter" or "thanks", classifies as general alone. The user message before it then decides, up to three user messages back, so the follow-up keeps the conversation's topic. Those messages were classified when they were sent, so the answer usually comes from the cache.

Tool results are not user messages: every step of an agent's tool loop is classified on the request that started it. Spare capacity is scored on every request, so two turns on the same topic can still go to different models when one is busy. Without a classifier, the heuristics read the whole conversation.

Keep a conversation on its model

Routing every turn afresh lets a conversation move between models as its turns change difficulty or wording, or as load shifts, and each move changes style and quality mid-task. With Stay with a model while it answers well on under Conversations in Settings › Routing (sticky_enabled on /api/v1/settings/auto-router, off by default), a conversation stays on the model that answered its last turn and moves up a level only when that model is not doing a good job:

• the client sends the same request again after it was answered (a retry or regenerate);
• the classifier reads the user's newest message as saying the last answer was wrong (one more single-token question, asked only on follow-up turns);
• the last answer was broken: no content and no tool call, or tool-call arguments that are not JSON;
• the model made the same tool calls three turns in a row, or two tool turns in a row came back with errors.

A move up raises the conversation's floor one level on its topics and leaves the model that failed out; at the top level it goes to another model there. A harder turn raises the floor to its difficulty, and the floor never drops. On a new topic the model stays when it holds a level there at least the turn's difficulty; otherwise the turn is routed within the new topic, no lower than the conversation's level. When the model cannot take a request (it is down, the context is too long, a capability is missing, or the key may no longer use it), the turn goes to another model at the conversation's level or above. The first turn of a conversation is routed as before.

The gateway keeps no conversation id and stores none of the conversation's text. After a turn is served it stores the model and level for two hours under a hash of the user message and the answer (the tool-call ids for a tool turn; reasoning and whitespace are ignored), and the next request finds it by hashing its last assistant message. A conversation whose client edits the last answer is routed as a new one.

Every auto response names its model in x-obleth-auto-model. With stickiness on, x-obleth-auto-reason says why:

obleth_auto_routes_total{model,reason} counts the same reasons (per_turn when stickiness is off), and the request's auto_route span records the decision.

Tune it in Settings

Settings › Routing shows readiness findings in its header (listed at the end of this page), then sets the classifier (on or off, its model and timeout), the default model for Anthropic clients, the default model for verdicts, whether conversations keep their model, and the weights, from a profile or by hand:

Changing any value makes the profile Custom. Changes reach every gateway within 15 seconds.

The same settings are GET and PUT /api/v1/settings/auto-router. A PUT changes only the fields it sends:

curl -X PUT http://localhost:9180/api/v1/settings/auto-router \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"classifier_enabled": true, "classifier_model": "qwen3-0.6b", "classifier_timeout_ms": 250}'

GET /api/v1/router/readiness returns the readiness findings.

Try a prompt in the Playground

The Playground's Router tab shows where a prompt would go without sending it, and how each candidate scored.

The weight sliders change only the simulation, with live and draft scores side by side, until you choose Apply to gateway…. See Playground.

Simulating a decision

POST /api/v1/router/simulate runs the same code as the data plane against the live fleet and returns the decision without dispatching anything:

curl -X POST http://localhost:9180/api/v1/router/simulate \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"prompt": "Prove the AM-GM inequality", "classify": true}'

Every body field is optional and overrides a live input: prompt or messages, max_tokens, tenant_id, effort, the capability flags, any weight, busyness (in-flight counts per model) and uniform (a fixed random draw). classify: true uses the live classifier instead of the heuristics, and reads a messages conversation turn by turn as a real request is read. The response lists the pick, the tags and difficulty with their source, every scored candidate, and each dropped model with its filter.

A traced request also records an auto_route span explaining the decision it served.

Per-model controls

On a model's page, under Routing:

• Eligible for auto off keeps the model out of auto. Clients can still call it by name.
• Route bias multiplies the model's score (0.1 to 3.0, 1.0 neutral), to prefer or avoid a model without retagging or repricing it. A small bias such as 1.05 breaks a price tie.

spare     = 1 − in_flight / cap        # cap: the model's max in flight, or default_soft_cap (8)
cost      = 1 for the cheapest estimate of this request, 0 for the dearest
base      = capacity_weight × spare + cost_weight × cost
score     = tag_weight × tag_match + (1 − tag_weight) × base    # base alone when the request has no tags
final     = score × route_bias