obleth

You needA fast drafter model, and a deployment of the target whose backend serves prompt_logprobs

Turn it onSettings › Boons › Speculation, then the model's page › Boons

It costsDraft and verify calls, billed as speculation_draft and speculation_verify

A fast drafter writes the answer, and a scoring deployment of the target checks every draft token in one prompt_logprobs prefill, far cheaper than the target writing those tokens. A draft that passes is sent; one that doesn't is dropped and the target answers. The client only ever gets text the target verified or wrote.

Turn it on

1
Prepare a scoring endpoint
The target needs a deployment whose backend supports prompt_logprobs and serves /tokenize at the server root, at a direct URL rather than through another gateway. Backends that can't take prompt_logprobs scoring may fail under it.
2
Turn Speculation on
Open Settings › Boons, switch Speculation on and Open its row. Choose a Profile, pick the fleet Drafter and, for the Calibrated profile, a Classifier. Under Advanced, set the Scoring endpoint URL template if your scoring deployments follow a naming rule. Save with the save bar.
3
Grant it to the target model
On the target model's page, switch Speculation on under Boons. Its cascade settings appear: pick this model's own Drafter, or leave the fleet default. Under Advanced wiring, set Scoring endpoint URL if this model's scorer isn't covered by the fleet template, and Scoring endpoint serves if that backend uses a different model name. Save.

Or through the Management API:

curl -X PUT http://localhost:9180/api/v1/settings/boons \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "speculation_enabled": true,
    "speculation_draft_model": "qwen3-4b",
    "speculation_verify_url_template": "http://{upstream}-scoring.serving.svc.cluster.local:8000/v1"
  }'

curl -X PUT http://localhost:9180/api/v1/models/$MODEL_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "boons": ["speculation"], "draft_model": "qwen3-4b" }'

Profiles

Profile	What it does
Calibrated	Drafts only in categories where drafts verify reliably (coding, math, summarization, writing), each with its own ship floor, and sends infrastructure, planning, database, regex, explanation and debugging straight to the target before any draft is paid for. Needs a classifier.
Cautious	Drafts every eligible request and holds it to one strict gate (the defaults below). Works without a classifier.
Custom	Shown when you change any value by hand.

A request where it helps

Nothing changes for the client:

curl http://localhost/v1/chat/completions \
  -H "Authorization: Bearer $OBLETH_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5-3",
    "stream": true,
    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}]
  }'

The response carries x-obleth-boons: speculation when the cascade was armed. The gateway logs whether the draft shipped, was deferred or escalated, and why.

How a draft is judged

The scorer returns two numbers for a draft: agreement, the share of draft tokens the target would itself have picked first, and the mean log-probability of the draft tokens. A draft at or above both ship floors (Ship when) is released.

For a streaming client the draft is checked as it grows: first after First check tokens, then every Every tokens.

Zone	Condition	What happens
Pass	agreement ≥ ship floor and log-probability ≥ ship floor	The verified span is sent
Give up	agreement or log-probability below the Give up when floors	Drafting stops; the target answers
Defer	anything between	Keep drafting and check again

If nothing has been sent by Decide by tokens, the target answers. Once a span has been sent, a later failed check has the target continue the sent text rather than start over, so the client never sees its answer rewritten. Sent spans play back at Streaming pace.

Settings

Settings label	Field	Default
Speculation on	`speculation_enabled`	`false`
Drafter	`speculation_draft_model`	none; the fleet default for targets without their own
Classifier	`speculation_classify_model`	none
Categories	`speculation_category_gates`	`[]`, as `[{tag, speculate, agree_min, lp_min}]`
Try a draft for categories not listed	`speculation_unlisted_categories_speculate`	`true`
Scoring endpoint	`speculation_verify_url_template`	empty
Agreement to ship	`speculation_agree_min`	`0.5` (0 to 1)
Confidence to ship	`speculation_lp_min`	`-1.0` (0 or below)
Agreement to give up	`speculation_abort_agree`	`0.45`
Confidence to give up	`speculation_abort_lp`	`-1.6`
First check	`speculation_first_chunk_tokens`	`80` tokens
Every	`speculation_chunk_tokens`	`250` tokens
Decide by	`speculation_decide_by_tokens`	`450` tokens
Longest draft	`speculation_max_draft_tokens`	`2048` tokens
Streaming pace	`speculation_pace_ms`	`9` ms; `0` sends each span at once
Budget	`speculation_timeout_ms`	`45000` ms before anything is sent
Drafter template arguments	`speculation_draft_chat_template_kwargs`	none; for example `{"reasoning": false}`, sent to the drafter only

An omitted field keeps its stored value, and so does 0 for the token counts and the budget. An out-of-range floor is ignored. "" clears the drafter, the classifier or the template; {} clears the template arguments.

On the model: draft_model, verify_api_base and verify_upstream_model.

When it applies

A request is drafted only when all of these hold:

Speculation is on, and the model has speculation in its boons list.
A drafter resolves (the model's own, else the fleet default) that is registered, enabled and not the target itself.
A scoring endpoint resolves (the model's verify_api_base, else the fleet template).
The request is a chat completion with text-only messages and one choice, without response_format, logit_bias, modalities, audio, logprobs, top_logprobs, prompt_logprobs or echo.
The key is not an internal probe key, so health checks always measure the target.
No other boon needs the target's own answer: structured output, output guardrails and a tool loop with real MCP tools all rule it out.

Any failure before the first span is sent (a draft error, a verify error, a failed gate, the budget running out) makes the boon abstain, and the request goes down the normal path with its usual retries, failover and caching.

Categories and the classifier

With a classifier and at least one category set, each request is first tagged with one of the category names. A category with Try a draft off goes straight to the target at no draft cost. One with it on uses its own ship floors; the give-up floors stay global. A blank floor on a category uses the built-in default (0.5 agreement, −1.0 confidence), not the value under Ship when. A request in no listed category, or when classification is off or fails, follows Try a draft for categories not listed.

When the model also holds image generation or web search and the request can't be classified, speculation abstains, because a draft can never produce the picture or the search the user may need.

The scoring URL template

The template may use {upstream} (the model's upstream name) and {model} (its API name), in the host and in the path or query, never in the scheme or credentials. A value put in the host must be plain DNS labels and the result must not be an IP address; a value in the path is percent-encoded. A model whose values can't fill the template safely has no scoring endpoint and is served normally. The template and every verify_api_base pass the same outbound URL checks as a model's api_base, on save and on restore.

The verifier is the target model with its address swapped for the scoring endpoint, so verification uses the target's upstream key.

Billing

The client's request is billed as a request for the target model, with the shipped draft's tokens when a draft was sent. Draft calls are speculation_draft rows under the drafter's name; verification prefills are speculation_verify rows under the target's name at its rates. Both count toward term budgets. A draft dropped mid-stream before anything was sent is not billed, since its usage is unknown and closing the connection stops it.

PreviousWeb search

NextKnowledge