Verdicts

Send a state (a ticket, a document, a record, your application's current state) and a map of typed questions. Each question comes back as a typed verdict with a probability distribution and a confidence. The answer is read from the model's first-token probabilities over a fixed set of labels, so it cannot come back malformed.

Use it where you now ask a model a question and parse its reply: triage (is this urgent, which team owns it), gating (escalate, block or approve, with a probability to threshold on), rubric scoring, record matching and batch labeling. Verdict examples has requests to copy for several of these.

Ask a question

curl -s http://localhost:8088/v1/verdicts \
  -H "Authorization: Bearer $OBLETH_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b",
    "state": {"ticket": "My card was charged twice for order A-104. Please refund the duplicate."},
    "questions": {
      "is_urgent": {
        "type": "boolean",
        "instructions": "Does this need action today?",
        "criteria": {"true": "Money, security or an outage", "false": "Can wait for the normal queue"}
      },
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {"billing": "Payments, refunds, invoices", "technical": "Bugs, errors, outages", "account": "Login, profile, access"}
      },
      "frustration": {
        "type": "score",
        "instructions": "How frustrated is the customer?",
        "criteria": ["Calm and factual", "Mildly annoyed", "Frustrated", "Very angry"]
      }
    }
  }'

model is a chat model name, alias, family or auto (the auto router then picks among chat models). Leave it out to use the model set under Settings › Routing › Verdicts, or auto when none is set. A model family (a name that picks a member by reasoning effort) is answered by its none member, because a verdict is read off the first token and a member that thinks first cannot give one. state is a string, object or array; structured values are rendered as indented JSON. instructions and criteria descriptions may also be JSON. Unknown request fields are ignored.

Ask about images

Add images to judge photos, screenshots or scans. Each is a data:image/…;base64, URL or an http(s) URL that the model's backend fetches itself. With images, state is optional.

{
  "model": "auto",
  "state": "A customer's photo of the phone they are returning.",
  "images": ["data:image/jpeg;base64,/9j/4AAQSkZJRg..."],
  "questions": {
    "visible_damage": {"type": "boolean", "instructions": "Is the phone visibly damaged, such as cracked glass or a dent?"}
  }
}

• The model must take images itself: give it the vision tag, which sets supports_vision. Other models return 400, and auto picks only among models with vision. The Vision boon does not apply here.
• A request takes up to 8 images and 20 MiB of image URLs. Each question is its own call and carries the images, so image bytes times questions may be at most 64 MiB.
• The images come before the question text, so with prefix caching the backend can reuse them across the request's questions, like the state.
• Input guardrails scan the state and the questions, not the images.

Verdict examples has a full request.

Question types

A choice with more than 26 options is answered with two-letter codes (see How it works). A request holds 1 to 32 questions, and may mix types. Question ids are up to 256 bytes and option names up to 200. The rendered state may be up to 2 MiB.

The response

{
  "model": "qwen3-8b",
  "verdicts": {
    "department": {"type": "choice", "value": "billing",
      "probabilities": {"account": 0.0007, "billing": 0.9962, "technical": 0.0031}, "confidence": 0.97},
    "frustration": {"type": "score", "value": 2,
      "probabilities": [0.11, 0.78, 0.09, 0.02], "confidence": 0.62, "expected_value": 2.02},
    "is_urgent": {"type": "boolean", "value": true,
      "probabilities": {"false": 0.06, "true": 0.94}, "confidence": 0.71}
  },
  "usage": {"prompt_tokens": 430, "completion_tokens": 3, "total_tokens": 433}
}

• model is the model that answered, with aliases and auto resolved.
• value is the most likely answer. probabilities sum to 1 over the answer labels.
• expected_value (score only) is the probability-weighted level, a smoother signal when neighbouring levels split the probability.
• confidence, from 0 to 1, says how decisively the model answered. It is near 0 for a 50/50 split, and also low when most of the model's probability went to tokens that are not answer labels.
• usage sums every call the request made, retries included.
• answered appears only on a question the model gave no answer to, when the request set on_no_answer to low_confidence (see below).

Gate automated actions on both: probabilities say how likely the answer is, confidence says whether the model engaged with the labels at all. Floats are written as plain decimals at full precision, never in scientific notation.

When the model gives no answer

Sometimes none of the model's likely first tokens is an answer label, for example on a question that makes no sense. By default the request then fails with 502 and names the question. Set on_no_answer to answer it anyway:

"is_urgent": {"type": "boolean", "value": true,
  "probabilities": {"false": 0.5, "true": 0.5}, "confidence": 0.0, "answered": false}

Use low_confidence when you score many questions in bulk and would rather count an unanswered question as wrong than retry the whole request. Backend failures (a timeout, an unreachable model) still return an error.

How it works

Each question is one chat call to the model's own backend with max_tokens: 1, temperature: 0, logprobs: true and top_logprobs: 20. Booleans answer yes or no; choice options (in sorted name order) and score levels get the letters A to Z, which are single tokens in common vocabularies. The first token's probabilities over those labels become the answer.

The state goes in a system message that is identical for every question in the request, and all questions go to one endpoint at once. With prefix caching on the backend (vLLM --enable-prefix-caching, SGLang), the state is processed once and each further question costs about one cache hit and one output token. Ask every question about one state in the same request.

Thinking and harmony models. A model that opens with a <think> block or a harmony channel token spends its first token there, not on an answer. When less than 5% of the first-token probability lands on answer labels, the gateway retries that question once: with an empty <think></think> block already closed (continue_final_message, supported by vLLM and SGLang), or, for harmony models such as gpt-oss, through the backend's /v1/completions with the final channel already open. The form that works is remembered per model for the life of the process, so later requests skip the first attempt. Both attempts are billed, and the answers often still come back with low confidence, so prefer an instruction-tuned model and make one the verdicts default.

Transient failures. A 5xx, an unreachable backend, or a 200 with no usable logprobs gets one retry after 50 ms. If a question still fails, the request returns 502 naming the question and the model.

Admission and cost

• The request is authenticated and checked like any other, including the tenant's model allowlist and input guardrails, which scan the state and each question before admission.
• The whole request takes one slot in the model's pool, however many questions it has. An admission timeout returns 503 with Retry-After: 5.
• Budgets are reserved from an estimate (each image counts as 1,500 tokens per question) and settled once with the real usage, as one usage row with request type verdict. When a question fails, the calls that did run are still billed.
• With tracing on for the key, each question is a verdict:q:<id> span with its answer, confidence and token counts.
• Boons, output guardrails and the response cache do not apply.

To make a high-volume question set cheaper, a small model can be distilled from a large model's verdicts. See Training a classifier.