Training a classifier

Much judgment work is classification: is this ticket urgent, does this abstract meet the criteria, which model should serve this request. A small model asked cold is an unreliable judge. Distillation closes the gap: large teacher models label text through POST /v1/verdicts, and a small student learns to reproduce their probabilities.

The running example is the auto router's classifier: one yes/no question per routing tag (nine) plus a three-level difficulty score. The same recipe works for any verdicts task.

When it is worth doing

Distill when a judgment is high-volume, narrow and repeated. Run it a hundred times and you can prompt a large model; a hundred thousand times, train a small one. You need no labeled data: the teachers generate the labels.

Models asked cold fail in measurable ways. One 30B model answered yes to every category (p(yes) 0.84 to 0.98, including vision for text-only prompts), and from a lettered list put about 0.7 on whichever option came first. A student learns from thousands of confident "no" examples, which prompting cannot supply.

Stage 0: audit the teachers

Send a handful of inputs whose answer is not in doubt through /v1/verdicts, and compare p(yes) on categories that fit with categories that do not. From one fleet, on the routing task:

Size predicts little. Use three teachers from the top band if you can; with two, one odd judgment moves a label by half its range.

Prefer instruction-tuned models to reasoning models. A verdict reads the first token, and a reasoning model's judgment lives in the thinking it skips: gpt-oss-120b answered "no" at p=0.89 to "is this math?" on a grade-school math problem.

Stage 1: generate soft labels

Stream prompts from a corpus and ask each teacher every question in one /v1/verdicts call. The router shows the classifier one user message at a time, so label single user messages, short follow-ups included. The routing example used four openly licensed datasets: allenai/WildChat-1M (real traffic), HuggingFaceH4/ultrachat_200k (variety), ise-uiuc/Magicoder-OSS-Instruct-75K (coding) and openai/gsm8k (math).

• Keep the probabilities, not the top answer. A label of p(coding) = 0.62 teaches the student to be unsure where the teachers were, which makes its confidence worth thresholding on.
• Filter per question, not per input. Teachers often agree on eight questions and split on the ninth. Dropping the whole input throws away eight good labels.
• Check the averages per source. Math prompts should score high on math, code prompts on coding; a flat table means the teachers are not discriminating. One poor teacher pushed rejections to 83% of inputs; two audited teachers, to about 14%.

Each input costs one prefill plus one output token per question per teacher, because the questions share a prompt prefix on one pinned backend replica. Run labeling under its own tenant at weight 1: it uses capacity nobody else wants and loses every contended slot to interactive traffic. See Priority boost.

Stage 2: distill

Train the student on the KL divergence between its distribution over the answer labels and the teachers'. That is the quantity the gateway reads at inference, so training and serving optimize the same number.

• Train on the exact production prompt. Render training examples exactly as the gateway sends them (see Verdicts), and pin your renderer to that format with a test, so a prompt change without new training data fails the build.
• Ship a plain chat template. A template that opens a <think> block puts a thinking token, not an answer, first. Use plain ChatML for a one-token judge.
• Reward mass on the labels. Confidence is discounted by the probability that falls outside the answer labels. A stock Qwen3-0.6B puts about 4e-11 of its first-token mass on the labels; an extra loss term pulls that toward 1 within a few dozen steps.

A full fine-tune of a 0.6B model over about 100,000 inputs takes a few hours on one accelerator.

Stage 3: evaluate and deploy

Measure, in order: discrimination (the p(yes) margin, as in stage 0; below 0.5 the model is unusable), agreement with the teachers on held-out inputs, and calibration (expected calibration error, Brier score).

Serve the student like any other chat model and register it:

vllm serve runs/model-v1 --served-model-name my-judge --max-model-len 2048

For routing, point the classifier at it:

curl -X PUT http://localhost:9180/api/v1/settings/auto-router \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"classifier_enabled": true, "classifier_model": "my-judge"}'

Evaluate the served model, not only the checkpoint. Simulate with "classify": true compares decisions without dispatching anything. Keep the previous classifier registered until the new one has served real traffic for a day; switching back is one PUT.

Adapt it to your own task

Only the task definition changes. A task supplies:

1. The questions: your own booleans, choices and scores, such as inclusion criteria or severity levels.
2. The rendering: the exact strings production will send.
3. The corpus: text from your domain.
4. The probe set: a handful of inputs with certain answers, for stage 0 and the discrimination check.

The teacher audit, the soft-label ensemble, the loss and the checks carry over unchanged.

If you use your own operational data instead of public corpora, confirm that its data classification permits it, remove identifiers, and get consent where the text came from people. Distillation copies what is in the corpus into a model you keep.