Capacity auto-tune

Run a probe

1. 1

   Open the dialog

   On the model's page, under Capacity, choose Auto-tune capacity. Run it in a quiet window: the probe sends real concurrent load to the model's backend.

2. 2

   Describe the backend

   Set Replicas running, the copies of the model serving now; Latency tolerance, how much slower than one idle request you accept at peak (Tight 2×, Balanced 4×, Relaxed 8×); and the Workload: short chat turns, or coding with a large context and longer replies.

3. 3

   Run and apply

   Choose Run probe. The dialog shows the recommended slots, the throughput there, the latency budget, why the ramp stopped, and a row per step. Apply sets the model's pool to the recommendation and switches it to the tuned capacity mode.

Keep a model on a metered cloud API on a fixed pool instead: the probe would spend real money, and a fixed number bounds your spend. If the backend's replica count changes on its own, use the discovered mode, with the tuned value divided by the replicas as the per-replica concurrency.

How the probe works

The probe runs in the Management API and sends requests straight to the model's API base, bypassing obleth's own admission, so it measures the backend rather than the current pool. It carries the model's upstream headers.

1. It steps concurrency up 1, 2, 4, 8 and so on to a ceiling, holding each step for 2.5 seconds and recording throughput and p50 and p99 latency.
2. It sets a latency budget from the concurrency-1 baseline: the baseline p99 × the tolerance, never below 100 ms.
3. It always runs the whole ladder, so the result is reproducible; only the safety limits below, or a step where every request failed, cut it short.
4. It recommends the highest concurrency that stayed within the budget and still added throughput.

The ceiling is max_concurrency if you give it (at most 512), otherwise 32 × Replicas running, otherwise 64.

Capacity modes

static and tuned both use max_in_flight, so between them the mode is a record for you and the audit log. You can switch back to static without losing the value.

Management API

# Run a probe (changes nothing)
curl -X POST "http://localhost:9180/api/v1/models/$MODEL_ID/autotune" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"workload": "chat", "latency_headroom": 4.0, "replicas": 2}'

# Apply a recommendation
curl -X POST "http://localhost:9180/api/v1/models/$MODEL_ID/autotune/apply" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_in_flight": 32}'

The probe answers with recommended_max_in_flight, knee_reason, baseline_p99_ms, latency_ceiling_ms, recommended_throughput_rps and a steps array. Applying sets max_in_flight, switches the mode to tuned, stamps capacity_tuned_at and updates every gateway replica. Probes and applies are recorded in the audit log. See Management API.