Capacity Auto-tune

Each registered model can carry its own max_in_flight slot cap — an optional ceiling on concurrent requests for that route, inside the global fairshare scheduler limit (OBLETH_GLOBAL_MAX_IN_FLIGHT). Auto-tune helps you set that cap for self-hosted chat and embedding models by driving real load directly at the upstream and measuring where throughput stops climbing or tail latency breaches your SLO.

Cloud-hosted models should stay on a static cap so spend stays bounded. Auto-tune is intentionally recommend-only: the probe never writes config; you apply the suggestion explicitly.

From the dashboard, open a chat or embedding model and switch to its Capacity tab to set the admission weight, capacity mode (static/tuned), max-slot cap, and to launch the auto-tune probe:

Static vs tuned

Both modes use the same max_in_flight field. The mode is metadata for operators and audit — it does not change admission behavior on the data plane.

How the probe works

The ramp probe runs in the Management API (obleth-admin). It:

1. Sends real requests directly to the model's api_base, bypassing obleth's gateway admission, so the measurement reflects the upstream's true capacity rather than the gateway's current cap.
2. Steps concurrency up a geometric ladder — 1 → 2 → 4 → 8 → … up to the ramp ceiling — sustaining each level for 2.5 seconds and recording sustained throughput (successful rps), p50, and p99 of total request time.
3. Runs the complete ladder every time rather than stopping at the first knee: probing the whole curve on every run is what makes the recommendation reproducible. Only the hard safety limits below — or a step in which every request failed — cut it short.
4. Derives a latency ceiling from the concurrency-1 baseline: baseline_p99_ms × latency_headroom, with a 100 ms absolute floor so a very fast model isn't tripped by ordinary jitter.
5. Recommends the knee from the finished curve, reporting why the ramp ended:
   • latency_degraded — a step's p99 climbed past the latency ceiling
   • plateau — throughput gain vs the previous step fell below ~7% (saturation)
   • max_concurrency — reached the ceiling with no knee; the real knee may be higher
   • no_data — no step produced usable samples (upstream unreachable)

The recommended max_in_flight is the highest concurrency that stayed under the latency ceiling while still meaningfully improving throughput.

The ramp ceiling

The ceiling is sized from whichever of these you supply, in order — so a one- or two-replica backend is never pounded all the way to 512:

Workload profile

Latency, and therefore the in-flight knee, is very different for a short chat turn than for a large-context coding turn, so the probe request is shaped to match real usage:

Safety caps

Defaults when omitted: workload = chat, latency_headroom = 4.0, and a ramp ceiling of 64 when neither replicas nor max_concurrency is given.

Management API

All three routes are audit-logged. See Management API — Models for the full listing.

Set capacity mode

curl -X PUT "http://localhost:9180/api/v1/models/$MODEL_ID/capacity-mode" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"capacity_mode": "static"}'

Run probe (recommend only)

curl -X POST "http://localhost:9180/api/v1/models/$MODEL_ID/autotune" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"workload": "chat", "latency_headroom": 4.0, "replicas": 2}'

Example response (truncated):

{
  "model_id": "...",
  "model_name": "qwen3-8b",
  "modality": "chat",
  "workload": "chat",
  "recommended_max_in_flight": 32,
  "knee_reason": "latency_degraded",
  "baseline_p99_ms": 280,
  "latency_ceiling_ms": 1120,
  "latency_headroom": 4.0,
  "max_concurrency": 64,
  "recommended_throughput_rps": 18.4,
  "duration_ms": 12400,
  "steps": [
    {
      "concurrency": 16,
      "throughput_rps": 17.2,
      "p50_ms": 420,
      "p99_ms": 890,
      "requests": 430,
      "errors": 0
    },
    {
      "concurrency": 32,
      "throughput_rps": 18.4,
      "p50_ms": 510,
      "p99_ms": 1120,
      "requests": 460,
      "errors": 0
    }
  ]
}

Apply recommendation

curl -X POST "http://localhost:9180/api/v1/models/$MODEL_ID/autotune/apply" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"max_in_flight": 32}'

This sets max_in_flight, flips capacity_mode to tuned, stamps capacity_tuned_at, syncs the data plane, and records an audit entry. You can also apply a value from a saved probe report without re-running the probe.

Control plane

On the Models page, expand a chat or embedding route:

• Capacity mode — Static / Tuned toggle in operational controls (calls PUT …/capacity-mode).
• Max slots — manual max_in_flight editor (calls PUT …/capacity).
• Auto-tune capacity panel — opens a dialog that warns about live upstream load and token cost, takes Replicas running, Latency headroom, and the workload shape, runs the probe, shows the recommendation — including the measured baseline p99 and the latency ceiling it implied — plus a per-step curve table, and offers one-click Apply.

The auto-tune panel is hidden for non-probeable modalities (image, audio_*, etc.).

When to use it

Per-model slots sit inside the global scheduler cap. Effective admission is the minimum of global max_in_flight, per-model slots, tenant/group limits, and fairshare. See Capacity Provider and Scaling.