Model Health

Model health checks judge each registered route with the cheapest trustworthy signal available, in order:

1. Passive — a recent real 2xx for that model in the ClickHouse usage ledger. Free, and as good as a probe: the model demonstrably served.
2. Active minimal inference — a real forward pass against the endpoint the model's model_type actually serves (a one-token chat completion, a one-string embedding, one character of speech, a 0.1-second silence clip to transcribe).
3. Catalog existence — for types with no cheap inference probe (image, and any unrecognized type), GET {api_base}/models membership.

Active probes are real requests, so they consume a negligible number of upstream tokens. Those tokens are recorded against an internal health_probe tenant and tagged request_type = health_probe, so probe traffic never lands on a client's bill and stays out of usage stats by default.

A healthy result covers the parts that matter to operators:

• the model registry lookup and route configuration;
• the configured upstream being reachable; and
• the upstream actually accepting a real request for the route's upstream_model on the endpoint its model_type implies.

Transient conditions — an overloaded upstream, a single network blip, an unverifiable catalog — are classified as degraded or unknown rather than unhealthy, so a working model doesn't flap to "down" and fire false alerts.

Statuses

Passive signal vs active probe

The passive window follows the model's own check interval, with a floor of 300 seconds — so at the default 900-second interval a model with steady traffic is settled for free instead of probed. Only an observed success short-circuits the probe. Recent errors are deliberately not trusted as a verdict: they may be stale, and acting on them would suppress the very probe whose success would clear them, pinning a recovered model at unhealthy until the window aged out. Anything short of an observed success falls through to a live probe as ground truth.

What the active probe sends

Probing each type against its own endpoint is what keeps embedding, TTS, and transcription routes from being rejected by a chat endpoint and left sitting falsely degraded. Catalog lookups are cached per api_base for 60 seconds, so a sweep across many models sharing one upstream lists it once.

Classification

The probe makes up to three attempts (one try plus two retries with exponential backoff) on 408/429/5xx or a network error before recording a result. The final outcome maps as:

Configuration mistakes are not outages

A 400/404/422 rejection means either "the model is gone" or "wrong endpoint for this modality". obleth disambiguates against the upstream catalog:

• the catalog lists the model → degraded, with a message naming the endpoint that rejected it and pointing at model_type. It does not count toward failure alerts.
• the catalog does not list it → unhealthy, with the catalog as evidence.
• the catalog is a wildcard or unreachable → the original result stands unchanged. A wildcard adds no information and may never confirm health.

A config fix takes effect immediately

Changing a model's api_base, upstream_model, or model_type resets its health state — status back to unknown, failure streak zeroed, alert state cleared, and the next check scheduled for now — so a fixed route re-verifies within one worker tick instead of carrying a stale "down" badge. Check history rows are kept.

Pre-flight validation

POST /api/v1/models/validate answers, without writing anything, whether an upstream is reachable, whether its catalog is a wildcard pass-through, and whether it lists a given upstream_model. The dashboard calls it when you create or edit a model and surfaces the returned warnings; a save is never blocked by it.

curl -X POST http://localhost:9180/api/v1/models/validate \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "api_base": "https://provider.example/v1",
    "upstream_model": "meta-llama/Llama-3-70b-instruct",
    "model_type": "chat"
  }'

Per-endpoint health

When a model defines multiple endpoints, the worker probes each enabled endpoint independently with the same per-modality probe and records a per-endpoint health_status. Disabled endpoints are recorded as disabled without a network call. The data plane only routes to endpoints that are enabled and neither unhealthy nor disabled — unknown and degraded soft-pass — so a dead cluster drops out of rotation on its own while the rest of the model keeps serving.

A degraded endpoint probe lifts a stale unhealthy (it is still a real liveness signal) but never downgrades a confirmed healthy, so an endpoint that is reachable but whose upstream id doesn't appear in its /v1/models list can recover into rotation on its own.

Models with no static api_base (dynamic endpoints)

Slurm-provisioned models have no static api_base — their live URLs live entirely in the endpoint pool, which changes as replicas come and go. For these, the model-level status is derived from the endpoint pool instead of from an empty base. An endpoint counts as serving when its probe came back healthy or degraded (that is, reachable), and the pool is graded against the managed spec's min_replicas:

A recent passive success may only settle a pool that still has a live endpoint serving. Once every replica has gone away, a success recorded just before they died is stale, and the pool reports its own reality instead of riding that 2xx to healthy.

Because the endpoints are probed from the gateway, a Slurm replica that the provisioner promoted can still report unhealthy here if the gateway itself cannot reach the node — see the Slurm troubleshooting notes.

Dashboard

Open Models in the control plane.

The compact table shows each production model's latest health status and last check latency. Click a row to expand details:

• route fields and model status;
• admission weight and per-model max in-flight slots;
• health config, including interval, failure threshold, alerts, and maintenance;
• recent latency trend; and
• recent check history.

Benchmark fixture endpoints such as benchmark-endpoint are hidden by default. Use Show benchmark when you need to inspect or check them.

Manual checks

Run one check:

curl -X POST http://localhost:9180/api/v1/models/$MODEL_ID/health/check \
  -H "Authorization: Bearer $TOKEN"

Run checks for eligible models:

curl -X POST http://localhost:9180/api/v1/models/health \
  -H "Authorization: Bearer $TOKEN"

Bulk checks skip routes that are disabled, hidden for maintenance, or have checks disabled.

Scheduled checks

The worker is enabled by default:

OBLETH_MODEL_HEALTH_ENABLED=true
OBLETH_MODEL_HEALTH_INTERVAL_SECS=900
OBLETH_MODEL_HEALTH_TIMEOUT_SECS=30
OBLETH_MODEL_HEALTH_RETENTION_DAYS=30

Checks are claimed with Postgres row locking, so multiple Management API replicas can run safely without a single in-memory leader. Per-model jitter keeps large fleets from checking every route at once.

Alerts

Slack alerts use the existing incoming webhook config:

OBLETH_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
OBLETH_SLACK_ALERT_MIN_INTERVAL_SECS=300

Each model has its own failure threshold, default 2. obleth sends one down alert when the threshold is reached and a recovery alert when the model becomes healthy again.

The health badge tracks the same threshold as the alert: a model only reads unhealthy once it has crossed its failure threshold — not on the latest single probe — so the badge flips exactly when an alert would fire rather than flickering on one bad check. Combined with the active probe's extra retry-with-backoff, a brief blip (common on cold Slurm nodes) doesn't register as down at all.

Alerts are suppressed when:

• alerts are disabled for that model;
• the model is in an active maintenance window; or
• the route itself is disabled.

Failed checks are still stored during maintenance, so the history remains useful after the window ends.