Health checks

Check a model

On a model's page, Check health now runs a check at once; on a model that is off, Check and turn on runs one and turns the model on if it passes. The page's Health section sets the schedule, alerts, the failure threshold and a maintenance window, and lists recent checks. On Models, select rows to check several models together.

# One model
curl -X POST http://localhost:9180/api/v1/models/$MODEL_ID/health/check \
  -H "Authorization: Bearer $TOKEN"

# Every model that is on, has checks on and is not in maintenance
curl -X POST http://localhost:9180/api/v1/models/health \
  -H "Authorization: Bearer $TOKEN"

What a check does

A check uses the cheapest signal it can trust:

1. Recent traffic. A successful client request for the model within its check interval (at least 300 seconds) settles it as healthy without a probe. Recent errors are never trusted on their own; they fall through to a probe.
2. A real request to the endpoint the model's type serves.
3. The upstream catalog (GET {api_base}/models) for image and video models, where a real request would be a costly generation.

Probe tokens are recorded under an internal tenant with request_type = health_probe, so they never land on a client's bill and are hidden from the request log. Probes and catalog requests carry the model's upstream headers.

What the active probe sends

A probe tries up to three times, backing off, on a 408, 429, 5xx or network error before it records a result.

Scheduled and bulk checks skip a retired model, whose backend is usually gone. A staged model is checked like an active one.

States

A model reads unhealthy only once it has failed its failure threshold of checks in a row (2 by default), the same point at which the down alert goes out. A single failed check doesn't flip it.

Catalog checks for image and video models

An image or video model is healthy when the upstream catalog lists it. Because some servers mount their catalog only at /v1/models/, a 404 is retried once with the trailing slash. When the upstream answers the catalog with another 4xx, for example a gateway that doesn't serve /models or guards it differently from inference, the model is unknown rather than down: it stays routable and doesn't alert. No answer at all is unhealthy, a 5xx is degraded, and a catalog that answers without the model is unhealthy. Catalogs are cached per API base for 60 seconds.

What clears a failure

• A passing check. The next scheduled or manual check that comes back healthy clears the failure and sends a recovery alert.
• Fixing the configuration. Changing a model's API base, upstream model, type or upstream headers resets its health: the state goes back to unknown, the failure count to zero, the alert state clears, and the next check runs within one worker pass. Check history is kept.
• A maintenance window. Until it ends, failed checks are still recorded but don't alert, and the model shows as in maintenance.

Alerts

Alerts go through the channels in Settings › Alerts; see Alerting. Each model sends one alert when it reaches its failure threshold and one when it recovers. Nothing is sent while alerts are off for the model, it is in maintenance, or it is off.