Replicas and failures

How replicas run

Every pass (15 seconds by default), the provisioner compares each deployment with its jobs:

1. Below the replica count, it submits jobs. Each job's script gets a block before it that binds the first free port in the replica's port window (serving_port + i × OBLETH_PORT_SPAN) and exports it as OBLETH_SERVING_PORT, so replicas can share a node. The server must listen on that port. A job on several nodes runs one task per node. Memory 0 is sent as the smallest node's memory in the partition less what Slurm keeps back for the node (MemSpecLimit), so a site's own default can't replace it; when the nodes can't be read, 0 is sent as is.
2. For a running job, it checks the health path on each port in the window from where the provisioner runs. When one answers 2xx, it sends one warm-up request and adds the replica as an endpoint at http://<node address>:<port>/v1.
3. A job that ends, by preemption or walltime, is marked lost and replaced. Above the replica count, it cancels queued jobs first, then loading, then serving ones, oldest first; a cancelled replica leaves the endpoints before its job stops.
4. A replica whose job reports running but whose server has died is restarted: after its health checks fail for about five minutes (OBLETH_PROVISIONER_RESTART_AFTER_FAILURES, 20 passes), or after two failed checks from the gateway. At most one replica per model is restarted per pass.

The model's own health comes from its endpoints; see Health checks.

Failures

• Failed launches. The provisioner stops submitting while a deployment has as many ended jobs on record as its Stop after failed launches (0 never stops), counting a job it found ended in the same pass. Ended jobs are kept for OBLETH_PROVISIONER_LOST_RETENTION_SECS (900 seconds). The deployment's page shows Slurm's last word on each; Try once more clears them at once.
• Refused jobs. When slurmrestd rejects a submit, for example a wrong account, partition or QoS, the deployment's page leads with Slurm's own sentence and says what to change, with the full answer folded underneath. Fix it; the message returns if the next launch also fails.
• A job's output. Each job writes <job name>-<job id>.out and .err in the deployment's Job logs folder. Without one, Slurm writes them in the job's working directory, /tmp on the node it ran on. When a deployment stops, its page says where the last job's output is.
• Uncertain answers. A pass changes nothing when the Management API errors, a job lookup fails, a 404 lacks Slurm's own job-not-found error, or every tracked job vanishes at once, which usually means the wrong cluster, URL or API version. After 10 minutes of held passes, the slurm_reconcile_held alert fires; slurm_reconcile_recovered follows.

Troubleshooting

A job runs but the replica never serves. The provisioner's check runs from wherever the provisioner runs, not from the compute node, so a curl that works on the node proves nothing. From inside the provisioner:

getent hosts gpu-node-01
curl -s -o /dev/null -w '%{http_code}\n' http://gpu-node-01:8000/health

A replica can serve for the provisioner while the model reads unhealthy, because the gateway checks the endpoints and routes requests from its own pods. Both need to reach the node. The simplest fix is Node addresses in the Slurm connection. On Kubernetes the chart also offers nodeResolution.hostAliases, nodeResolution.dnsConfig and nodeResolution.dnsPolicy for the obleth and provisioner pods. With Compose, add extra_hosts for both the obleth-provisioner and obleth services in an override file kept out of version control, then recreate the containers.

If the log says job is RUNNING but slurmrestd returned no nodes, or submits fail with schema errors, the API version doesn't match the cluster's /openapi/v3.