Slurm Provisioning

The Slurm provisioner is an optional plugin service — obleth-provisioner — that keeps managed models alive on a preemptible Slurm cluster. It submits batch jobs via slurmrestd, probes replicas for health, and promotes each healthy replica into obleth's model endpoint pool so the gateway routes traffic to it automatically.

The provisioner is not part of the obleth core stack. You opt in via config — add slurm to COMPOSE_PROFILES in your .env — after which it builds and starts with the same single command as the rest of the stack (or run the binary yourself).

How it works

Every reconcile tick (OBLETH_PROVISIONER_INTERVAL_SECS, default 15s):

1. Read Slurm settings from the Management API. If Slurm is not globally enabled, the tick is skipped.
2. List enabled managed models via the Management API.
3. List owned Slurm jobs from slurmrestd, identified by a job-name prefix.
4. For each managed model: probe starting replicas, submit new jobs when below target, promote healthy replicas into the endpoint pool, mark lost replicas (preempted/finished), cancel excess jobs.
5. Drain models that left the managed set — cancel jobs, detach endpoints, GC rows.
6. Cancel orphan jobs — any cluster job matching the prefix with no tracking row.

If either the Management API or slurmrestd is unreachable the tick bails out without taking any destructive action; it retries on the next tick.

Each tick's outcome — ok, idle (with the reason), or error (with the failure text) — is reported back to the gateway on the next poll, so the dashboard can tell a live reconcile loop from a frozen one. See Step 2.

Prerequisites

Before enabling the provisioner, verify that your cluster has:

• slurmrestd running and reachable from the provisioner. The provisioner authenticates with X-SLURM-USER-NAME + X-SLURM-USER-TOKEN headers; no username/password login is used.
• apptainer (or singularity) installed on all compute nodes. Jobs run as apptainer exec --nv <image> <launch_command>. The --nv flag passes NVIDIA GPU devices through; Apptainer must be able to find it on the job's PATH (/opt/apptainer/bin and /opt/singularity/bin are included in the job's default PATH automatically).
• A Slurm JWT for authentication. Generate one on your Slurm head node with scontrol token, or via your cluster's token service. Paste the raw JWT into the dashboard — it is encrypted at rest with the gateway's OBLETH_ENCRYPTION_KEY.
• Shared storage accessible by all nodes. The Apptainer .sif image file must be reachable by all nodes that could run the job (NFS, Lustre, GPFS, etc.). The path you give in the image field is used as-is on every compute node.

Step 1: Start the provisioner

Docker Compose

The provisioner is gated behind the slurm profile. Opt in via config by adding slurm to COMPOSE_PROFILES in deploy/docker/.env (copy it from .env.example if you haven't):

# deploy/docker/.env
COMPOSE_PROFILES=benchmark,edge,observability,slurm

Then one command — run from deploy/docker/, with no -f flags — builds and starts everything, the provisioner included:

cd deploy/docker

docker compose up -d --build      # build + start the whole stack (provisioner too)
docker compose down               # tear it all down

Running from deploy/docker/ lets Compose auto-load .env (so the profile activates) and auto-merge a local docker-compose.override.yml if you use one (see Troubleshooting). No --profile flag and no extra -f arguments are needed. The container only needs OBLETH_ADMIN_TOKEN (inherited from the core stack's .env) and OBLETH_ADMIN_BASE_URL; Slurm connection details are configured in the dashboard, not in .env.

Kubernetes (Helm)

Set provisioner.enabled: true in your values. provisioner.intervalSecs, provisioner.restartAfterFailures, provisioner.logLevel, and provisioner.resources map to the environment variables below. slurmrestd connection details and the master enable switch are configured at runtime in the dashboard, not in values.

systemd

A unit file ships at deploy/systemd/obleth-provisioner.service. Put env vars in /etc/obleth/provisioner.env, then:

sudo cp deploy/systemd/obleth-provisioner.service /etc/systemd/system/
sudo cp obleth/target/release/obleth-provisioner /usr/local/bin/
sudo systemctl daemon-reload
sudo systemctl enable --now obleth-provisioner

Cargo (development)

OBLETH_ADMIN_TOKEN=dev-admin-token \
OBLETH_ADMIN_BASE_URL=http://localhost:9180 \
cargo run -p obleth-provisioner

Step 2: Configure Slurm system settings

Open the dashboard → Settings → Slurm.

After saving, click Test connection to verify:

• JWT health — whether a JWT is set, whether it is expired, and when it expires.
• slurmrestd ping — whether the GET /slurm/{version}/ping endpoint responds with 2xx and how fast.

The provisioner reads these settings from the Management API on every tick, so changes take effect within one interval without a restart.

The Settings → Slurm tab also shows whether the provisioner process is currently running and, when the provisioner reports it, its version (and short commit). Because the provisioner ships as its own image and is deployed separately from the gateway, this makes a stale provisioner deployment obvious at a glance.

It also shows the outcome of the provisioner's last reconcile tick — ok, idle, or error with the failure text — and how long a non-ok streak has lasted. A heartbeat only proves the process is alive: a provisioner can poll green for days while every tick fails against slurmrestd, holding all replica state frozen. The tab therefore distinguishes "running" from "running but failing since X", and when reconciliation has been held for 10 minutes while Slurm provisioning is enabled, a deduplicated alert fires (with a recovery notice when it clears).

Node address overrides

Slurm allocations name compute nodes by short hostname (scgh001). Where the pods running obleth resolve those names unreliably — a corporate resolver behind a long DNS search list, for instance — that flakiness lands on every health probe and on every proxied request, because the data plane resolves the endpoint host per call. One missed lookup then surfaces as an instant 502 upstream request failed.

The Node address overrides list takes DNS off that path. Each entry is a hostname and an IP literal; the address must parse as an IP, because a hostname there would just push the flaky lookup back onto the proxy. The provisioner registers each replica's endpoint by IP and probes by IP, so neither the health check nor proxied traffic depends on per-request DNS. Edits take effect on the next provisioner tick without a restart.

Two behaviours follow from this, and neither needs any configuration:

• Existing endpoints migrate in place. An endpoint already registered by node name is rewritten to the resolved address on the next tick, keeping its name, priority, weight, and enabled flag — so running replicas stop depending on DNS immediately after an upgrade, without being re-provisioned.
• Un-aliased hostnames still resolve more robustly. With the list empty, a node name is resolved at promotion with a short retry, a per-node success cache, and a last-known-address fallback, so one transient miss no longer decides where an endpoint points. Model warm-up requests target the resolved address too.

On Kubernetes the chart offers a lower-level alternative for the same problem: nodeResolution.hostAliases (static /etc/hosts entries), nodeResolution.dnsConfig (extra nameservers and search domains), and nodeResolution.dnsPolicy, applied to the obleth and provisioner pods only. Use those when you want the whole pod to resolve cluster names; use the overrides above when you only want obleth's Slurm endpoints pinned.

Step 3: Create a Slurm-provisioned model

When creating a model in the dashboard → Models → New model, choose Slurm provisioned as the hosting mode. This choice is made at creation time — you cannot change it after creation.

Slurm-provisioned models have no static upstream endpoint. The provisioner registers replicas dynamically as they become healthy, so you do not provide api_base or api_key during creation.

Picking that hosting option lists the available deployment recipes — an sbatch script an admin has already tested, with a small YAML header carrying the routing metadata (engine, API model name, port, health path, replica targets) and the default placement. Deploying one creates the model route and its managed spec in a single step. Recipes are read from OBLETH_RECIPES_DIR (default: recipes/ beside the dashboard) and from templates authored in the dashboard; the deploy dialog pre-fills Deploy overrides (partition, QoS, replicas, time limit, plus any variables the recipe declares) and shows the launch script in an editable box. Overrides and script edits apply to that deployment only — the recipe file is unchanged. #SBATCH directives in the body are lifted into the submit fields, because slurmrestd ignores them.

Everything below describes what the provisioner does with the resulting spec, and applies equally to a spec you fill in by hand on the model's Provisioning tab.

Apptainer image requirements

When the spec has no script_body (the hand-filled path), the provisioner assembles a Slurm batch script that runs:

#!/bin/bash
set -euo pipefail
<preamble>
apptainer exec --nv <image> <launch_command>

With image blank, launch_command runs directly on the node instead. A non-empty script_body — what a recipe produces — is submitted verbatim and replaces this assembly entirely.

• <image> — the path to an Apptainer .sif image on shared cluster storage. This path is used verbatim on each compute node, so all nodes must be able to read it.
• <preamble> — the spec's preamble lines, verbatim, before the exec line (module loads, PATH changes).
• <launch_command> — the inference server startup command to run inside the container, e.g. python -m vllm.entrypoints.openai.api_server --model /data/models/llama3 --port 8080.
• --nv — passes NVIDIA GPU devices through to the container. Remove this flag from launch_command if you are not using GPUs.

Either way, the provisioner prepends a short bash block that binds the first free port in this replica's port window and exports it as OBLETH_SERVING_PORT. The window is serving_port + i × OBLETH_PORT_SPAN (span defaults to 8) for the smallest i no live replica already holds, so two replicas can share a node without colliding. Have your launch command listen on $OBLETH_SERVING_PORT rather than a hardcoded port — the provisioner probes the whole window to find where the server actually came up.

Building an Apptainer image for vLLM:

# Pull the vLLM Docker image and convert it to a .sif
apptainer pull vllm.sif docker://vllm/vllm-openai:latest

# Or build from a definition file for a custom environment
apptainer build vllm-custom.sif vllm.def

Place the .sif on your cluster's shared filesystem (NFS, Lustre, GPFS) and use that path as the image field in the model spec.

Image checklist:

• The inference server must listen on the port you set in serving_port.
• It must respond to HTTP GET <health_path> with a 2xx when healthy (used by the provisioner's health probe).
• GPU drivers are injected at runtime by --nv; you do not need to bundle CUDA in the image (though bundling it is fine too).
• If your cluster uses a module system (module load apptainer), make sure apptainer is on the default PATH for batch jobs, or prepend the load command to launch_command.

Managed model spec fields

The spec is set via the Provisioning tab on the model detail panel (or directly via the Management API PUT /api/v1/models/{id}/managed).

Monitoring replicas

On the model detail panel, the Provisioning tab appears only for Slurm-provisioned models. It shows:

• Edit provisioning spec — change partition, GRES, image, launch command, replica count, etc.
• Replica panel — live table of each replica: state (pending, starting, healthy, draining, lost), node, and Slurm job ID, plus a warning banner when those states can no longer be trusted (see below).

Replicas transition through states automatically:

• pending — job submitted, waiting on the Slurm queue for an allocation
• starting — job submitted, not yet health-probed
• healthy — health probe passed; the replica is in the endpoint pool
• lost — job preempted or gone; the endpoint is detached and a new job will be submitted on the next tick
• draining — the replica is being removed on purpose (scale-down, disable, cancel); the row says why

Warmup on promotion

A replica passes its health check as soon as the inference server answers /health, but its first request can still be slow — the model has to do its first forward pass (graph capture, cache warmup), which on a cold box can take long enough to surface to a user as a 502/504. To avoid that, the provisioner fires one throwaway request at each replica right after it is promoted, so the cold first-token cost is paid by the gateway instead of by the first real user.

Warmup is on by default. Tune or disable it with OBLETH_PROVISIONER_WARMUP_TIMEOUT_SECS (default 600s; 0 disables) — see Environment Variables.

Restarting a replica

Each Slurm-backed endpoint has a Restart action on the model's Reliability tab: it cancels that replica's Slurm job, and the provisioner launches a fresh one to hold the target count. Static (non-managed) endpoints are unaffected. Cancelling a replica deregisters its endpoint before the job is killed, so it leaves the routing pool immediately rather than serving a few more requests into a dying process.

Self-healing zombie replicas

A Slurm job can report RUNNING while the inference server inside it is dead or wedged. The provisioner restarts such a replica on either of two independent signals:

1. Its own probe has been failing for a sustained window. Each tick the provisioner probes every healthy replica whose job reports RUNNING. A miss increments a per-replica counter; a pass decays it by one rather than resetting it. Once the counter reaches OBLETH_PROVISIONER_RESTART_AFTER_FAILURES (default 20, 0 disables), the replica is restarted. The decay is what keeps a flapping replica alive: a busy single-threaded server such as llama.cpp misses the occasional probe and passes the next, so its counter hovers near zero and never trips, while a genuinely dead server climbs steadily. At the default 15 s tick, 20 ticks is roughly five minutes of sustained failure.
2. The gateway's endpoint check says the server can't infer. The gateway runs a real one-token inference against each registered endpoint on a slow cadence. Two consecutive unhealthy verdicts on a recent check are enough — this catches the zombie the provisioner's own GET probe cannot see, a server that answers metadata instantly but hangs forever on actual inference.

Restarts are capped at one per model per tick, so a probe-side network problem rolls replicas gradually instead of mass-cancelling a fleet.

A related repair needs no signal at all: a replica marked healthy whose endpoint was removed out of band — a manual delete in the Reliability tab, or a cancellation that only half-landed — used to sit healthy forever while the model ran one endpoint short. The provisioner detects the dangling reference and registers a fresh endpoint on the next tick.

When a launch fails

When slurmrestd rejects a submit — a bad account, partition, or QoS — the reason is stored on the managed spec (last_provision_error) and shown as a banner on the model. Fix the cause and dismiss the banner; it returns on its own if the next launch also fails. Cancel failures are logged with the response body from slurmrestd (for example Access/permission denied from slurm_kill_job2), not just the HTTP status, so causes like a JWT user / job owner mismatch are visible.

The Placement and Service fields on the Provisioning tab are validated in the browser before the save — time-limit format, port range, replica counts, node and CPU numbers — so a malformed value is highlighted inline instead of surfacing as an opaque slurmrestd 500.

Endpoints and the OpenAI /v1 root

When a replica passes its health probe, the provisioner registers it as a model endpoint with api_base = http://<resolved-address>:<discovered-port>/v1 — the node's override or resolved IP, and the port in its window that answered. The /v1 root matches the convention used by every statically-registered model and the model health check, which appends /models to a model's api_base expecting …/v1/models. OpenAI-compatible inference servers (vLLM, SGLang, LiteLLM, Ollama) all serve their OpenAI surface under /v1.

This is distinct from the provisioner's own liveness probe, which uses the spec's health_path (e.g. Ollama's native /api/tags, or vLLM's /health) against the bare node — it does not assume /v1.

How model health is shown for Slurm models

A Slurm-provisioned model has no static api_base — its live URL lives in the endpoint pool, which changes as replicas come and go. So the model-level health badge is derived from the endpoint pool, not from a single static base:

• healthy — at least min_replicas live endpoints are serving
• degraded — some endpoints are serving, but fewer than min_replicas
• unhealthy — no live endpoint is serving, or none is registered
• unknown — every live endpoint came back unverified (a model type that is not auto-probed), so the pool is non-alerting rather than confirmed down

"Serving" counts an endpoint that answered, whether or not the upstream catalog advertised the model id — the provisioner already confirmed the model was up before promoting the replica, so reachability is the right signal here. This is why a Slurm model whose framework renames the id (qwen2.5-0.5b vs qwen2.5:0.5b) is not marked down for it.

A replica can be promoted to healthy (the provisioner reached it) while the model badge still shows unhealthy if the gateway cannot reach the endpoint — the two checks run from different places. See Troubleshooting.

Troubleshooting

A job reaches RUNNING but the replica never becomes healthy

There are two independent health checks, and a replica is only promoted when the provisioner's probe passes:

1. Provisioner probe (promotes the replica). Each tick the provisioner does GET http://<address>:<port><health_path> from wherever the provisioner runs (its container or host — not from the compute node), for every port in the replica's window, concurrently. <address> is the node's resolved address — the override if one is set, otherwise the cached or freshly resolved IP. The replica is promoted on the lowest port that answers 2xx; the outcome is logged either way:

   health probe: healthy          replica_id=... job_id=19359 port=8000
   health probe: not yet healthy  replica_id=... job_id=19359 port_base=8000 port_span=8
   

   not yet healthy almost always means the provisioner cannot reach the node, not that the model is down. A curl that works on the node itself proves nothing — the probe originates from the provisioner. Verify reachability from inside the provisioner container:

   getent hosts gpu7                                              # name resolves?
   curl -s -o /dev/null -w '%{http_code}\n' http://gpu7:8000/api/tags   # port reachable?
   

   If the log instead says job is RUNNING but slurmrestd returned no nodes, that is a different problem — slurmrestd is not returning the allocated nodelist. Check that your API version matches /openapi/v3.

2. Model health check (the red/green badge). This runs in the gateway (obleth) container against the registered endpoints, so the gateway also needs to reach <node>:<serving_port>. This is why a replica can be healthy while the model badge is still unhealthy.

   The model detail Health tab shows the probe history — handy for spotting a model that has just started recovering. In the run below the endpoint was registered before the /v1 fix, so the gateway probes returned unhealthy (HTTP —, 0 ms) until a fresh promotion moved it to degraded:

   

Giving containers a route to cluster nodes

Before reaching for container plumbing, try the node address overrides in Settings → Slurm: they pin each node to an IP inside obleth itself, so both the probe and proxied traffic stop depending on name resolution, and they need no restart.

When the container cannot route to the node at all, fix it at the container level. Both the provisioner and the gateway need to resolve and route to your cluster node hostnames. Add a gitignored deploy/docker/docker-compose.override.yml so no node IPs land in git, and put extra_hosts on both services:

services:
  obleth-provisioner:
    extra_hosts:
      - "gpu7:10.0.0.24"        # real routable IP of the node
  obleth:                        # the model health check + data plane need it too
    extra_hosts:
      - "gpu7:10.0.0.24"

Run the single command from deploy/docker/ (docker compose up -d --build) and the override is auto-merged. extra_hosts is applied at container-create time, so recreate the containers to pick it up (--force-recreate if Compose thinks they are current).

A promoted replica was registered before the /v1 fix

Promotion only creates an endpoint; it never rewrites an existing one. A replica promoted by an older provisioner keeps its pre-/v1 api_base until a fresh promotion. To refresh it, force a new replica — toggle the model's provisioning off and back on, or set target_replicas to 0 then back — so the next promotion registers at the /v1 root.

Disabling Slurm globally vs per model

v1 limitations

• No replica self-registration. A replica does not announce itself; the provisioner discovers it by probing the replica's port window. serving_port is the base of that window, not a fixed port.
• Single-node health probe. The health probe targets the first node of a job. Multi-node nodelist bracket-ranges (e.g. gpu[01-04]) are not expanded.
• No autoscaling. target_replicas is a fixed count. The provisioner only maintains it and replaces preempted jobs — it does not scale based on load.
• Global disable does not drain. Turning off the master switch idles the provisioner without cancelling existing Slurm jobs. Per-model disable does drain.

Management API

Slurm system settings (see also Management API reference):