Slurm Provisioning

The Slurm provisioner is an optional plugin service — obleth-provisioner — that keeps managed models alive on a preemptible Slurm cluster. It submits batch jobs via slurmrestd, probes replicas for health, and promotes each healthy replica into obleth's model endpoint pool so the gateway routes traffic to it automatically.

The provisioner is not part of the obleth core stack. You opt in via config — add slurm to COMPOSE_PROFILES in your .env — after which it builds and starts with the same single command as the rest of the stack (or run the binary yourself).

How it works

Every reconcile tick (OBLETH_PROVISIONER_INTERVAL_SECS, default 15s):

1. Read Slurm settings from the Management API. If Slurm is not globally enabled, the tick is skipped.
2. List enabled managed models via the Management API.
3. List owned Slurm jobs from slurmrestd, identified by a job-name prefix.
4. For each managed model: probe starting replicas, submit new jobs when below target, promote healthy replicas into the endpoint pool, mark lost replicas (preempted/finished), cancel excess jobs.
5. Drain models that left the managed set — cancel jobs, detach endpoints, GC rows.
6. Cancel orphan jobs — any cluster job matching the prefix with no tracking row.

If either the Management API or slurmrestd is unreachable the tick bails out without taking any destructive action; it retries on the next tick.

Prerequisites

Before enabling the provisioner, verify that your cluster has:

• slurmrestd running and reachable from the provisioner. The provisioner authenticates with X-SLURM-USER-NAME + X-SLURM-USER-TOKEN headers; no username/password login is used.
• apptainer (or singularity) installed on all compute nodes. Jobs run as apptainer exec --nv <image> <launch_command>. The --nv flag passes NVIDIA GPU devices through; Apptainer must be able to find it on the job's PATH (/opt/apptainer/bin and /opt/singularity/bin are included in the job's default PATH automatically).
• A Slurm JWT for authentication. Generate one on your Slurm head node with scontrol token, or via your cluster's token service. Paste the raw JWT into the dashboard — it is encrypted at rest with the gateway's OBLETH_ENCRYPTION_KEY.
• Shared storage accessible by all nodes. The Apptainer .sif image file must be reachable by all nodes that could run the job (NFS, Lustre, GPFS, etc.). The path you give in the image field is used as-is on every compute node.

Step 1: Start the provisioner

Docker Compose

The provisioner is gated behind the slurm profile. Opt in via config by adding slurm to COMPOSE_PROFILES in deploy/docker/.env (copy it from .env.example if you haven't):

# deploy/docker/.env
COMPOSE_PROFILES=benchmark,edge,observability,slurm

Then one command — run from deploy/docker/, with no -f flags — builds and starts everything, the provisioner included:

cd deploy/docker

docker compose up -d --build      # build + start the whole stack (provisioner too)
docker compose down               # tear it all down

Running from deploy/docker/ lets Compose auto-load .env (so the profile activates) and auto-merge a local docker-compose.override.yml if you use one (see Troubleshooting). No --profile flag and no extra -f arguments are needed. The container only needs OBLETH_ADMIN_TOKEN (inherited from the core stack's .env) and OBLETH_ADMIN_BASE_URL; Slurm connection details are configured in the dashboard, not in .env.

systemd

A unit file ships at deploy/systemd/obleth-provisioner.service. Put env vars in /etc/obleth/provisioner.env, then:

sudo cp deploy/systemd/obleth-provisioner.service /etc/systemd/system/
sudo cp obleth/target/release/obleth-provisioner /usr/local/bin/
sudo systemctl daemon-reload
sudo systemctl enable --now obleth-provisioner

Cargo (development)

OBLETH_ADMIN_TOKEN=dev-admin-token \
OBLETH_ADMIN_BASE_URL=http://localhost:9180 \
cargo run -p obleth-provisioner

Step 2: Configure Slurm system settings

Open the dashboard → Settings → Slurm.

After saving, click Test connection to verify:

• JWT health — whether a JWT is set, whether it is expired, and when it expires.
• slurmrestd ping — whether the GET /slurm/{version}/ping endpoint responds with 2xx and how fast.

The provisioner reads these settings from the Management API on every tick, so changes take effect within one interval without a restart.

The Settings → Slurm tab also shows whether the provisioner process is currently running and, when the provisioner reports it, its version (and short commit). Because the provisioner ships as its own image and is deployed separately from the gateway, this makes a stale provisioner deployment obvious at a glance.

Step 3: Create a Slurm-provisioned model

When creating a model in the dashboard → Models → New model, choose Slurm provisioned as the hosting mode. This choice is made at creation time — you cannot change it after creation.

Slurm-provisioned models have no static upstream endpoint. The provisioner registers replicas dynamically as they become healthy, so you do not provide api_base or api_key during creation.

Apptainer image requirements

The provisioner submits a Slurm batch script that runs:

#!/bin/bash
set -euo pipefail
apptainer exec --nv <image> <launch_command>

• <image> — the path to an Apptainer .sif image on shared cluster storage. This path is used verbatim on each compute node, so all nodes must be able to read it.
• <launch_command> — the inference server startup command to run inside the container, e.g. python -m vllm.entrypoints.openai.api_server --model /data/models/llama3 --port 8080.
• --nv — passes NVIDIA GPU devices through to the container. Remove this flag from launch_command if you are not using GPUs.

Building an Apptainer image for vLLM:

# Pull the vLLM Docker image and convert it to a .sif
apptainer pull vllm.sif docker://vllm/vllm-openai:latest

# Or build from a definition file for a custom environment
apptainer build vllm-custom.sif vllm.def

Place the .sif on your cluster's shared filesystem (NFS, Lustre, GPFS) and use that path as the image field in the model spec.

Image checklist:

• The inference server must listen on the port you set in serving_port.
• It must respond to HTTP GET <health_path> with a 2xx when healthy (used by the provisioner's health probe).
• GPU drivers are injected at runtime by --nv; you do not need to bundle CUDA in the image (though bundling it is fine too).
• If your cluster uses a module system (module load apptainer), make sure apptainer is on the default PATH for batch jobs, or prepend the load command to launch_command.

Managed model spec fields

The spec is set via the Provisioning tab on the model detail panel (or directly via the Management API PUT /api/v1/models/{id}/managed).

Monitoring replicas

On the model detail panel, the Provisioning tab appears only for Slurm-provisioned models. It shows:

• Edit provisioning spec — change partition, GRES, image, launch command, replica count, etc.
• Replica panel — live table of each replica: state (starting, healthy, lost), node, Slurm job ID.

Replicas transition through states automatically:

• starting — job submitted, not yet health-probed
• healthy — health probe passed; the replica is in the endpoint pool
• lost — job preempted or gone; the endpoint is detached and a new job will be submitted on the next tick

Warmup on promotion

A replica passes its health check as soon as the inference server answers /health, but its first request can still be slow — the model has to do its first forward pass (graph capture, cache warmup), which on a cold box can take long enough to surface to a user as a 502/504. To avoid that, the provisioner fires one throwaway request at each replica right after it is promoted, so the cold first-token cost is paid by the gateway instead of by the first real user.

Warmup is on by default. Tune or disable it with OBLETH_PROVISIONER_WARMUP_TIMEOUT_SECS (default 600s; 0 disables) — see Environment Variables.

Restarting a replica

Each Slurm-backed endpoint has a Restart action on the model's Reliability tab: it cancels that replica's Slurm job, and the provisioner launches a fresh one to hold the target count. Static (non-managed) endpoints are unaffected. Cancelling a replica deregisters its endpoint before the job is killed, so it leaves the routing pool immediately rather than serving a few more requests into a dying process.

Endpoints and the OpenAI /v1 root

When a replica passes its health probe, the provisioner registers it as a model endpoint with api_base = http://<node>:<serving_port>/v1. The /v1 root matches the convention used by every statically-registered model and the model health check, which appends /models to a model's api_base expecting …/v1/models. OpenAI-compatible inference servers (vLLM, SGLang, LiteLLM, Ollama) all serve their OpenAI surface under /v1.

This is distinct from the provisioner's own liveness probe, which uses the spec's health_path (e.g. Ollama's native /api/tags, or vLLM's /health) against the bare node — it does not assume /v1.

How model health is shown for Slurm models

A Slurm-provisioned model has no static api_base — its live URL lives in the endpoint pool, which changes as replicas come and go. So the model-level health badge is derived from the endpoint pool, not from a single static base:

• healthy — at least one live endpoint is healthy
• degraded — no healthy endpoints, but at least one is degraded (e.g. reachable but not yet advertising the model)
• unhealthy — every live endpoint is down, or there are no live endpoints

A replica can be promoted to healthy (the provisioner reached it) while the model badge still shows unhealthy if the gateway cannot reach the endpoint — the two checks run from different places. See Troubleshooting.

Troubleshooting

A job reaches RUNNING but the replica never becomes healthy

There are two independent health checks, and a replica is only promoted when the provisioner's probe passes:

1. Provisioner probe (promotes the replica). Each tick the provisioner does GET http://<node>:<serving_port><health_path> from wherever the provisioner runs (its container or host — not from the compute node). The replica is promoted only on a 2xx. Every attempt is logged:

   health probe ... api_base=http://gpu7:8000 healthy=false
   

   healthy=false almost always means the provisioner cannot reach the node, not that the model is down. A curl that works on the node itself proves nothing — the probe originates from the provisioner. Verify reachability from inside the provisioner container:

   getent hosts gpu7                                              # name resolves?
   curl -s -o /dev/null -w '%{http_code}\n' http://gpu7:8000/api/tags   # port reachable?
   

   If the log instead says job is RUNNING but slurmrestd returned no nodes, that is a different problem — slurmrestd is not returning the allocated nodelist. Check that your API version matches /openapi/v3.

2. Model health check (the red/green badge). This runs in the gateway (obleth) container against the registered endpoints, so the gateway also needs to reach <node>:<serving_port>. This is why a replica can be healthy while the model badge is still unhealthy.

   The model detail Health tab shows the probe history — handy for spotting a model that has just started recovering. In the run below the endpoint was registered before the /v1 fix, so the gateway probes returned unhealthy (HTTP —, 0 ms) until a fresh promotion moved it to degraded:

   

Giving containers a route to cluster nodes

When you run the stack in Docker on a workstation, both the provisioner and the gateway need to resolve and route to your cluster node hostnames. Add a gitignored deploy/docker/docker-compose.override.yml so no node IPs land in git, and put extra_hosts on both services:

services:
  obleth-provisioner:
    extra_hosts:
      - "gpu7:10.0.0.24"        # real routable IP of the node
  obleth:                        # the model health check + data plane need it too
    extra_hosts:
      - "gpu7:10.0.0.24"

Run the single command from deploy/docker/ (docker compose up -d --build) and the override is auto-merged. extra_hosts is applied at container-create time, so recreate the containers to pick it up (--force-recreate if Compose thinks they are current).

A promoted replica was registered before the /v1 fix

Promotion only creates an endpoint; it never rewrites an existing one. A replica promoted by an older provisioner keeps its pre-/v1 api_base until a fresh promotion. To refresh it, force a new replica — toggle the model's provisioning off and back on, or set target_replicas to 0 then back — so the next promotion registers at the /v1 root.

Disabling Slurm globally vs per model

v1 limitations

• Fixed serving port per model. All replicas of a model serve on the same port. There is no replica self-registration yet.
• Single-node health probe. The health probe targets the first node of a job. Multi-node nodelist bracket-ranges (e.g. gpu[01-04]) are not expanded.
• No autoscaling. target_replicas is a fixed count. The provisioner only maintains it and replaces preempted jobs — it does not scale based on load.
• Global disable does not drain. Turning off the master switch idles the provisioner without cancelling existing Slurm jobs. Per-model disable does drain.

Management API

Slurm system settings (see also Management API reference):