Launch on Slurm

Start the provisioner

The provisioner is a separate process, obleth-provisioner. It reads the Slurm connection and each deployment from the Management API, submits and cancels jobs through slurmrestd, checks each replica, and adds healthy ones to the model's endpoints. Saving a connection alone launches nothing.

• Docker Compose: add slurm to COMPOSE_PROFILES in deploy/docker/.env, then docker compose up -d --build from deploy/docker/.
• Helm: set provisioner.enabled: true. The chart runs exactly one replica with the Recreate strategy, so two provisioners never submit jobs for the same model; don't scale it.
• systemd: a unit ships at deploy/systemd/obleth-provisioner.service; put its variables in /etc/obleth/provisioner.env.

It needs only OBLETH_ADMIN_BASE_URL and OBLETH_ADMIN_TOKEN; the Slurm connection comes from the dashboard. On a stop signal it finishes the pass in progress, so a submitted job is always recorded.

Connect slurmrestd

On Deployments, choose Set up Slurm on the Slurm card (Slurm connection once it's set up, or open /deployments?slurm=1).

The Cluster defaults tab holds the values recipes refer to, so none of them hard-codes a path:

Test the connection pings slurmrestd and reports the JWT's expiry. The sheet also shows when the provisioner last checked in and whether its passes are failing. Changes reach the provisioner on its next pass, without a restart. Turning Slurm off while jobs run asks first.

Launch a model

1. 1

   Pick what to run

   On Deployments, choose New deployment, then a Saved or Library recipe. Or choose From Hugging Face and enter a repo id: the dashboard reads the repo and lists the engines that can run it, vLLM and SGLang for safetensors, llama.cpp and Ollama for GGUF. A gated repo needs the Hugging Face token from Cluster defaults.

2. 2

   Where and how it runs

   One form holds every setting the launch sends: the partition (each shows whether a replica fits and how many fitting nodes are idle), account, QoS, walltime, nodes, GPUs, CPUs and memory per node (0 is the whole node), Only on nodes with, Avoid nodes, the recipe's inputs, Job logs, replicas and the health check. The API model name is the name clients send and the name forwarded to the server, so one recipe can be launched under several names.

   Each value says where it came from: the recipe, a cluster default, Learned from past launches, or changed by you. Edit as › YAML edits the same settings as text, checked line by line. Only accounts the partition takes are offered, and account, QoS and walltime are checked before submit.

3. 3

   Review and launch

   Check the job script with your values filled in, and what is sent to Slurm. Tick Also save these settings as a recipe to keep them under Saved, then choose Launch.

Launching adds the model to Models, staged, and opens its deployment page. Requests get 503 until a replica is healthy. See Deployments.

When a recipe's recent launches all used one account and QoS, the form starts with them. From past runs suggests more, with the evidence: a longer walltime after jobs hit the limit, a node to avoid, more nodes after running out of memory. Nothing is applied until you pick it.

How obleth keeps each replica running, and what to do when jobs fail, is in Replicas and failures.

Management API+

Method	Path	Does
GET, PUT, DELETE	/api/v1/models/{id}/managed	A model's Slurm spec. DELETE stops launching it
GET	/api/v1/managed	Every spec
GET	/api/v1/models/{id}/replicas, /api/v1/replicas	Replicas
POST	/api/v1/replicas/{id}/restart	Cancel a replica's job; a new one is submitted
POST	/api/v1/models/{id}/replicas/clear-lost	Clear ended jobs (Try once more)
PATCH	/api/v1/models/{id}/managed/provision-error	Dismiss the last refused-job message
GET, PUT	/api/v1/settings/slurm	The connection and cluster defaults (JWT and Hugging Face token masked)
POST	/api/v1/settings/slurm/test	Ping and JWT expiry
GET	/api/v1/deployments/launches	Launch history, newest first; ?model_id=, ?recipe_id=, ?limit= (50, at most 500)
GET	/api/v1/slurm/resources	Partitions with the accounts they allow and deny, nodes and their state, the user's accounts, QoS and associations

A spec set through the API can, instead of a script_body, give an Apptainer image, preamble lines and a launch_command; the provisioner then runs apptainer exec --nv <image> <launch_command>, or the command directly when image is blank. enabled: false pauses it. See Management API.