67 docs indexed

Serving GGUF Models

Run llama.cpp on a Slurm cluster with obleth: the shipped llama.cpp recipe, what its launch script has to do about the serving port and model name, and how to keep large GGUF weights off a slow shared filesystem.

llama.cpp is handed weights — a Hugging Face repo to pull, or files already on disk — rather than a model name. obleth does not fetch weights for you: what the job needs must already exist on the cluster, or be downloaded by the job script itself before llama-server starts.

A llama.cpp model is deployed like any other Slurm-provisioned model, from a deployment recipe: a small YAML header carrying the routing metadata, plus the raw sbatch script obleth submits verbatim. See Slurm Provisioning — Step 3 for how recipes are listed, overridden, and deployed. The repo ships one to start from at control-plane/recipes/glm-5.2-multiuser.recipe.

What the launch script has to get right

Three things in the script are load-bearing for obleth. Everything else is ordinary llama-server tuning.

Listen on $OBLETH_SERVING_PORT

The provisioner gives each replica a disjoint port window — serving_port + i × OBLETH_PORT_SPAN (span defaults to 8) — and prepends a bash block to the submitted script that binds the first free port in that window and exports it as OBLETH_SERVING_PORT. It then probes every port in the window to find where the server actually came up. Bind that variable, not a hardcoded port, and two replicas can share a node without colliding:

llama-server -hf unsloth/GLM-5.2-GGUF:UD-IQ2_M \
  --host 0.0.0.0 --port "$OBLETH_SERVING_PORT" --alias glm-5.2 \
  -ngl 99 --ctx-size 1048576 --parallel 4 --threads "$(nproc)"

--host 0.0.0.0 matters too: the provisioner's health probe and the gateway's proxied traffic both reach the server from outside the compute node.

Serve the name clients will ask for

Deploying a recipe registers the model route with upstream_model set to the header's api_model_name, so that exact string is what obleth forwards upstream. Keep llama-server --alias equal to the header's api_model_name (glm-5.2 in the shipped recipe) or the upstream will not recognise the model it is asked for.

Answer the health path

The provisioner promotes a replica only once GET <health_path> returns a 2xx. The header's health_path wins when set; otherwise the default is /health, or / for the ollama engine. llama-server serves /health, so the shipped recipe leaves it unset.

Placement comes from the header and the overrides

#SBATCH directives in the body are lifted into the submit fields, because slurmrestd ignores them in the script. Header fields override the directives, and the deploy dialog's Deploy overrides (partition, QoS, replicas, time limit, plus any variables the recipe declares) override both — for that deployment only. The shipped llama.cpp recipe deliberately leaves partition, QoS, and time limit out of the script so it stays cluster-agnostic and you supply them at deploy time.

Keeping weights off a slow shared filesystem

Large GGUF models read poorly over NFS. The shipped recipe points LLAMA_CACHE at node-local scratch, so llama-server -hf pulls the weights onto fast local storage:

# EDIT FOR YOUR CLUSTER: node-local scratch path for weight caching.
export LLAMA_CACHE=/path/to/job_tmp/glm52

If you would rather stage the files yourself, the script is plain bash — put the download before the launch line and point llama-server at what you staged:

HF_REPO=unsloth/GLM-5.2-GGUF
HF_DIR=UD-IQ2_M
STAGE=/mnt/job_tmp/glm52/$HF_DIR; mkdir -p "$STAGE"
FILES="GLM-5.2-UD-IQ2_M-00001-of-00006.gguf GLM-5.2-UD-IQ2_M-00002-of-00006.gguf"
for f in $FILES; do
  curl -fsSL -C - -o "$STAGE/$f" \
    "https://huggingface.co/$HF_REPO/resolve/main/$HF_DIR/$f" &
done; wait

curl -C - resumes a partial download, so a preempted job that restarts on the same node with its local storage intact does not re-fetch what it already has.

Note:

The job runs on whatever node Slurm allocates, so anything written to node-local scratch is fetched again after a reschedule. Weights on shared storage are read once per job either way — the trade is transfer time against read throughput during serving.

Editing the spec after deployment

The model's Provisioning tab shows the submitted script in an editable Launch script box, alongside the placement and service fields. Saving relaunches replicas with the new settings. A recipe-deployed model keeps its script as the spec's script_body, so the Apptainer image, preamble, and launch-command fields that the hand-filled path uses do not apply and are hidden.