Serving GGUF models
Run llama.cpp's llama-server on a Slurm cluster through obleth, from the shipped llama.cpp engine recipe.
3 min readUpdated for v2.4.0Edit on GitHub
llama-server on the compute nodes or in an imagellamacpp.recipe in control-plane/recipes/Launch it
A llama.cpp model is launched like any other model on Slurm: from a recipe, a YAML header plus the sbatch script obleth submits. In New deployment, choose From Hugging Face and enter a GGUF repo: obleth lists its quantizations and offers llama.cpp, with the llama.cpp engine recipe filled in as repo:quant, a mid-sized quantization such as Q4_K_M picked first. You can also launch that recipe from the Engines list on the Recipes tab and type the model yourself, or choose Copy to edit to make it your own. See Launch on Slurm.
obleth itself doesn't fetch weights. The shipped recipe runs llama-server -hf, which downloads the file on the node; a script of your own must have the weights on the cluster, or download them before llama-server starts.
What the script must get right
Three things in the script matter to obleth. Everything else is ordinary llama-server tuning.
llama-server -hf {{model}} \
--host 0.0.0.0 --port "$OBLETH_SERVING_PORT" --alias {{api_model_name}} \
-ngl 99 --ctx-size {{context}} --parallel {{parallel}} --threads "$(nproc)"
- Listen on
$OBLETH_SERVING_PORT. obleth adds a block before your script that picks a free port in the replica's own port window and exports it, so two replicas can share a node. Bind that variable, not a fixed port. - Listen on
0.0.0.0. The provisioner's health probe and the gateway's traffic reach the server from outside the compute node. - Serve the name you launch it as. The API model name you choose at launch is also the name obleth forwards upstream. The recipe passes
{{api_model_name}}to--alias, so it always matches; a script of your own must do the same.
llama-server answers GET /health, the recipe's health path. A replica goes into rotation once that path returns a 2xx.
The recipe also asks for the context length, how many requests run at once (--parallel), a container image (empty runs the llama-server on the node's PATH), and More llama-server flags, such as --n-cpu-moe 60 to keep a mixture-of-experts model's experts in CPU memory.
Keep weights off a slow shared filesystem
Large GGUF files read slowly over NFS. The recipe sets LLAMA_CACHE to its Weight cache input, which defaults to the cluster's weight cache. Point it at node-local scratch at launch so llama-server -hf pulls the weights onto fast local storage.
To stage the files yourself, download them before the launch line and point llama-server at the copy:
HF_REPO=example-org/Example-Model-GGUF
HF_DIR=Q4_K_M
STAGE=/mnt/job_tmp/example/$HF_DIR; mkdir -p "$STAGE"
FILES="Example-Model-Q4_K_M-00001-of-00002.gguf Example-Model-Q4_K_M-00002-of-00002.gguf"
for f in $FILES; do
curl -fsSL -C - -o "$STAGE/$f" \
"https://huggingface.co/$HF_REPO/resolve/main/$HF_DIR/$f" &
done; wait
curl -C - resumes a partial download, so a job that restarts on the same node doesn't fetch again what it already has.
A job runs on whichever node Slurm allocates, so weights on node-local scratch are fetched again after a job moves. Weights on shared storage are read once per job either way: the trade is transfer time against read speed while serving.
Placement
The recipe asks for one GPU, 16 CPUs and 64G of memory in its #SBATCH lines, and leaves partition, QoS and walltime out so it fits any cluster. You choose those in the second step of New deployment, which also shows whether a replica fits each partition; raise CPUs and memory there for a large model.
After launch
Set the model's Quantization to gguf on its page, so GET /v1/models reports the format and the name doesn't have to carry it; add any old, suffixed spelling as an alias. See Register a model. The model starts staged: make it active once it answers. To change the script later, edit it under Script on the deployment's page: replicas started after you save use it, and Restart on a replica picks it up now.