64 docs indexed

Serving GGUF Models

Point the llama.cpp backend at single-file or sharded GGUF weights, use the Models directory setting to pre-fill the path, and optionally stage large shards to node-local NVMe before the job starts.

The llama.cpp backend is handed a file on disk — unlike vLLM (which takes a Hugging Face repo id) or Ollama (which takes a model name). The launcher's Model path field accepts three forms:

  • a single .gguf file — used as-is;
  • the first shard of a split model (…-00001-of-00006.gguf) — llama.cpp auto-loads the remaining shards by the -NNNNN-of-MMMMM naming pattern;
  • a directory — obleth resolves the first shard (or a lone .gguf) inside it at job start. Use this when you do not want to spell out the shard filename.

Pre-filling the model path

Set a cluster Models directory under Settings → Slurm. obleth pre-fills the Model path field with that root whenever you create or edit a llama.cpp model, saving you from retyping the prefix each time.

For example, if your shared filesystem keeps weights under /shared/models, set that as the Models directory. You can then enter a subdirectory or filename relative to that root — or type an absolute path to override it entirely.

See the Slurm Provisioning guide for the full Settings → Slurm field reference.

Downloading weights at job start (optional)

obleth does not fetch model weights for you — they must already exist on the cluster before the job runs. If your shared filesystem is slow for large sequential reads (common with NFS on multi-hundred-GB GGUF files), you can download the weights to node-local NVMe inside the job by pasting a snippet into the launcher's Extra preamble field.

The snippet runs before apptainer exec, so by the time llama.cpp starts the weights are already on fast local storage.

# Download GGUF shards from Hugging Face to node-local NVMe, then point -m at them.
HF_REPO=unsloth/GLM-5.2-GGUF       # repo id
HF_DIR=UD-IQ2_M                    # quant subfolder
BASENAME=GLM-5.2-UD-IQ2_M          # shard filename stem
SHARDS=6                            # number of shards
STAGE=/mnt/job_tmp/glm52/$HF_DIR; mkdir -p "$STAGE"
WIDTH=$(printf '%05d' "$SHARDS")
for s in $(seq 1 "$SHARDS"); do
  n=$(printf '%05d' "$s")
  f="${BASENAME}-${n}-of-${WIDTH}.gguf"
  curl -fsSL -C - -o "$STAGE/$f" \
    "https://huggingface.co/$HF_REPO/resolve/main/$HF_DIR/$f" &
done; wait

After the snippet runs, set the launcher's Model path to the staging directory (/mnt/job_tmp/glm52/UD-IQ2_M) and obleth resolves the first shard automatically.

Adapt the variables at the top of the snippet for your model. The -C - flag resumes a partial download if the job was preempted and restarted with the same local storage intact.