Getting GGUF models from Hugging Face into KoboldCpp

Can KoboldCpp run safetensors?

Not text models. KoboldCpp reads text models as GGUF only, and when we pointed it at a safetensors file it stopped with invalid or unsupported model file, does not seem to be a GGUF file and exit code 3 F82. Image models are different: upstream accepts .safetensors or .gguf for them. If a model has no GGUF version yet, see converting.

Find the GGUF version of a model

1. Open the original model page

   Start where you heard of the model.
2. Look for its GGUF version

   The model card often links to quantized versions, or search Hugging Face for the model name plus “GGUF”.
3. Prefer the publisher's own repo

   Then a widely used quantizer that the model card names.
4. Check the licence and whether the repo is gated

   Gated repos need a Hugging Face login.
5. Open Files and versions

   That page lists the files you can download.

Pick one file

A GGUF repository lists many files, one per quantization. You need exactly one. The name ends in a code such as Q4_K_M and the size is shown next to it: choose the biggest that fits in your memory, and start with Q4_K_M (what the codes mean; check the fit). A vision model also needs its matching mmproj file (vision).

KoboldCpp Huggingface search: the HF Search button

The launcher’s Quick Launch tab has an HF Search button beside the Text Model field (since 1.91, upstream). It searches Hugging Face for GGUF files and fills in the model field with the download link, which KoboldCpp then fetches at launch. We have no display here, so we could not run the dialog; the download behaviour behind it is in the next section.

Load a model straight from a link

--model accepts a link to a GGUF file (since 1.66) and several links for a split model (1.84). We tested the single-link case against a local web server F82:

• KoboldCpp printed Downloading <link> and saved the file as tiny-Q8_0.gguf in the folder given to --downloaddir.
• The model name reported by the API was koboldcpp/tiny-Q8_0.
• A second start with the same command printed already exists, using existing file and downloaded nothing.

Build the command in the command builder and put the link in the model field.

Split GGUF files

Very large models are published as parts named name-00001-of-00003.gguf. Keep all parts in one folder and point KoboldCpp at part 1. We tested this by splitting a model ourselves with the project’s gguf-split tool: part 1 loaded all three parts; part 2 alone failed with model must be loaded with the first split; with part 3 missing it failed with failed to open GGUF file and the part’s name F97. Our fit calculator reads the parts together (calculator). To build the tools yourself see building from source.

Check the download before loading

Compare the size with the one on Hugging Face, and the SHA-256 shown on the file’s page (how to check a hash). --analyze <file> prints the GGUF header (version, tensor count, keys) and tells you at once if the file is a model F82. A bad file gives one of three messages we reproduced:

Cannot find text model file: <path>1

load_model: error: invalid or unsupported model file '<path>'1Could not load text model: <path>

llama_model_load: error loading model: tensor 'output.weight' data is not within the file bounds, model is corrupted or incomplete1

More in model won’t load.

Convert a safetensors model to GGUF

If a model has no GGUF version, convert it in two steps (upstream; we did not run the conversion): turn the safetensors files into a full-size GGUF with llama.cpp’s convert_hf_to_gguf.py, then shrink it with quantize_gguf from koboldcpp_tools.zip on the release page. It needs Python and a lot of free disk space, and most popular models already have a GGUF, so look first.

Where models are stored and how to delete them

KoboldCpp keeps no model library: a model is the file you chose, or the file saved in --downloaddir, or the working folder. To remove one, delete the file; nothing else is installed. Keep models outside the folder you run KoboldCpp from, so an update does not touch them.