GGUF quants explained: which file to download and what you give up

What GGUF quantization does to a model

A model is billions of numbers, called weights. Stored in full they take 16 bits each. Quantization stores them with fewer bits, in small blocks that share a scale, so the file and the memory it needs shrink. The price is a small loss of precision, which shows up as slightly worse answers, and the loss grows as the bit count falls.

File size is about the number of weights times the bits per weight (BPW) divided by 8. Small models carry more bits than the name suggests, because some tensors stay at higher precision: a Q4_K_M file cost 4.90 BPW on Llama 3.1 8B and 6.24 on a 0.5B model F70. KoboldCpp prints the figure when it loads a file:

print_info: file type   = (guessed) unknown, may not work1print_info: file size   = 462.96 MiB (6.16 BPW) 2

This is what you download for Llama 3.1 8B at each quant, from the real Hugging Face files:

Which GGUF quant should I download?

The bits per weight below are from the real 8B files (about the same for 12B models, higher for small ones). We have not measured the quality of each step, so the “Use when” column is the common advice, which the calculator turns into a fit check for your card.

Check it against your own card:

koboldcpp --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --gpulayers 33 --contextsize 16384

Quant names decoded

Q4, Q5, Q6, Q8

About how many bits per weight (4 to 8)

_K

A k-quant: weights in blocks, each with its own scale

_S, _M, _L

Small, medium, large: M and L keep some tensors at higher precision

IQ

An i-quant, made with an importance matrix; smaller at the same quality, built for tight fits

_0, _1

Older, simpler formats (for example Q8_0)

F16, BF16

Not quantised: 16 bits per weight

MXFP4

A 4-bit format that some models, such as GPT-OSS, are published in

imatrix, UD, dynamic

Extra tuning of which tensors get more bits; some need a recent KoboldCpp build to load

These are summaries; the exact block layouts are in the llama.cpp quantization documentation.

Our CPU test: smaller is not faster

Memory use follows the file: peak RAM was about the file size plus a fixed overhead F14.

Why would a bigger quant not be slower? Our interpretation, not something we measured: generation on a CPU is limited by how fast the weights can be read from memory, but the smaller formats also need more arithmetic to unpack each weight, and on these machines the two roughly cancel. Do not assume the same on a GPU, where fitting more layers into VRAM is what matters. The benchmark mode shows the same pattern at full context: 97.8 tokens per second processing and 16.3 generating for Q4_K_M, 121.9 and 16.1 for Q8_0 on our Linux VM F16.

Our GPU test

We have not yet run the quant comparison on a physical GPU, so there is no GPU table here. When we have, it will cover Q4_K_M to Q8_0 on an 8B model with every layer on the card: file size, VRAM after load, generation and prompt speed. Until then the rule is the one from memory: a quant that lets every layer fit on the GPU beats a larger one that spills layers into RAM (GPU layers).

How much quality do you lose?

We have not measured it. Five test prompts would be too small a sample for a claim, and we do not publish a percentage. The llama.cpp project publishes perplexity comparisons across quants for its reference models; read those for a larger measurement, and treat Q4_K_M as the usual balance and Q2_K as the point where answers visibly degrade, which users report.

Bigger model at a lower quant, or smaller model at a higher quant?

We have not compared the two on the same prompts. The common advice from users is that a larger model at Q4_K_M usually beats a smaller one at Q8_0 of a similar file size, as long as the quant stays at 4 bits or above. A Mistral Nemo 12B at Q4_K_M is 6.96 GiB and Llama 3.1 8B at Q8_0 is 7.95 GiB: the calculator shows which fits your card and context.

GGUF, GGML, safetensors and other formats

KoboldCpp’s main format is GGUF, supported since 1.41. Legacy GGML .bin files for LLaMA (ggml, ggmf, ggjt), GPT-J, GPT-2, GPT-NeoX, MPT and RWKV still load through older built-in libraries, but newer features may not work with them. A text model that only exists as safetensors or PyTorch must be converted to GGUF and quantised first (Hugging Face guide when it is published). GPTQ, AWQ and EXL2 are formats for other programs and KoboldCpp does not load them as text models (per its README; we did not try one). ONNX is not supported. Image models may be .safetensors or .gguf.

Make your own quant

Convert a Hugging Face model to GGUF with llama.cpp’s convert_hf_to_gguf.py, then quantise it with quantize_gguf from koboldcpp_tools.zip, which is attached to the kcpp_tools_rolling release on the KoboldCpp GitHub page. The full steps will be on the Hugging Face page; we have not tested them yet.

KV cache quantization is a different setting

--quantkv compresses the context memory (the KV cache), not the model file. On our small model at 32K of context, peak RAM went from 976 MB to 794 MB (q8_0) and 698 MB (q4_0) F10. See context size.