Fix "out of memory" errors in KoboldCpp

The interactive memory calculator is built with the memory calculator page; until then, the formula and measured sizes below let you work it out.

KoboldCpp out of memory: which message do you have?

We could only reproduce one of these ourselves. Our lab has no GPU with a limited memory, and forcing small Vulkan limits did nothing on the software GPU F54, so the GPU messages are reported by users, not tested. The one we reproduced ran out of system RAM:

ggml_backend_buft_alloc_buffer_n_default: failed to allocate CPU buffer of size 64455966721llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache2gpttype_load_model: error: failed to load model '<models>/qwen2.5-0.5b-instruct-q4_k_m.gguf'3

What fills your memory

Four things share your memory: the model weights, the KV cache, the compute buffer and any extra models. This is Llama 3.1 8B Q4_K_M with all 33 layers on the GPU, from allocations we logged F36. On an 8 GB card, 8K context fits and 32K does not:

Values are MiB measured from KoboldCpp’s own log lines (the card size is nominal; some memory is used by your system). A further 692.8 MiB token table and 266.5 MiB of buffers stay in system RAM.

• Weights are about the file size: 3,992.5 MiB for a 4.58 GiB file once the token table stays on the CPU.
• KV cache grows in a straight line with context. The formula matched our logged sizes exactly: 51, 195 and 1,539 MiB at 4K, 16K and 128K on our 0.5B model F10, and 1,056 and 4,128 MiB on the 8B model F36. KoboldCpp adds 256 tokens to the context you ask for F06.
• Compute buffer depends on batch size and the model: 124.3 MiB at 8K and 148.3 MiB at 32K here.

Generation speed stayed within ±10% from 2K to 128K F11. RAM numbers from a CPU lab say nothing about VRAM; the shape is what matters.

Out of memory while loading

Work through these in order and change one thing at a time. After each change, reload and check the KV buffer size line.

1. Lower the context size. The default is 16384 F05. On the 8B model, 8K needs 1,056 MiB of cache, 32K needs 4,128 MiB (calculated from measured allocations). See context size.
2. Quantise the KV cache with --quantkv q8_0 or q4_0. In our CPU lab, peak RAM at 32K went from 976 to 794 to 698 MB F10. We have not yet measured GPU savings or quality.
3. Let autofit choose the layers. --gpulayers -1 is the default and leaves a margin: --autofitpadding is 1024 MB in our logs. Or set fewer layers by hand: GPU layers.
4. Use a smaller quant: quantization.
5. Lower the batch size (--batchsize, default 512). We have not measured the memory saved.
6. Keep the cache in RAM with --lowvram. It is slower.

To undo any of these, remove the flag or reset the launcher to its defaults.

Out of memory when the chat gets long

The cache fills as the chat grows. If the context is set higher than your card can hold, the failure comes only after many turns. Users report illegal memory access crashes with a quantised cache and ContextShift (#2088 with 60 comments, #1469, #1412); we have not reproduced them. Lower the context, return to --quantkv f16, and update, because several of these were fixed in later releases. See context size and illegal memory access.

Image, vision and voice models need memory too

A vision projector, a Whisper model, a TTS model and an image model each take their own memory. Move them to the CPU where KoboldCpp allows it, for example --mmprojcpu, or leave them out while you load a large model. Users report that autofit accounts for loaded extras (#2471); we have not confirmed it. See image generation, vision and voice.

Windows: “shared GPU memory” instead of an error

On Windows, the NVIDIA driver can let a model spill into system RAM instead of failing. There is no error, but speed drops sharply. Users report this on #1413 (25 comments). We have not measured it, because we have no NVIDIA GPU. The driver setting “CUDA - Sysmem Fallback Policy” can be set to “Prefer No Sysmem Fallback”, which gives a clear error instead of a slow run (users report).

Out of system RAM (not VRAM)

Layers that are not on the GPU live in system RAM, and so does the weights file on a CPU start. The log prints Detected Available RAM: … MB, for example 15,282 MB on our lab machine. With an enormous context, the cache itself can fail to fit: a 524,288-token context needs 6,147 MiB of KV cache even on the 0.5B model, and the load failed with the lines above F55. Close other programs, lower the context, or use a smaller file. The mmap option is off by default, and our test shows turning it on lowers only the reported working set on Windows F12.

Mac: unified memory limits

Metal can use only part of the unified memory, which users report as about 65 to 75% by default (#24); we have not measured it. Our macOS runner is a virtual machine, so its speeds say nothing about a real Mac F29. See the Mac guide.

Several GPUs

If one card runs out while another has room, set a tensor split with --tensor_split. Users report this on #1882. See multi-GPU.

Not sure what is wrong? Start the decision tree. Slow, not failing? Speed.