How many GPU layers should you use in KoboldCpp?

How many KoboldCpp GPU layers fit?

Pick your model, card and context. The result says whether everything fits, how many layers go on the GPU, and what to type in KoboldCpp.

koboldcpp --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --gpulayers 33 --contextsize 16384

For any other model, a Hugging Face link, a local file, or the extra options, use the full calculator. All sizes come from the model’s own file header, so they are exact for the file you pick. The compute buffers are estimates.

What a GPU layer is

A language model is a stack of identical blocks, called layers. An 8B model has 32 of them. Each layer’s weights live either in video memory (VRAM) or in system RAM. A layer on the GPU is computed by the GPU, a layer in RAM is computed by the CPU, and the slowest part sets the pace for every word.

VRAM holds more than layers. It also holds the KV cache (the memory of the conversation, which grows with the context you ask for), a compute buffer for working space, and any other model you loaded, such as a vision projector or an image model. The layers get what is left.

The model’s weights are the biggest block, the KV cache is the one you control with the context size, and the free space is what autofit keeps as margin and what you could use for more context. Two things stay in system RAM even when every layer is on the GPU: the token-embedding table and, in our runs, the output matrix F63.

What the layer number means

KoboldCpp counts one more layer than the model has. In a sweep of a 24-layer model, --gpulayers 12 moved 11 repeating layers to the device, and only --gpulayers 25 (24 + 1) or more, or -1, moved all 24 F65. The table shows the sizes KoboldCpp logged. These runs used a software Vulkan device in our lab, so they show where memory goes, not how fast it is.

The calculator knows this: it prints layers + 1 in the launch command and the readout counts repeating layers. If you only want everything on the GPU, -1 or any number above the layer count does the same.

Step 1: Choose a GPU backend

The backend decides which engine KoboldCpp uses. In the launcher, open Quick Launch and set Backend:

• NVIDIA: Use CUDA. Command line: --usecuda.
• AMD, Intel, or NVIDIA with the smaller file: Use Vulkan. Command line: --usevulkan.
• Mac: Metal is automatic.
• Use CPU puts nothing on the GPU, whatever the layer number says.

Close

The shot is from a test PC without a GPU, which is why Backend reads “Use CPU” F30. On a PC with a supported GPU the list also offers CUDA or Vulkan. The Hardware tab holds the GPU layers box and the autofit fields used below:

Close

Step 2: Leave layers on auto and launch

Leave GPU Layers at -1, which is the default --gpulayers. With -1 KoboldCpp runs autofit: it measures free VRAM, counts the context you asked for plus 256 tokens F06, counts any other model you loaded, keeps 1024 MB free, and chooses the layers. On Quick Launch and Hardware the launcher shows this as Force AutoFit, and Autofit Padding (MB) sets the margin (--autofit, --autofitpadding).

Automatic layer choice has changed over the years, so older guides disagree with the current program:

• 1.71: --gpulayers -1 first estimates the layers.
• 1.72: the launcher defaults GPU layers to -1, and the estimate counts image and Whisper models.
• 1.75: the default is -1. Before that it was 0, which is why old guides tell you to type a number.
• 1.104: --autofit appears.
• 1.108: -1 turns on autofit by itself, the layer box is hidden, and --autofitpadding is added.

In our log, a run with -1 printed the reserve and the result F65:

Unable to detect VRAM, but autofit may still be used if applicable.1Autofit Reserve Space: 1024 MB2Autofit Success: 1, Autofit Result: -c 4224 -ngl -13load_tensors: offloaded 25/25 layers to GPU4

We have not yet measured autofit against the largest layer count that works by hand on a physical card, so we cannot say how conservative it is on a real GPU. People on forums report that it can leave VRAM unused (one report: 7 of 41 layers on a card that could hold more). The next steps explain how to find out whether yours does.

Step 3: Check where the model really runs

This is the step that matters. The console prints a line such as offloaded 25/25 layers to GPU. That line counts the layers assigned to the GPU path. It does not prove that the GPU holds them. In our test with no GPU at all, with --gpulayers 99, KoboldCpp printed offloaded 25/25 layers to GPU and kept everything on the CPU F17. The lines that tell you where memory really is end in buffer size.

load_tensors: offloaded 25/25 layers to GPU1llama_kv_cache:        CPU KV buffer size =    51.00 MiB2

load_tensors: offloaded 25/25 layers to GPUload_tensors:      Vulkan0 model buffer size =   235.76 MiB1llama_kv_cache:    Vulkan0 KV buffer size =    51.00 MiB2sched_reserve:    Vulkan0 compute buffer size =    39.77 MiB3

The second log comes from a software Vulkan device, not a physical card, because we do not yet have a real GPU in the lab. The line format is the same on real GPUs, but we have no real CUDA log yet.

NVIDIA, CUDA

Lines start with CUDA0 (users report; we have no CUDA log yet)

Vulkan

Lines start with Vulkan0 (tested, F35)

Apple Metal

Lines start with MTL0 (users report)

In system RAM

Lines start with CPU, or CPU_Mapped

If KoboldCpp prints No GPU or CPU backend was selected…, Unable to detect VRAM. and Auto Selected Default Backend (flag=0), it chose the CPU for you F18. Go to Not using the GPU.

Or paste your log and get a verdict:

Step 4 (optional): Tune by hand

Only bother if, after a long prompt, more VRAM is free than your margin. Check VRAM while the model reads a long prompt, because the compute buffer grows during prompt processing. Then, in this order:

1. Lower the padding. Set Autofit Padding (MB), or --autofitpadding, to a smaller number, for example half. This keeps the automatic behaviour.
2. Set the layers. Type a number in --gpulayers from the calculator, then raise it a few layers at a time, each time loading the model and sending a long prompt.

We have not yet measured the step size and margin that are safe on a physical card, so there is no rule of thumb here. The calculator’s margin is the same 1024 MB autofit uses. Remember the extra layer in KoboldCpp’s count (above).

Layers versus speed

Every layer left on the CPU slows the whole answer, so the last layers you add matter most. We have not measured the curve on a physical GPU yet, so we do not publish one: our software Vulkan device is not a GPU and its speeds mean nothing. What we did measure is the CPU end: Llama 3.1 8B on four CPU cores generated 5.3 tokens per second F37, which is reading speed. A card that holds all layers is much faster, and partial offload sits between the two. When we have a real card, this section gets the layers-versus-speed chart.

The Windows shared GPU memory trap

On Windows, NVIDIA’s driver can spill video memory that does not fit into system RAM (“Shared GPU memory”) instead of failing. A model that is slightly too big then loads and becomes very slow. Users report this often, and the same on AMD (issue #1413). We have not measured it. To see whether it is happening, watch Shared GPU memory next to Dedicated GPU memory in Task Manager while the model generates.

Users report that the setting is in the NVIDIA Control Panel, Manage 3D settings, CUDA - Sysmem Fallback Policy, and that choosing “Prefer No Sysmem Fallback” turns the spill into an out-of-memory error. That is the point: you then lower the layers or the context. We have not seen the menu ourselves, so check the wording on your driver version. To undo, set the policy back to the driver default.

Fit more on the GPU

In rough order of how much they help. The calculator shows the saving for your own model.

1. Lower the context. The KV cache is linear in context F10. See context size.
2. Quantise the KV cache with --quantkv q8_0 or q4_0. At 32K of context on our small model, peak RAM fell from 976 MB to 794 MB and 698 MB F10. Flash attention is on by default F09, which the quantised cache needs.
3. Use a smaller quantisation of the same model.
4. Keep the vision projector on the CPU with --mmprojcpu.
5. Unload models you do not need: image, speech and text-to-speech models take VRAM.
6. Lower the batch size with --batchsize; the compute buffer grows with it.
7. Keep the KV cache in RAM with --lowvram (the “No KV offload” box). More layers fit on the GPU, and generation slows down.
8. Close other programs that use the GPU.

MoE models: keep experts in RAM

A mixture-of-experts model such as Qwen3-30B-A3B holds many expert networks but uses only a few for each word, so the experts can sit in system RAM while the attention layers stay on the GPU. --moecpu keeps the expert weights of the first N layers in RAM (all layers if you give no number). --ffncpu does the same for dense feed-forward weights, --moe-cache-mib keeps a VRAM cache for experts that otherwise live on the CPU, and --overridetensors gives full control. The launcher fields are “MoE CPU Layers”, “FFN CPU Layers” and “MoE Cache MiB”. Autofit can also set MoE placement by itself.

For a 30B-A3B model at Q4_K_M on a 12 GB card, the calculator shows a partial fit without --moecpu and a full fit of the attention layers with it. We have not measured MoE speed yet.

Laptops, integrated GPUs and “Unable to detect VRAM”

On a laptop with two GPUs, choose the NVIDIA one by its GPU ID on the Hardware tab or in --usecuda; the built-in graphics chip is usually 1 or 0 depending on the machine, so check the list in the console.

Unable to detect VRAM, please set layers manually. means autofit could not measure the card, as with some AMD and Intel chips and with machines that lack the vendor tools (issue #1748). Set --gpulayers yourself from the calculator. No GPU backend found, or could not automatically determine GPU layers. means a CPU-only build or no backend. See Not using the GPU and the error list.

Several GPUs

Autofit splits the model across all selected GPUs. --tensor_split and the main GPU are covered on the multi-GPU page when it is published.