Will it fit? VRAM and GPU layers calculator

koboldcpp --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --gpulayers 33 --contextsize 16384

How to read the result

The big line is one of four states.

• Fits completely. Every layer, the context and the buffers fit on the GPU. This is the fastest way to run a model.
• Fits with N of M layers on the GPU. The first N layers run on the GPU and the other layers run on the CPU from system RAM. It works, and it is slower, because the CPU layers set the pace. More layers on the GPU means faster output. See GPU layers.
• Runs on the CPU only. Nothing fits on the GPU with this context, or there is no GPU.
• Doesn’t fit in GPU and RAM. The model needs more memory than the GPU and system RAM together offer. The list under the result shows what to change and how much each change saves.

Below the line, the bar shows where the memory goes. This is a measured example (F36): Llama 3.1 8B Q4_K_M with 8K of context on a 12 GB card, using the allocation sizes KoboldCpp logged.

• Model weights are the layers on the card. They are the biggest block and depend on the quantisation.
• KV cache is the memory for the conversation so far. It grows with the context you ask for. See the formula below.
• Compute buffer is working space for each batch of tokens. It is small next to the other two.
• Other models are a vision projector, Whisper or an image model, if you turned them on.
• A hatched block means memory that does not fit anywhere.

The launch values are what to give KoboldCpp: --gpulayers for the layers, --contextsize for the context and --quantkv when you chose a smaller cache. Leaving --gpulayers at its default of -1 is often fine: autofit picks a number for you. The calculator is for the cases where you want to know in advance, want the largest context a card can hold, or want to check a model before you download it.

How much VRAM does context use? The KV cache formula

The KV cache stores the keys and values the model has already computed, so it does not redo them for every new token. Its size follows one formula, in words:

2 (keys and values) × layers × context tokens × KV heads × head size × bytes per value

KoboldCpp allocates the context you ask for plus 256 tokens F06, and the default context is 16384 F05. For Qwen2.5-0.5B (24 layers, 2 KV heads of 64 values) at 16384 that is 16640 cells, 2 × 24 × 16640 × 2 × 64 × 2 bytes = 195 MiB at f16, and KoboldCpp’s log says KV buffer size = 195.00 MiB F10. The table is computed by the same code the calculator runs, and the build stops if it ever stops matching the logged sizes.

A bytes-per-value of 2 is f16. The quantised caches use less: q8_0 is 1.0625 bytes, q5_1 is 0.75 and q4_0 is 0.5625. In the same test, peak memory at 32K of context fell from 976 MB to 794 MB with q8_0 and to 698 MB with q4_0 F10. A quantised cache needs flash attention, which is on by default F09; with it off, only the K cache is quantised.

Two kinds of models change the arithmetic. A sliding-window model such as Gemma 2 keeps the full context in only some layers; the rest keep a window of tokens, so its cache is much smaller at long context. A mixture-of-experts model has its own attention sizes, and its weights split into attention and expert parts, which is what --moecpu moves to system RAM. The calculator handles both when it can read the layout from the file header, and tells you when it cannot.

Is 8 GB, 12 GB, 16 GB or 24 GB of VRAM enough?

It depends on the model and the context. These tables are calculated by the calculator with the defaults (Windows desktop reserve of 512 MB, safety margin of 1024 MB, KV cache f16, 32 GB of RAM). They show the largest quantisation that fits completely. “Tight” means less than 5% of the usable memory is left. These are calculated values: we have not yet measured these cards, so treat them as a guide and use the calculator for your exact case.

At 8K of context

At 16K of context

Two rules of thumb fall out of the tables. A 7B to 9B model at a 4-bit quantisation fits an 8 GB card with room for a normal context. The step to 12 GB buys a 12B to 14B model or a longer context; 16 GB and 24 GB make 30B-class models practical. For what to run, see Which model should I use? and quantisation.

What GPU layers are

A language model is a stack of identical layers, 32 in a typical 8B model. Every layer you put on the GPU runs fast; every layer you leave behind runs on the CPU, from system RAM, and sets the speed of the whole answer. “N of M layers on the GPU” is the split.

--gpulayers sets the number. -1, the default, asks KoboldCpp to work it out with autofit, and 0 keeps everything on the CPU. KoboldCpp also accepts -ngl and --n-gpu-layers as other names for the same flag. The GPU layers guide covers how to choose the number by hand and how to check it worked.

What this LLM VRAM calculator assumes

Each assumption is labelled with how we know it: measured in our own runs, upstream (from KoboldCpp’s own help or source), or estimate.

Accuracy and our tests

We tested the calculator in the following ways.

• KV cache sizes. The formula reproduces the sizes KoboldCpp logged for Qwen2.5-0.5B exactly: 51.00, 195.00 and 1539.00 MiB at 4096, 16384 and 131072 tokens, and 108.84 MiB at 32768 with q4_0 F10.
• Predicted against measured memory. The table compares the calculator’s prediction with the allocations logged in five of our runs. All are within 10%. These runs used a software Vulkan device in our lab, so they check the arithmetic and not real-card behaviour.

• Header reading. The reader parses 13 test headers: four quantisations of one real model, a real vision-model text part, a real vision projector, a real embedding model, and synthetic headers built from public model configs for a dense 8B, a sliding-window 9B, a mixture-of-experts 30B, a tied-embedding model and a two-part split. Sizes from the real files add up to the real file sizes within 1.5%, the difference being the vocabulary stored in the full header.
• Network. In the browser test, local-file reading made no request at all, and a pasted link made requests only to huggingface.co.
• Local file speed. A 20 GB file reads in well under a second, because only the header is read.

Known limits. We do not have a real GPU, so the “predicted layers within ±2 of the maximum measured” check on real cards (T-MEM-01) has not been run, and the tool says “calculated” rather than “calibrated on a card”. Brand-new architectures with unknown keys may be sized wrongly, and the tool says so when it falls back to a guess. Image model sizes are presets.

Your files stay on your computer

Local file. When you pick a .gguf file, your browser reads only its first few megabytes with the File API, enough for the header and tensor list. The file is not uploaded and the page makes no network request while reading it. Hugging Face link. Your browser asks huggingface.co for the first few megabytes with an HTTP range request; the model weights are not downloaded and no other site is contacted. The page’s security policy allows these requests on this page only. See the privacy page.