KoboldCpp is running on the CPU instead of your GPU

KoboldCpp not using GPU? Check in 30 seconds

1. Start KoboldCpp from a terminal

   Run it from a Command Prompt or terminal so the text stays on screen: how.

2. Find the engine library line

   Look for Initializing dynamic library:. koboldcpp_default, koboldcpp_noavx2 and koboldcpp_failsafe are CPU paths. In our logs the Vulkan build printed koboldcpp_vulkan.so.

3. Find the KV buffer line

   Look for the line that ends in KV buffer size, and the other buffer lines. CUDA0, Vulkan0 or MTL0 means the GPU holds that part; CPU means it does not.

4. Check dedicated GPU memory while it generates

   In Task Manager open Performance, then your GPU, and watch Dedicated GPU memory. On NVIDIA you can also run nvidia-smi. The “3D” graph can look idle while the GPU computes, so do not rely on it (users report this). We have not measured it on a real GPU yet.

5. Check it worked

   The KV buffer line now starts with CUDA0, Vulkan0 or MTL0, and replies are faster.

Compare the two logs. On the left, a machine with no GPU and --gpulayers 99: no error, and the log still says 25 of 25 layers. On the right, a Vulkan device that was found.

Auto Selected Default Backend (flag=0)1Initializing dynamic library: koboldcpp_default.so2load_tensors: offloaded 25/25 layers to GPU3llama_kv_cache:        CPU KV buffer size =    51.00 MiB4

Initializing dynamic library: koboldcpp_vulkan.so1ggml_vulkan: Found 1 Vulkan devices:2load_tensors:      Vulkan0 model buffer size =   106.39 MiB3llama_kv_cache:    Vulkan0 KV buffer size =    23.38 MiB4

The right-hand log comes from a software Vulkan device, not a physical card, because we do not yet have a real GPU in the lab; the line format is the same.

Why “offloaded 25/25 layers to GPU” can be wrong

We started a build with no GPU in three ways: a CUDA flag on the nocuda build, --usevulkan with no Vulkan device, and --gpulayers 99. In each, KoboldCpp started without an error, printed offloaded 25/25 layers to GPU and kept CPU KV buffer size. Speed stayed at CPU speed F17:

The same cases on our other test machines gave the same result: speed matched the CPU. (On Windows, the CUDA build with a CUDA flag fails instead when there is no NVIDIA driver, see the driver section F56.)

One exception is deliberate: --lowvram keeps the KV cache in system RAM, so a CPU KV buffer size line there is expected, not a fault.

The backend is set to CPU

In the launcher, the Backend dropdown decides the engine. On a PC with no GPU it reads “Use CPU” F30:

Close

• NVIDIA: choose CUDA. Command line: --usecuda.
• AMD or Intel: choose Vulkan. Command line: --usevulkan.
• Mac: Metal is automatic, see Mac.
• Force CPU: --usecpu (the launcher says “Use CPU”).

If you choose nothing, a machine KoboldCpp cannot read prints three lines F18:

No GPU or CPU backend was selected. Trying to assign one for you automatically...
Unable to detect VRAM.
Auto Selected Default Backend (flag=0)

That means no GPU backend was selected. See Auto Selected Default Backend and No GPU or CPU backend was selected.

You downloaded the wrong file

• koboldcpp-nocuda has no CUDA. An NVIDIA owner should choose Vulkan with it, or download koboldcpp.exe.
• koboldcpp.exe (CUDA) on a PC without NVIDIA runs on the CPU without any error, but starts 2 to 4 times slower: ready in 11.7 s against 6.5 s on Windows Server 2022, generation 42.3 against 47.4 tok/s F02.
• koboldcpp-oldpc always loads the no-AVX2 library, about 40% slower on a modern CPU F03.

The download picker chooses the right file. If your file has a different name, see renamed files. Download only from GitHub: fake sites.

“Unable to detect VRAM”

KoboldCpp could not read your video memory, so it cannot size the GPU layers by itself. Users report it on 25 issues (#1748 for an AMD iGPU, #1413). The variants in 1.124.2 F18:

• Unable to detect VRAM.
• Unable to determine GPU Memory
• No GPU backend found, or could not automatically determine GPU layers. You may prefer to set layers manually.

On a PC with no GPU these lines are correct. On a PC with a GPU: choose the backend yourself (above), set --gpulayers by hand using the memory calculator, and check the driver (below). See Unable to detect VRAM.

Autofit put only a few layers on the GPU

With the default --gpulayers -1, KoboldCpp fits layers automatically. Autofit leaves a safety margin: --autofitpadding is 1024 MB by default in our logs. Users report cases such as 7 of 41 layers. We have not yet measured how much autofit leaves unused on a real 12 GB card, so we cannot say how cautious it is.

To get more layers on the GPU, in this order: lower the context size, unload image or voice models, lower the padding, or set the number of layers by hand just under what fits, using the memory calculator. Read GPU layers.

Laptops with two GPUs

Hybrid laptops have a weak integrated GPU and a stronger dedicated one. KoboldCpp may list the integrated one first. Choose the dedicated one by its ID: --usecuda 1 or --usevulkan 1 (the number is the device index; it is printed in the log when the backend lists devices), or the launcher’s GPU ID field. Plug in the charger, because many laptops limit the GPU on battery. We have not tested whether the Windows Graphics settings change anything, so we do not recommend them.

Driver and old-GPU problems

If CUDA is not offered, or the CUDA file will not run, update the NVIDIA driver from nvidia.com. In our CI runs a CUDA flag on a machine without the driver failed with a missing koboldcpp_cublas.dll or libcuda.so.1 F56. GTX 900 and older cards may need koboldcpp-oldpc (a CUDA 11.4 build) or Vulkan; the oldest cards work with Vulkan only (users report; we have not tested). Architecture errors are explained under CUDA error. See NVIDIA and it won’t start.

AMD and Intel GPUs

Vulkan through koboldcpp-nocuda is the official path. ROCm on Windows is a third-party fork. Intel Arc works through Vulkan only. See AMD and backends.

Mac

Metal is on by default, and the log shows MTL0 buffers. On GitHub’s macOS virtual machine Metal was slower than the CPU (4.3 to 5.7 against 7.5 to 10.1 tok/s), so a VM says nothing about a real Mac F29. See the Mac guide.

The GPU is used but sits at low utilisation

If only some layers are on the GPU, the GPU waits for the CPU, so utilisation is low. That is how partial offload works (“use GPU and CPU together”). Fit all the layers if you can: GPU layers. Otherwise accept the mix, and see speed. We have not measured the layers-versus-speed curve on a real GPU yet.

Not sure what is wrong? Start the decision tree. Out of memory instead? Out of memory.