Why KoboldCpp is slow and how to make it faster

Why is KoboldCpp slow? Read your speed first

KoboldCpp prints a line after every reply. Two speeds matter: Processed is how fast it read your prompt, Generated is how fast it wrote the reply. The first is the wait before the first word, the second is the pace of the words.

[12:40:51] CtxLimit:1682/4096, Init:0.01s, Processed:1650 in 14.84s (111.22T/s), Generated:32/32 in 0.96s (33.16T/s), Total:15.81s1

Apps can read the same numbers from /api/extra/perf (last_process_speed, last_eval_speed). A comfortable reading speed is about 5 tokens per second. These lanes show what different speeds look like, measured on one of our four-core Windows machines with a small model, with one, two (the automatic choice) and four CPU threads F07:

The speed checklist

In order of how much they usually matter:

1. Fit the whole model on the GPU

This is the biggest lever. Every layer left in system RAM runs at CPU speed, and the slowest part sets the pace. Use the calculator to see how many layers fit your card at your context size, then lower the context (context size) or pick a smaller quantisation until they all do. The GPU layers guide has the steps. We have not measured the speed-versus-layers curve on a physical GPU yet, so we publish no number for the cliff. The CPU end we did measure: Llama 3.1 8B on four CPU cores generated 5.3 tokens per second F37.

2. Make sure the GPU is really used

KoboldCpp can start without error on the wrong build and run on the CPU. In our test with no GPU, the log still said offloaded 25/25 layers to GPU F17, and the CUDA build on a PC without an NVIDIA card runs on the CPU without a message F02. The proof is in the lines that end in buffer size: CUDA0, Vulkan0 or MTL0 is the GPU, CPU is not. See GPU layers, step 3 and Not using the GPU.

3. Avoid the shared-memory spill

On Windows, a model that is slightly too big for VRAM can spill into “Shared GPU memory” and slow to a crawl without an error. Watch Task Manager’s shared memory while it generates. We have not measured it yet; users report it widely. How to check and change the driver policy: the shared memory section.

4. Stop reprocessing long chats

If the Processed count equals the whole chat every turn, KoboldCpp is rereading everything. In our 32-turn chat, a full window cost 23 processed tokens a turn with ContextShift and 1,946 with --noshift, and 25,729 against 749 over the whole chat F66. Typical causes are a changing Memory or World Info near the top, --noshift or --nofastforward, and sliding-window models like Gemma. Details and fixes: ContextShift and fast-forward.

5. Use the right build and backend

The file you download decides which engine runs. On our Windows Server 2022 test machine (no GPU, 0.5B model) generation speed was:

The CUDA file on a PC without NVIDIA also took 11.7 s to start against 6.5 s F02. koboldcpp-oldpc always loads the no-AVX2 library, so on a modern CPU it is about 40% slower F03. --noavx2 and --failsafe are for CPUs that need them: failsafe was 8 times slower F15. Use the download picker if you are unsure. NVIDIA owners can choose between CUDA and Vulkan; we have not measured the difference yet. On a Mac, Metal is the default; on GitHub’s macOS test VM it was slower than the CPU F29, which says nothing about a real Mac.

6. Threads

On the CPU and for the layers left on the CPU, the thread count matters. With no --threads the log said Threadpool set to 2 threads and 2 blasthreads on every four-vCPU machine we tried F07. Four threads gave more generation speed than the automatic two on all of them:

These are cloud machines with four virtual CPUs, not desktops. A second run repeated the direction on Ubuntu (31.5 to 43.1) and both Windows machines (46.1 to 56.7 and 43.4 to 48.5), but cloud CPU scheduling is noisy, so treat the size of the gain as approximate. On a desktop with hyper-threading the best value may be the physical core count; try it and compare the Generated speed, because we have not measured that. The launcher fields are Threads and Batch Threads (--threads, --blasthreads). Threads help much less when every layer is on the GPU. --highpriority is experimental and can make the PC lag.

7. Prompt processing speed: batch size and flash attention

--batchsize (default 512, formerly --blasbatchsize) sets how many prompt tokens are processed together, and --ubatchsize (since 1.122) is the inner micro-batch. We tried seven batch sizes with a 1,650-token prompt on our CPU test machine:

On this machine prompt processing was about 20 to 25% slower at batches of 32 to 128 (135 to 149 tokens per second) than at 256 and above (170 to 181), and there was no clear gain beyond 256. The differences between 256, 512, 1024 and 2048 are within the run-to-run noise on this shared machine (generation speed, which the batch size should not affect, varied between 26 and 36), so do not read a winner into them. The default of 512 is fine on the CPU.

On a GPU the picture differs, and memory use grows with the batch: the compute buffer is larger for a bigger batch, so lowering it saves VRAM (see GPU layers). Users report that a lower batch size can stop freezes on weak GPUs and with Vulkan; we have not measured that.

Flash attention is on by default since 1.107, so adding --flashattention changes nothing F09. Turn it off with --noflashattention only when testing a slowdown or garbled output report. Older advice about “BLAS”, “cuBLAS” or CLBlast is out of date: CLBlast was removed in 1.107, and cuBLAS is the old name of the CUDA backend (--usecublas became --usecuda in 1.96).

Quant choice and speed

On the CPU, a smaller quantisation is not automatically faster. Q8_0 processed a 1,650-token prompt faster than Q4_K_M (for example 133 against 111 tokens per second on Ubuntu) and generated at about the same speed, and Q2_K was no faster to generate F13. F16 generated 25 to 35% slower. The reason to pick a smaller quant is memory, not speed. We have not measured a GPU yet. See quantisation when it is published.

Speculative decoding: draft models and MTP

A small “draft” model can guess several tokens that the big model then checks in one step. Set it with --draftmodel (the draft needs the same vocabulary as the main model) and --draftamount (default 4), or use --usemtp for models that have built-in multi-token-prediction layers (since 1.115). In the launcher these are on the Loaded Files tab.

We tested the mechanism with the models we have: Qwen2.5-0.5B at F16 as the main model and the same model at Q2_K as the draft, on four CPU threads, three runs each:

The draft model’s guesses were right most of the time (about 84% accepted), and generation still got about three times slower, because the draft model also runs on the CPU and the main model is already tiny: the extra work costs more than the saved steps. Where the main model is big and slow and the draft is small and on a fast GPU, the balance can turn the other way.

This is a tiny main model on a CPU, which is the worst case for a draft, so it says nothing about a real 8B model with a 1B draft on a GPU. Measure yours: the Generated speed with and without the draft is the only test that matters.

Loading speed: mmap, mlock and direct I/O

Start-up time is mostly unpacking the single-file program and reading the model: small builds were ready in 3.5 to 7 seconds with a 0.5B model, the CUDA builds in 9 to 12, and oldpc in up to 25 seconds F04. Memory-mapping (--usemmap) is off by default since 1.82; turning it on lowered the memory Windows reported by about 90 MB and changed nothing on Linux or macOS F12. --usemlock locks the model in RAM so the system cannot swap it out. --usedirectio (since 1.120) loads with direct I/O, which can help cold loads on some disks; we have not measured it.

Benchmark your own setup

--benchmark (or Run Benchmark on the Hardware tab) fills the whole context, generates 100 tokens and writes one CSV row F16. The columns are Timestamp, Backend, Layers, Model, MaxCtx, GenAmount, ProcessingTime, ProcessingSpeed, GenerationTime, GenerationSpeed, TotalTime, Output and Flags. On our Linux test VM it reported 97.8 tokens per second processing and 16.3 generating, lower than chat because the context is full. To compare two settings: use the same context, run it three times, and change one thing.

Several users at once

By default KoboldCpp serves one request at a time and queues the rest. --parallelrequests lets it work on several together (continuous batching, experimental). Our two tests disagree on how much that helps on a CPU, so measure yours. On a CI runner, two simultaneous requests took 1.69 s and 3.15 s by default and 3.28 s each with two slots: no gain F26. On our Linux test machine, one request took 3.4 s and four simultaneous requests all finished in 6.2 s, which is 41 tokens per second in total against 19 for one request (with --threads 4: 68 against 33), and each request ran slower than it would alone F75. The gain is for several users or tools at once; a single chat does not speed up. We have not measured a GPU.

Speeds we measured

These are our test machines, which are cloud and lab VMs with four virtual CPUs and no GPU, with a 0.5B model. They set expectations for CPU speed, not for a desktop PC or a graphics card. We will add real hardware as we test it.

It got slower after an update

Regressions happen: users report them with Vulkan and flash attention on Windows (issues #1933 and #1826) among others. Keep the previous file, run --benchmark on both with the same settings, and report with both logs. The download page lists the files for every release.