Context size in KoboldCpp: memory, ContextShift and the KV cache

What context size means

A model reads and writes tokens, pieces of words. The context size is the number of tokens it can hold in view at once: everything you send plus the reply it writes. When a chat is longer than that, the oldest text no longer fits and is cut.

Tokens are not words, and the ratio depends on the text and the model. With the Qwen2.5 tokenizer, a simple story sentence came to 1.07 tokens per word, and technical text with numbers and names came to 2.20 tokens per word F68. So there is no single rule of thumb.

Everything in the prompt uses the window. In KoboldAI Lite the order is: Memory, then World Info entries that matched, then the story, with the Author’s Note added near the end. What is left over is the room for the reply (Lite).

1. Memory and World Infoalways kept; they come first
2. Story or chat so farthe oldest part is cut first when the window is full
3. Author's Noteadded near the end
4. Room for the replythe reply length counts against the same window

How much memory context needs

The model keeps a KV cache: for each token it has read, two values (a key and a value) for each attention layer. In words:

2 (keys and values) × layers × context tokens × KV heads × head size × bytes per value

KoboldCpp allocates the context you ask for plus 256 tokens F06:

A worked example from our logs: Qwen2.5-0.5B has 24 layers and 2 KV heads of 64 values. At 16384 requested that is 16640 cells, and the log prints the line below, which equals the formula exactly F10:

llama_context: n_ctx                 = 166401llama_kv_cache: size =  195.00 MiB ( 16640 cells,  24 layers,  1/1 seqs), K (f16):   97.50 MiB, V (f16):   97.50 MiB2

For real models the cache is much bigger, because they have more layers and more KV heads. These sizes are calculated by the same code as the calculator; Llama 3.1 8B matches the sizes KoboldCpp logged (1,056 MiB at 8K, 4,128 MiB at 32K) F36, and the others are calculated from their file headers.

On the CPU, the cache lives in system RAM, so context raises RAM use. On our small test model, peak memory rose from 616 MB at 2K to 2,123 MB at 128K F11:

These are lab numbers for a tiny model and do not carry over to others; the calculator does the arithmetic for yours.

Try your own model

Choose a model and move the context slider. The result shows the KV cache and whether everything still fits.

koboldcpp --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --gpulayers 33 --contextsize 16384

Does a bigger context make it slower?

Not for short prompts. On our CPU test, generation speed did not fall as the context grew F11:

The differences are run-to-run noise on a shared machine, not a speed-up. What costs time is a long prompt, which must be read before the first word of the reply, and reprocessing a whole chat when the cache cannot be reused (below). We have not measured the same on a GPU yet.

Choosing your KoboldCpp context size

In this order:

1. Not above the model’s training length. The log prints it as n_ctx_train. If you ask for less, KoboldCpp says so with a harmless line: n_ctx_seq (2304) < n_ctx_train (32768) -- the full capacity of the model will not be utilized. That only means you chose a smaller window.
2. Small enough that all layers still fit on the GPU: GPU layers. Lower the context before lowering layers.
3. The same number in your client, see below.

The range KoboldCpp accepts is 256 to 524288. Outside it, it exits at once with error: argument --contextsize/--ctx-size/-c: must be within [256, 524288] and loads nothing F22. The default has moved several times, which is why old guides disagree: 2048 (1.20), 4096 (1.70), 8192 (1.96), 16384 (1.115), 12288 (1.115.2) and 16384 again (1.122). --contextsize shows the current value.

Our guidance, not a measured rule:

Set the same context in Lite and SillyTavern

KoboldAI Lite and SillyTavern each have their own maximum context and maximum output settings. KoboldCpp does not set them for you, and they can differ. We saw that Lite sends its own context size, not the server’s F61. A client set higher than the server gets its prompt trimmed by the server; one set lower leaves memory unused. Set both to the same number. In Lite the field is in the settings; for SillyTavern see the SillyTavern guide when it is published.

ContextShift and fast-forward: why long chats do not reprocess

Two features keep a long chat fast:

• Fast-forward reuses the start of the prompt that is unchanged since the last turn, so KoboldCpp only reads the new text.
• ContextShift (since 1.48, on by default) handles a full window: instead of reprocessing, it drops the oldest tokens from the cache and slides the rest along.

Both need the beginning of the prompt to stay the same. --noshift turns off ContextShift. --nofastforward turns off both. The launcher has them on the Context tab.

We ran a scripted 32-turn chat at 2048 context and counted the tokens KoboldCpp processed each turn, from its own log F66. The window fills at turn 20.

With the defaults, a turn reads only the 23 new tokens. The log shows what happens when the window is full:

[Context Shifting: Erased 102 tokens at position 22]1

Notice that the Memory edit cost one full reprocess, then shifting resumed. Anything that changes the start of the prompt does that. These make a chat reprocess, and what avoids them:

• World Info or lorebook entries inserted near the top. Entries that appear and disappear change the early prompt. Insert them lower, or keep the set of active entries stable.
• Random or time-based text (macros, a clock in the prompt) near the top. Move it to the end or remove it.
• Editing an old message. Everything after the edit is new to the cache. Expected: the cost is one reprocess.
• Changing Memory. One full reprocess, as measured above.
• A sliding-window model such as Gemma: it cannot shift at all, see below.
• ContextShift or fast-forward turned off: --noshift or --nofastforward. Remove the flag.
• Switching between chats with different starts: see SmartCache below.

SmartCache: switch chats without reprocessing

With two chats open, each turn replaces the cache with the other chat, so every switch reprocesses everything. --smartcache (since 1.104) keeps snapshots of the cache in RAM so returning to a chat is quick. It needs fast-forward. We alternated two chats for 8 turns each F66:

In the log, each return to a chat prints SmartCache Match of 1.00 restored from slot 0. Two slots cost about 33 MB more RAM with chats of up to about 800 tokens; a slot holds a cache snapshot, so the cost grows with the chat. The older --smartcontext is superseded by ContextShift and not recommended.

Quantized KV cache (—quantkv)

--quantkv stores the cache at lower precision: f16 (the default), bf16, q8_0, q5_1 or q4_0. Named values work since 1.112, and the old numbers 0 to 3 are still accepted. It needs flash attention, which is on by default F09; without it only the K cache is quantised.

On our small model, at 32K of context, peak RAM went from 976 MB (f16) to 794 MB (q8_0) and 698 MB (q4_0), and the cache itself was 108.8 MiB with q4_0 F10. In the calculator above, q8_0 is about half and q4_0 about a quarter of the f16 size.

Does it break ContextShift? Different sources disagree, so we tested: on 1.124.2, on the CPU, with --quantkv q8_0, shifting worked exactly as with the default (23 tokens a turn once full, Context Shifting lines in the log) F66. We did not test a GPU, and we have not measured whether answers get worse.

To undo it, use --quantkv f16 or remove the flag.

Sliding-window models (Gemma and others)

Models such as Gemma use sliding-window attention (SWA): most layers only look at the last few hundred tokens. KoboldCpp turns SWA on automatically (since 1.114), which makes the cache much smaller, and the price is that ContextShift is disabled F67:

SWA Mode is ENABLED!1Context shifting is DISABLED!2llama_kv_cache_iswa: creating non-SWA KV cache, size = 2304 cells3llama_kv_cache_iswa: creating     SWA KV cache, size = 1152 cells4

In the same 32-turn chat, the SWA model processed 1,967 tokens every turn once the window was full, and with --noswa it processed 24, at a cost of memory: peak RAM 499 MB with SWA and 629 MB without, at only 2048 context. The window cache size follows the batch size: 896, 1152 and 1664 cells at batches of 256, 512 and 1024, and it does not grow with the context. --swapadding adds padding that controls how far you can rewind before a forced reprocess. Pick --noswa only if you need shifting more than you need the memory. This is one small model; the calculator uses the same rule for others, and says so.

Going past the training length: RoPE

Models can be stretched past the context they were trained on by scaling the position encoding (RoPE). KoboldCpp does this automatically from the model’s trained context. You can set it by hand with --ropeconfig (a frequency scale, then a base), or tell it the model’s native length with --overridenativecontext. We have not tested how well models hold up past their training length (the needle-in-a-haystack runs are not done), so we publish no advice beyond this one: leave it automatic unless a model’s card says otherwise, and treat anything above the training length as lower quality.