KoboldCpp sampler settings: good starting points and what each one does

Two kinds of settings

If replies are slow or the program fails, look at the first kind. If replies repeat, ramble or feel flat, look at the second.

KoboldCpp best settings: starting points by use case

What we measured

On a prompt built to make a model loop, we measured how much text repeated for each setting F83. Lower means less repetition, which is not the same as better text:

The lesson: fix a loop with repetition penalty or DRY, not with temperature alone, and a score of zero is not the goal.

What each sampler does

Versions are from the release notes. Field names are the ones /api/v1/generate takes (API fields).

Temperature

Higher makes unlikely words more likely. At 0 the same prompt gives the same text for any seed F83. Our test showed the other end too: 2.0 with nothing cutting the tail gave unreadable text. A seed is not portable between machines: the same model, prompt and seed gave different text on Linux and on Windows and macOS (F33).

Top-K and Top-P

Top-K keeps the K likeliest words; Top-P keeps the smallest group whose chances add to P. Many guides turn both off when using Min-P.

Min-P (1.48)

Drops every word whose chance is below a share of the likeliest word’s chance, so it adapts to how sure the model is. Field min_p.

Repetition penalty

Makes words used in the last rep_pen_range tokens less likely. We measured 1.1 cutting repeats from 45% to 6% F83, and 1.3 drifting off topic. The server default behaves as 1.0.

DRY (1.70)

Penalises a word that would extend a sequence already in the text. Fields dry_multiplier, dry_base, dry_allowed_length. At 0.8 it matched the repetition penalty’s effect in our test F83.

XTC (1.74), dynamic temperature (1.55), smoothing (1.57), Adaptive-P (1.105)

XTC sometimes removes the likeliest word to add variety. Dynamic temperature (dynatemp_range) varies temperature with the model’s certainty. Smoothing (smoothing_factor) reshapes the chances around the top word. Adaptive-P steers towards a target probability. We did not test these four. The old --usemirostat flag was removed after 1.59; the API still lists a mirostat field.

Sampler order

Keep the default order. Users report that custom orders cause odd output; the field is sampler_order. Change it only if a guide for your model says so.

Where to change them

• KoboldAI Lite: Settings, then the Samplers tab. On 1.124.2 its preset [Default] (“same as Simple Balanced”) shows temperature 0.75, repeat penalty 1.05, top-P 0.92, top-K 100, Min-P 0, a repetition range of 360 and the sampler order 6,0,1,3,4,2,5; DRY, XTC, Mirostat and the other advanced samplers have their own boxes, and a preset list offers Simple Logical, Balanced and Creative, Basic Min-P, Creative Min-P, Basic DynaTemp and others F104. Lite’s repeat penalty of 1.05 is not the server’s 1.0.

Close

• SillyTavern: the sampler panel (SillyTavern).
• API: per request (API fields).
• Server defaults: --gendefaults fills fields a request leaves out. We set {"temperature":0} and two seeds gave identical text when the request omitted temperature; with --gendefaultsoverwrite a request asking for 1.5 was still forced to 0 F83.

Replies too short or cut off

The server default is 2048 tokens (/api/v1/config/max_length, F05; --defaultgenamt). A request’s max_length wins: we asked for 20 and got about 10 words F83. In Lite the box is “Amount to generate”. --genlimit caps what any request can ask for. If a reply ends after a few words with EOS token triggered!, suspect the chat template first (template symptoms).

Repetition and loops

Fix in this order: check the template (chat templates); then DRY or a repetition penalty of 1.05 to 1.1; then temperature. A long chat at the context limit can also look like a loop (context).

Model-specific settings

Use the values on your model’s card for Gemma, Qwen, Mistral or GLM; we do not copy numbers we did not test. Per-family template and thinking facts are in the template finder.