Get KoboldCpp running with your first model
Download the KoboldCpp file for your system from GitHub, start it, and the launcher opens. Click HF Search, pick a small GGUF model (a 3B to 8B model in Q4_K_M is a safe first choice), then press Launch. Your browser opens KoboldAI Lite at http://localhost:5001: close the welcome box, type a message, and the model answers on your own computer.
Checked against KoboldCpp 1.124.2 · updated 13 Nov 2026
- lab-lnx: Linux test VM · 4 vCPU · 15 GB · no GPU
- gha-w25: Windows Server 2025 test machine · 4 vCPU · 16 GB · no GPU
- gha-w22: Windows Server 2022 test machine · 4 vCPU · 16 GB · no GPU
- gha-u24: Ubuntu 24.04 test machine · 4 vCPU · 16 GB · no GPU
At a glance
- Time needed
- Not timed on a real PC yet
- Difficulty
- Beginner
- You need
- 64-bit Windows, Linux or an Apple Silicon Mac
- Cost
- Free
On this page
- How to use KoboldCpp: what you are about to set up
- No suitable computer?
- Step 1: Download the right file
- Step 2: Start KoboldCpp
- Step 3: Pick your first model
- Three safe first models
- Will it fit? Check in 5 seconds
- Get it with HF Search (inside the launcher)
- Or: already have a .gguf?
- Or: start from a ready-made template
- Step 4: Press Launch
- Step 5: Your first reply in KoboldAI Lite
- Check it worked
- Start it the same way next time
- What to do next
- Undo / remove
How to use KoboldCpp: what you are about to set up
Three pieces work together: KoboldCpp is the program, a GGUF file is the model, and KoboldAI Lite is the chat page that KoboldCpp serves in your browser. KoboldCpp loads the model onto your CPU and GPU and answers requests. Nothing leaves your computer. See what KoboldCpp is and KoboldAI vs KoboldCpp.
- Youtype a message
- KoboldAI Litethe chat page in your browser
- KoboldCppthe program you start
- The model filea .gguf on your CPU or GPU
No suitable computer?
- Only a phone: Android runs it in Termux.
- An old laptop with no GPU: it still works on the CPU with a small model: our lab machines run everything on CPU F04. Or use Colab.
- Want to try without installing: use Kobold Lite online with the AI Horde: KoboldAI.
Step 1: Download the right file
Answer the questions to get your file.
.exe and put it in its own folder. Full guide: Windows.chmod +x on it. Full guide: Linux.Step 2: Start KoboldCpp
./koboldcpp-linux-x64 (or the file you chose) in a terminal../koboldcpp-mac-arm64 in Terminal.The first start unpacks the program, so it takes a few seconds F04. Then the launcher opens on the Quick Launch tab. Its defaults are already sensible: Launch Browser, ContextShift and flash attention on, context 16384 F30.
Do not change anything else yet.
Step 3: Pick your first model
A model is one .gguf file. A bigger model is smarter but needs more memory. The “Q4_K_M” part is the quant: it trades quality for size (quantization).
Three safe first models
Prove it works
Qwen2.5 0.5B Instruct
Apache-2.0TestAny PC- Recommended file
- qwen2.5-0.5b-instruct-q4_k_m.gguf
- Quant
- Q4_K_M
- File size
- 0.5 GB
- Context we recommend
- 4,096
- Chat template
- Chat (auto-detected)
- Our test status
- works
Runs on any PC, quality is basic. This is the file our own tests use.
Laptop, CPU or a 4 to 6 GB GPU
Llama 3.2 3B Instruct
Llama 3.2 Community LicenceChatSmall PCs- Recommended file
- Llama-3.2-3B-Instruct-Q4_K_M.gguf
- Quant
- Q4_K_M
- File size
- 2.0 GB
- Context we recommend
- 8,192
- Chat template
- Llama 3 (auto-detected)
- Our test status
- untested
Not loaded in our lab yet. The community quant is by bartowski; Meta publishes no GGUF.
8 GB or more of GPU memory
Llama 3.1 8B Instruct
Llama 3.1 Community LicenceChatStories- Recommended file
- Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
- Quant
- Q4_K_M
- File size
- 4.9 GB
- Context we recommend
- 8,192
- Chat template
- Llama 3 (auto-detected)
- Our test status
- works
We measured its memory (F36). On 4 CPU cores it generated 5.3 tok/s (F37). The community quant is by bartowski.
Sizes are the file sizes on Hugging Face, checked on 14 Nov 2026. We have not tested the 3B model yet. For more, see models.
Will it fit? Check in 5 seconds
Memory decides. As a measured example, Llama 3.1 8B Q4_K_M needs about 5.1 GiB of GPU memory at 8K context and 8.1 GiB at 32K F36; the 0.5B model needs about 0.7 GB of RAM F14. Context costs memory: RAM rose from 616 MB at 2K to 976 MB at 32K on our small model F11. Leave context at the default for now. The memory calculator gives the exact answer; see also context size.
Get it with HF Search (inside the launcher)
- Click HF Search next to GGUF Text Model F30.
- Type the model name, pick the Q4_K_M file and confirm.
- Wait for the download. A model fetched from a link goes to the working directory unless Folder where models given as URLs are downloaded to and looked up in (default: working directory).Default Launcher Loaded Files tab: Download DirSince v1.107.2All details is set.
We have not recorded this flow on a real PC yet. For a model split in several files, pick part 1. See Hugging Face.
Or: already have a .gguf?
Use Browse.
Or: start from a ready-made template
The launcher’s Get Help menu has Newbie Templates, presets labelled for about 6, 12 and 24 GB of video memory (upstream). We have not yet checked what they download.
Step 4: Press Launch
Press Launch. The console prints, in order: the version, the backend, how many layers went to the GPU, the buffers, and the address. This is a real start with the 0.5B model:
Welcome to KoboldCpp - Version 1.124.21Auto Selected Default Backend (flag=0)2load_tensors: offloaded 0/25 layers to GPU3llama_context: n_ctx = 166404llama_kv_cache: CPU KV buffer size = 195.00 MiB5- Welcome to KoboldCpp - Version 1.124.2: Your version.
- Auto Selected Default Backend (flag=0): Which engine was chosen.
- load_tensors: offloaded 0/25 layers to GPU: Layers on the GPU. Do not trust this line alone (F17).
- llama_context: n_ctx = 16640: The context you asked for plus 256 (F06).
- llama_kv_cache: CPU KV buffer size = 195.00 MiB: Where the context memory lives. With a GPU this starts with CUDA0, Vulkan0 or MTL0.
How many layers go to the GPU is set by GPU layers; leave it on the default and KoboldCpp chooses.
Step 5: Your first reply in KoboldAI Lite
The browser opens, because Launch Browser is on by default F30. If it does not, open http://localhost:5001. A “Welcome To KoboldAI Lite” box asks you to choose Classic, Aesthetic or Corpo UI F31.
Lite opens in Instruct mode, and names the model and the endpoint. Type a first message:
Other modes (Chat, Story, Adventure) are covered in the Lite guide.
Check it worked
Check it worked
Speed depends on the machine. On four CPU cores a 0.5B model generated about 18 tok/s in our lab and an 8B model 5.3 tok/s F37. If the log shows the three no-GPU lines F18 or “CPU”, see not using the GPU. Something off? Paste the console text here:
Start it the same way next time
Use Save Config in the launcher to write a .kcpps file. Next time, Load Config, or start the program with the file. See .kcpps files and the Windows shortcut section.
What to do next
- A better model for your GPUWhat fits, by memory.
- Use SillyTavernConnect it to KoboldCpp.
- Memory and World InfoUse Lite well.
- Every launcher settingWhat each tab does.
- Make it fasterWhich number is low.
- Use it from your phoneSafely, with a password.
Undo / remove
Delete the program file, the model and your saved configs. Nothing else is installed: see Windows uninstall, Linux and Mac.
Frequently asked questions
Is KoboldCpp free and does it work offline?
Yes to both. KoboldCpp is free, open-source software with no account. Once you have the program and a model file, the model runs on your own computer, and the chat page is served locally.
What is the best first model for KoboldCpp?
A 3B to 8B model in Q4_K_M is a safe first choice, depending on your memory. To prove the setup works on any PC, start with a 0.5B model. We list three starter models with their sizes on this page.
Do I need a GPU to use KoboldCpp?
No. Our lab machines run everything on the CPU. A small model is usable on a laptop; on four CPU cores an 8B model generated 5.3 tokens per second in our test, about reading speed.
How do I change the model in KoboldCpp?
Close KoboldCpp and start it again with another model file, using the launcher's model field. To switch without restarting, start it with admin mode on.
Where do the models I download go?
A model downloaded from a link goes to the working directory unless you set a download folder with --downloaddir. A model you chose with Browse stays where you saved it.
Is my chat private when I use KoboldCpp?
Chats stay on your computer by default. Data leaves your PC only when you turn on something that needs the internet, such as a remote tunnel, the AI Horde, web search or a model download.