koboldcpp.dev

Using two or more GPUs with KoboldCpp

Select all GPUs (GPU ID "All") and leave GPU layers on auto: KoboldCpp splits the layers across the cards. To control the split, set --tensor_split in proportion to each card's free VRAM, and choose which card holds the extra buffers with --maingpu. Row split was removed in 1.118, pipeline parallelism is automatic since 1.122, and since 1.114 you can add a GPU from another PC over the network with RPC. We have no GPU in our lab, so only the RPC part is tested by us.

Checked against KoboldCpp 1.124.2 · updated 13 Nov 2026
  • lab-lnx: Linux test VM · 4 vCPU · 15 GB · no GPU
How we test
On this page
  1. Before you start with KoboldCpp multi GPU
  2. Let autofit split the model
  3. Set the split yourself: —tensor_split
  4. Main GPU
  5. Split modes: layer, tensor, and the end of row split
  6. Pipeline parallelism
  7. Mixing NVIDIA and AMD (or different generations)
  8. RPC: use a GPU on another computer
  9. Troubleshooting

Before you start with KoboldCpp multi GPU

Use one backend for all cards: CUDA when every card is NVIDIA, Vulkan for mixed vendors (users report this). The main GPU holds extra buffers, so give it free memory. How to choose a backend: backends.

Let autofit split the model

Select all GPUs in the launcher (GPU ID “All”, or --usecuda all; Vulkan has had “All GPUs” since 1.68) and leave at -1: autofit then places layers across the cards and the CPU (upstream). We could not measure the split. The calculator estimates it for several cards:

VRAM and layers calculator

Set the number of GPUs to see a per-card estimate.

Runs in your browser. Nothing is uploaded.

1. Model

Layers, KV heads and tensor sizes were read from the real file headers on Hugging Face. File size shown is the download size.

2. Your hardware
Reserve and safety margin

Defaults: 512 MB for the desktop on Windows and Linux (0 if the card drives no display), and KoboldCpp's autofit padding of 1024 MB. What this assumes

3. Context and cache

Llama 3.1 8B Instruct Q4_K_M · 16K context · KV f16

Fits completely

All 32 layers fit on the GPU, with the context and buffers.

calculated from the model's tensor sizes; compute buffers are estimates calibrated on two measured models (F36, F42).

GPU memory 6.06 GB of 10.5 GB usable of 12 GB
010.5 GB usable of 12 GB
  • Model weights 3.90 GB
  • KV cache 2.03 GB
  • Compute buffer 132 MB
  • Free 4.44 GB
System RAM 959 MB of 29.0 GB usable of 32 GB
029.0 GB usable of 32 GB
  • Weights on CPU 693 MB
  • Host compute buffer 267 MB
  • Free 28.1 GB

Launch values

  • --gpulayers 33
  • --contextsize 16384

KoboldCpp counts one extra layer for the output layer, so --gpulayers 33 puts all 32 repeating layers on the GPU (F65). Leaving --gpulayers on auto (-1) lets autofit choose; it keeps a 1 GB padding and may offload fewer layers than this calculation. GPU layers guide

koboldcpp --model Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --gpulayers 33 --contextsize 16384
Open in Config Builder

Weights 4.58 GB · 32 layers · 8 KV heads × 128 · KV cache 2.03 GB

The log lines for where each part went are CUDA0 and CUDA1, or Vulkan0 and Vulkan1, in the buffer lines (how to read them) F17.

Set the split yourself: —tensor_split

takes one number per card, in proportion to the memory you want on each. Work them out from each card’s free VRAM, not its size, because the desktop and other programs use some. The launcher’s Hardware tab has a Tensor Split box. (1.106.1) takes a llama.cpp-style list such as Vulkan0,Vulkan1 and overrides the selection (upstream; not tested by us).

Main GPU

(1.94; the launcher’s Main GPU box) picks the card that holds the extra buffers. Give it more free VRAM than the others.

Split modes: layer, tensor, and the end of row split

takes layer or tensor (a separate flag since 1.113; tensor is experimental). row is still accepted but is treated as tensor, with a warning, because row split was removed in 1.118 (the release notes say 1.117.1 was the last version offering it); the old --usecuda rowsplit is translated. removed row split.

Pipeline parallelism

It was opt-in in 1.104 and became automatic in 1.122, when the switch was dropped. The hidden --pipelineparallel and --nopipelineparallel flags show up in the startup Namespace(...) line and need no action.

Mixing NVIDIA and AMD (or different generations)

Use Vulkan for all cards, or RPC. Users report that an old card slows the whole set (issue 642, a regression with several Pascal cards). We could not test any mix. A software Vulkan device is a single device, so it could not stand in for a second card F35.

RPC: use a GPU on another computer

RPC (since 1.114) shares a GPU over the network. On the machine that shares its GPU run --rpcmode host (port , default 5551); on the main machine run --rpcmode connect with host:port,…. (1.124) keeps received tensors on the host so a reload skips sending them; the files are never deleted, and it needs you to trust every client (from the help text).

We tested it on one machine with two processes F100:

RPC host and clientF100 · KoboldCpp 1.124.2 · CPU as the shared device
Starting RPC server v8.0.11  endpoint       : 127.0.0.1:5551llama_prepare_model_devices: using device RPC0 (127.0.0.1:5551) (unknown id) - 16087 MiB free2RPC0[127.0.0.1:5551] model buffer size =   373.71 MiB3Backend 0: RPC4
  1. Starting RPC server v8.0.1: On the host.
  2. llama_prepare_model_devices: using device RPC0 (127.0.0.1:5551) (unkno: On the client: the remote device.
  3. RPC0[127.0.0.1:5551] model buffer size = 373.71 MiB: The model went over the connection.
  4. Backend 0: RPC: This line appears in the log whenever RPC is built in; it is normal.

The same 40-token request took 7.2 s through RPC and about 1.2 s running locally. That is a loopback test with the same CPU on both sides, so it shows the cost of the protocol, not of a real GPU or network: a real GPU and a fast link will differ. With no --rpchost, the host listened on all interfaces (0.0.0.0). A client pointed at a host that was not running stopped with Failed to connect to 127.0.0.1:5551 and a backtrace.

Troubleshooting

  • Out of memory on one card (cudaMalloc failed: out of memory): check --tensor_split and the main GPU’s free memory (out of memory).
  • Only one GPU is used: GPU ID was not “All”.
  • Uneven allocation: autofit leaves padding on each card; set the split by hand.
  • Low utilisation: with a layer split the cards take turns. Users report this as normal; we did not measure it.

Undo: set GPU ID back to one card, remove --tensor_split, and set --rpcmode disabled.

Frequently asked questions

Can KoboldCpp use two GPUs?

Yes. Select all GPUs and leave layers on auto, and it splits the layers across the cards. We have no multi-GPU machine, so this is upstream's description, not our measurement.

How do I set tensor split?

Give one number per card in proportion to its free VRAM, for example --tensor_split 3 1 for a card with three times the free memory. The calculator suggests a split for several cards.

Can I mix NVIDIA and AMD GPUs?

Users report Vulkan for all cards, or RPC. We could not test mixed cards.

What happened to row split?

The release notes say 1.117.1 was the last version that offered it. It is still accepted but treated as tensor split, with a warning.

Can I use a GPU in another PC?

Yes with RPC: run --rpcmode host on that PC and --rpcmode connect with --rpctargets on the main one. We tested it between two KoboldCpp processes: it worked, and it was about 6 times slower than local on loopback.