Use KoboldCpp as an OpenAI-compatible server

Endpoints at a glance

These are the OpenAI-style and Anthropic routes. Our status says “works” only where we called the route and got an answer: F23 for the first captures, F72 for the SDK tests. The full list is on the API page.

Speech routes (/v1/audio/transcriptions, /v1/audio/speech) are in the voice guide.

KoboldCpp OpenAI API: the base URL

http://localhost:5001/v1. On another device or through a tunnel only the host changes (remote access). KoboldCpp also answers OpenAI paths without /v1, so an app that adds /v1 itself works with the bare address, and in our test an SDK given the bare address worked too F72. If the app says “endpoint does not exist”, see the URL mistakes.

What “API key” means here

KoboldCpp has no accounts, keys or billing: the only key is the password you choose. Any page that tells you to get an API key from a dashboard is wrong. Do not commit a real --password value into a code repository.

The model field

/v1/models returns the loaded model with the id koboldcpp/<file name>. The value you send as model is ignored: in our test anything-at-all and gpt-4o both got an answer from the loaded model, and the reply’s model field said koboldcpp/qwen2.5-0.5b-instruct-q4_k_m F72. In router mode (--routermode) the field picks the model; see sharing.

Use it from the official SDKs

1. Start KoboldCpp with a model

   Load any model: see getting started.

2. Set the base URL

   In the SDK set base_url (Python) or baseURL (JavaScript) to http://localhost:5001/v1.

3. Set the key

   A placeholder if there is no password, otherwise the password.

4. Send a chat

   A normal chat completion. The reply’s model field reads koboldcpp/...: that is how you know it worked.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:5001/v1", api_key="placeholder")
r = client.chat.completions.create(
    model="anything",
    messages=[{"role": "user", "content": "Say OK and nothing else."}],
    max_tokens=12,
)
print(r.model, r.choices[0].message.content)

What each feature did, with the model we had (a 0.5B model, so we record the shape of the answer, not its quality):

Chat versus text completions

Chat completions take a list of messages, and KoboldCpp turns them into one prompt with a chat template: by default it guesses one (--chatcompletionsadapter, default AutoGuess), and --jinja uses the model’s own template. If replies continue your prompt or contain role tags, the template is wrong. Text completions send your prompt as it is, and you format it yourself. In our test completions stopped with length after 12 tokens, and chat answered OK F72.

Streaming

Set stream: true and KoboldCpp sends server-sent events in OpenAI’s chat.completion.chunk format, ending with data: [DONE] (18 events in our capture) F23:

data: {"id": "chatcmpl-A3023341", "object": "chat.completion.chunk", "created": 1794574088, "model": "koboldcpp/qwen2.5-0.5b-instruct-q4_k_m", "choices": [{"ind…1data: [DONE]2

Models that think write their reasoning in a separate field since 1.111, which broke some clients (per the release notes). We have not tested a thinking model.

Tools and function calling

KoboldCpp has supported tools since 1.70, automatic tool choice since 1.87, and --jinja for Jinja-based tool templates (from the release notes). In our test a tools request returned a parsed tool call on the default settings, and with --jinja the call arrived as text F72. We used a 0.5B model, so this says nothing about how well larger models choose tools. Users report wrong-tool and empty-output problems (#1696, #1448): if tool calls misbehave, try both settings and a model made for tools.

JSON mode and structured output

A strict json_schema response format returned valid JSON with the right keys and an integer where the schema asked for one, and json_object parsed F72. KoboldCpp added OpenAI structured outputs in 1.90 (release notes), and /api/extra/json_to_grammar converts a schema to a grammar.

Images, audio and vision through /v1

An image_url data URI in a chat message worked with a small vision model, and /v1/audio/transcriptions transcribed a sample correctly F27. /v1/audio/speech and /v1/images/generations need a speech or image model loaded; we have not tested them. See vision, voice and images.

Embeddings

With --embeddingsmodel and a GGUF embedding model loaded, /v1/embeddings returned 384-number vectors from the 22 MB MiniLM model F27 F72.

Responses API and Anthropic Messages

/v1/responses is an emulation added in 1.111 (release notes). The official OpenAI SDK’s responses.create returned completed with the text F72.

Anthropic Messages

/v1/messages (the Anthropic API) was basic in 1.111 and complete in 1.115 (release notes). The official Anthropic Python SDK worked for a normal and a streamed message, with the base URL given as the bare address or with /v1 F72.

Coming from Ollama

KoboldCpp emulates Ollama’s /api/chat, /api/generate, /api/tags, /api/embed and /api/show. In our test /api/tags, /api/chat, /api/generate and /api/show returned 200 and /api/tags listed the loaded model F72. Point the Ollama app at http://localhost:5001. /api/version answers 0.7.0, an emulated Ollama version rather than KoboldCpp’s, and /slots answers 501. We did not test an Ollama-only app, and we did not test /api/embed. Model files differ: Ollama stores its own blobs, while KoboldCpp loads a GGUF from Hugging Face.

Differences from OpenAI to expect

• One model at a time, unless you use router or admin mode.
• Requests queue one at a time by default F26.
• The default reply length is 2048 tokens when max_tokens is missing F05.
• A password protects generation but not /v1/models: with --password, a wrong key still got the model list F72.
• The request body can be up to 32 MB.