KoboldAI API in KoboldCpp: generate, stream and extras

KoboldAI API: the original and KoboldCpp

The original KoboldAI client is legacy, and its project points people to KoboldCpp (the difference). KoboldCpp serves the same /api/v1 routes and reports API version 1.2.5, which is the KoboldAI API version and not its own F24. The soft-prompt routes only exist for compatibility.

All the native routes

Generate text

POST /api/v1/generate (also /api/latest/generate). We sent a prompt and read the answer F85:

{
"prompt": "Hello",
"max_length": 8,
"temperature": 0,
"max_context_length": 2048
}

{
"results": [{
  "text": "\n\nI am trying to create a simple",
  "tool_calls": [],
  "finish_reason": "length",
  "logprobs": null,
  "prompt_tokens": 1,
  "completion_tokens": 8
}]
}

Generation parameters

The server publishes its own spec at /api?json=1. We saved it and built this table from it; the descriptions are ours. Sampler meanings are on the settings page. If a request leaves max_length out the server uses 2048 F05.

Streaming with SSE

POST /api/extra/generate/stream takes the same body and answers with server-sent events. For 12 requested tokens we received 6 events, the first with several tokens, and the last with an empty token and a finish reason F85:

event: messagedata: {"token": " 4, 5,", "finish_reason": null}1…event: messagedata: {"token": "", "finish_reason": "length"}2

Check progress and abort

While a request runs, POST /api/extra/generate/check with its genkey returns the text so far (732 characters after 5 seconds in our test). POST /api/extra/abort with the same genkey stops it: it answered {"success":"true","done":"true"} and the waiting generate call ended with 705 characters of partial text and status 200 F85.

The genkey is a string you choose and send with the generate call. It keeps your check and abort calls separate from other people’s on a shared server.

Server info

Tokens and logprobs

• Count: POST /api/extra/tokencount with {"prompt": "Hello world"} returned value: 3 and the token ids; /api/extra/tokenize is an alias. Note that the first id was the end-of-text token (151643) on this model.
• Decode: POST /api/extra/detokenize with {"ids": [...]} returned result and success.
• Logprobs: send "logprobs": true and the answer carries logprobs.content with top_logprobs per token. /api/extra/last_logprobs (GET, or POST with a genkey) returns the latest. At temperature 0 every value we saw was 0.0.
• Grammar: POST /api/extra/json_to_grammar turns a JSON schema into GBNF text you can pass as grammar.

All captured on the 0.5B test model F85.

Saves, preloaded stories and multiplayer

/api/extra/data/* needs --savedatafile (1.85), /api/extra/preloadstory needs --preloadstory, and /api/extra/multiplayer/* needs --multiplayer (1.79). We ran them only without their flags, where preloadstory answered {}. See multiplayer.

Admin endpoints

/api/admin/* (list options, reload config, save and load KV states) needs --admin with --admindir; protect it with --adminpassword. Without them list_options answered an empty list in our run F85.

Authentication

With --password clients send Authorization: Bearer <key>. It protects generation, not discovery F25. Details: the API page.