Connect other apps to KoboldCpp

The one rule: where does the app run?

A browser rule also matters for websites: in Chrome 153 an https page could call KoboldCpp on http://127.0.0.1 or http://localhost, but not on the PC’s own network address, which was blocked as mixed content F78. KoboldCpp itself allows pages on other origins to call it, including with an Authorization header F77.

The generic recipe

KoboldCpp Janitor AI: what to paste

Janitor AI is a website. A request from a website cannot reach localhost on your PC, so you need a public address: start KoboldCpp with --remotetunnel and --password, and give Janitor the address KoboldCpp prints, and the password as the key. The tunnel address changes at every start, so a “failed to fetch” after a restart usually means the address is old. A 401 means the key is wrong, and a 404 means the path is in the wrong form (with or without /v1: guides disagree, so try both).

untested We could not test Janitor AI, a tunnel (the sandbox refused to open one), or what Janitor’s server sees. The steps above are the generic recipe plus the usual causes of those three messages. If you use Colab instead of your PC, see Colab.

Chub

The same pattern: a website needs the tunnel address and password. Whether Chub offers a KoboldAI-type or an OpenAI-type connection is for you to check in its settings; untested we did not open it.

Open WebUI

Add an OpenAI connection (in Open WebUI’s admin settings) with base URL http://localhost:5001/v1; from Docker use your PC’s address instead of localhost. An alternative is its Ollama mode, since KoboldCpp answers Ollama routes (the Ollama client worked). untested Not run.

Agnaistic and RisuAI

Both can use a KoboldAI-type or an OpenAI-type connection with the generic recipe. A browser version is a website (needs the tunnel address unless it runs locally). untested Not run.

VS Code with Continue

Continue takes an OpenAI-compatible provider: provider openai, apiBase http://localhost:5001/v1, any apiKey (or your password), and a code model (choose one). Keep the context size large enough for the files you send (context). Other coding tools that speak the OpenAI protocol follow the same recipe. untested Continue itself was not run.

n8n

Create an OpenAI credential with a custom base URL of http://localhost:5001/v1 (from a Docker n8n, your PC’s address) and any key. untested

LangChain and other Python and JavaScript frameworks

Tested F86:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
  base_url="http://localhost:5001/v1",
  api_key="your --password, or any text if none is set",
  model="koboldcpp",
)
print(llm.invoke("Say hi in three words.").content)

invoke returned a reply with token counts and stream gave 10 chunks; with a password set, a wrong key and a placeholder both gave 401. Embeddings need an embeddings model loaded, or the vector comes back empty (F72, embeddings). The official OpenAI SDKs also work F72.

Home Assistant

Through an OpenAI-compatible conversation integration, with the generic recipe. We did not record the integration’s exact name or fields. untested

Ollama-only apps

Point them at http://localhost:5001. The Ollama Python client worked: chat answered and list returned the model, with the key sent as a Bearer header when a password was set F86. See the Ollama routes and Ollama compared.