*** Welcome to KoboldCpp - Version 1.124.2 Loading Chat Completions Adapter: /tmp/_MEIhayljB/kcpp_adapters/AutoGuess.json Chat Completions Adapter Loaded No GPU or CPU backend was selected. Trying to assign one for you automatically... Unable to detect VRAM. Auto Selected Default Backend (flag=0) Unable to detect VRAM. No GPU backend found, or could not automatically determine GPU layers. You may prefer to set layers manually. System: Linux #17~24.04.1-Ubuntu SMP PREEMPT Thu Sep 24 18:46:57 UTC 2026 x86_64 x86_64 Unable to determine GPU Memory Detected Available RAM: 14936 MB Initializing dynamic library: koboldcpp_default.so Loaded shared common library: koboldcpp_common.so ========== Namespace(admin=False, admindir='', adminpassword= adminunloadtimeout=0, agent=False, agent_api_key= agent_base_url=None, agentload='', allow_config_onready=False, analyze='', autofit=False, autofitpadding=1024, autoswapmode=False, autoswapthreshold=256, baseconfig='', batchsize=512, benchmark=None, blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=131072, debugmode=0, defaultgenamt=2048, device='', downloaddir='', draftamount=4, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=4096, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, ffncpu=0, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=0, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=False, jinja_kwargs='', jinjatemplate='', jinjathink='default', launch=False, lazy_mode='auto', lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mcpfile='', mmproj='', mmprojcpu=False, model=['models/qwen2.5-0.5b-instruct-q4_k_m.gguf'], model_param='models/qwen2.5-0.5b-instruct-q4_k_m.gguf', moe_cache_mib=0, moecpu=0, moeexperts=-1, multiplayer=False, multiuser=None, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken= nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nommq=False, nomodel=False, nopipelineparallel=False, noshift=False, noswa=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', parallelrequests=1, paralleltotalcontext=0, password= pipelineparallel=False, port=52031, port_param=52031, preloadstory='', prompt='', proxy_port=None, quantkv='f16', queuelimit=10, quiet=False, ratelimit=0, reasoningeffort='default', remotetunnel=False, reqtimeout=600, ropeconfig=[0.0, 10000.0], routermode=False, rpccache='', rpcdevice='', rpchost='0.0.0.0', rpcmode='disabled', rpcport=5551, rpctargets='', run_bundled_agent=False, savedatafile='', sdaudiovae='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipdevice=-2, sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdllm='', sdlora=[], sdloramult=[1.0], sdmaingpu=None, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl=None, sdthreads=0, sdtiledvae=512, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, sdvaedevice=-1, sdvramlimit=0, showgui=False, singleinstance=False, skiplauncher=True, smartcache=0, smartcontext=False, splitmode='layer', ssl=None, swapadding=0, tensor_split=None, testmemory=False, threads=2, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', ubatchsize=-1, unpack='', usecpu=False, usecuda=None, usedirectio=False, usemlock=False, usemmap=False, usemtp=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, visionmaxtokens=-1, visionmintokens=-1, websearch=False, whispermodel='') ========== Loading Text Model: /home//work/test-KoboldCpp/test-KoboldCpp/models/qwen2.5-0.5b-instruct-q4_k_m.gguf llama_model_loader: loaded meta data with 26 key-value pairs and 291 tensors from /home//work/test-KoboldCpp/test-KoboldCpp/models/qwen2.5-0.5b-instruct-q4_k_m.gguf (version GGUF V3 (latest)) print_info: file format = GGUF V3 (latest) print_info: file type = (guessed) unknown, may not work print_info: file size = 462.96 MiB (6.16 BPW) init_tokenizer: initializing tokenizer for type 2 load: 0 unused tokens load: control-looking token: '' was not control-type; this is probably a bug in the model. its type will be overridden load: printing all EOG tokens: load: - 128247 ('') load: - 151643 ('<|endoftext|>') load: - 151645 ('<|im_end|>') load: - 151662 ('<|fim_pad|>') load: - 151663 ('<|repo_name|>') load: - 151664 ('<|file_sep|>') load: special tokens cache size = 23 load: token to piece cache size = 0.9310 MB print_info: arch = qwen2 print_info: vocab_only = 0 print_info: no_alloc = 0 print_info: n_ctx_train = 32768 print_info: n_embd_inp = 896 print_info: n_embd = 896 print_info: n_embd_out = 896 print_info: n_layer = 24 print_info: n_layer_all = 24 print_info: n_head = 14 print_info: n_head_kv = 2 print_info: n_rot = 64 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: non_causal_type = 0 print_info: n_embd_head_k = 64 print_info: n_embd_head_v = 64 print_info: n_gqa = 7 print_info: n_embd_k_gqa = 128 print_info: n_embd_v_gqa = 128 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: f_attn_value_scale = 0.0000 print_info: n_ff = 4864 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: n_expert_groups = 0 print_info: n_group_used = 0 print_info: causal attn = 1 print_info: pooling type = -1 print_info: rope type = 2 print_info: rope scaling = linear print_info: freq_base_train = 1000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 32768 print_info: rope_yarn_log_mul = 0.0000 print_info: rope_finetuned = unknown print_info: model type = 1B print_info: model params = 630.17 M print_info: general.name = qwen2.5-0.5b-instruct print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = '<|endoftext|>' print_info: EOS token = '<|im_end|>' print_info: EOT token = '<|im_end|>' print_info: PAD token = '<|endoftext|>' print_info: LF token = 'Ċ' print_info: FIM PRE token = '<|fim_prefix|>' print_info: FIM SUF token = '<|fim_suffix|>' print_info: FIM MID token = '<|fim_middle|>' print_info: FIM PAD token = '<|fim_pad|>' print_info: FIM REP token = '<|repo_name|>' print_info: FIM SEP token = '<|file_sep|>' print_info: EOG token = '' print_info: EOG token = '<|endoftext|>' print_info: EOG token = '<|im_end|>' print_info: EOG token = '<|fim_pad|>' print_info: EOG token = '<|repo_name|>' print_info: EOG token = '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (load_mode = none) done_getting_tensors: tensor 'token_embd.weight' (q5_0) (and 278 others) moved from CPU_REPACK, using CPU instead load_tensors: offloading 0 repeating layers to GPU load_tensors: offloaded 0/25 layers to GPU load_tensors: CPU model buffer size = 434.91 MiB load_tensors: CPU_REPACK model buffer size = 28.05 MiB ..................................................... llama_context: constructing llama_context llama_context: n_seq_max = 1 llama_context: n_ctx = 131328 llama_context: n_ctx_seq = 131328 llama_context: n_batch = 512 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = enabled llama_context: kv_unified = true llama_context: freq_base = 1000000.0 llama_context: freq_scale = 1 llama_context: n_rs_seq = 0 llama_context: n_outputs_max = 512 llama_context: n_outputs_max_per_seq = 1 llama_context: n_ctx_seq (131328) > n_ctx_train (32768) -- possible training context overflow set_abort_callback: call llama_context: CPU output buffer size = 0.58 MiB llama_kv_cache: CPU KV buffer size = 1539.00 MiB llama_kv_cache: size = 1539.00 MiB (131328 cells, 24 layers, 1/1 seqs), K (f16): 769.50 MiB, V (f16): 769.50 MiB llama_kv_cache: n_rot_k = 0, n_embd_head_k_all = 64 llama_kv_cache: n_rot_v = 0, n_embd_head_k_all = 64 llama_context: enumerating backends llama_context: backend_ptrs.size() = 1 sched_reserve: reserving ... sched_reserve: max_nodes = 2328 resolve_fused_ops: resolving fused DeepSeek V4 HC support: resolve_fused_ops: fused DeepSeek V4 HC pre enabled resolve_fused_ops: fused DeepSeek V4 HC comb enabled resolve_fused_ops: fused DeepSeek V4 HC post enabled sched_reserve: CPU compute buffer size = 433.77 MiB sched_reserve: graph: nodes = 825, splits = 1, input objects = 4, input tensors = 9 sched_reserve: reserve took 2.48 ms, sched copies = 1 attach_threadpool: call Load Text Model OK: True Chat completion heuristic: ChatML (Qwen 2.5 based) Embedded KoboldAI Lite loaded. Embedded API docs loaded. Llama.cpp UI loaded. ====== Active Modules: TextGeneration Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl MCPBridge MusicGen RouterMode Enabled APIs: KoboldCppApi OpenAiApi OllamaApi AnthropicApi Note: For third party Ollama API Emulation, you should set the port to 11434. Starting Kobold API on port 52031 at http://localhost:52031/api/ Starting OpenAI Compatible API on port 52031 at http://localhost:52031/v1/ Starting llama.cpp secondary WebUI at http://localhost:52031/lcpp/ ====== Please connect to custom endpoint at http://localhost:52031 The reported GGUF Arch is: qwen2 Arch Category: 24 --- Identified as GGUF model. Attempting to Load... --- Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead! System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | Number of Backends: 2 Backend 0: RPC Backend 1: CPU This model does not use SWA Automatic RoPE Scaling: Using model internal value. Threadpool set to 2 threads and 2 blasthreads... This architecture has explicitly disabled the BOS token - if you need it, you must add it manually. Starting model warm up, please wait a moment... Processing Prompt (0 / 38 tokens) Processing Prompt (38 / 38 tokens) Generating (1 / 128 tokens) Generating (2 / 128 tokens) Generating (3 / 128 tokens) Generating (4 / 128 tokens) Generating (5 / 128 tokens) Generating (6 / 128 tokens) Generating (7 / 128 tokens) Generating (8 / 128 tokens) Generating (9 / 128 tokens) Generating (10 / 128 tokens) Generating (11 / 128 tokens) Generating (12 / 128 tokens) Generating (13 / 128 tokens) Generating (14 / 128 tokens) Generating (15 / 128 tokens) Generating (16 / 128 tokens) Generating (17 / 128 tokens) Generating (18 / 128 tokens) Generating (19 / 128 tokens) Generating (20 / 128 tokens) Generating (21 / 128 tokens) Generating (22 / 128 tokens) Generating (23 / 128 tokens) Generating (24 / 128 tokens) Generating (25 / 128 tokens) Generating (26 / 128 tokens) Generating (27 / 128 tokens) Generating (28 / 128 tokens) Generating (29 / 128 tokens) Generating (30 / 128 tokens) Generating (31 / 128 tokens) Generating (32 / 128 tokens) Generating (33 / 128 tokens) Generating (34 / 128 tokens) Generating (35 / 128 tokens) Generating (36 / 128 tokens) Generating (37 / 128 tokens) Generating (38 / 128 tokens) Generating (39 / 128 tokens) Generating (40 / 128 tokens) Generating (41 / 128 tokens) Generating (42 / 128 tokens) Generating (43 / 128 tokens) Generating (44 / 128 tokens) Generating (45 / 128 tokens) Generating (46 / 128 tokens) Generating (47 / 128 tokens) Generating (48 / 128 tokens) Generating (49 / 128 tokens) Generating (50 / 128 tokens) Generating (51 / 128 tokens) Generating (52 / 128 tokens) Generating (53 / 128 tokens) Generating (54 / 128 tokens) Generating (55 / 128 tokens) Generating (56 / 128 tokens) Generating (57 / 128 tokens) Generating (58 / 128 tokens) Generating (59 / 128 tokens) Generating (60 / 128 tokens) Generating (61 / 128 tokens) Generating (62 / 128 tokens) Generating (63 / 128 tokens) Generating (64 / 128 tokens) Generating (65 / 128 tokens) Generating (66 / 128 tokens) Generating (67 / 128 tokens) Generating (68 / 128 tokens) Generating (69 / 128 tokens) Generating (70 / 128 tokens) Generating (71 / 128 tokens) Generating (72 / 128 tokens) Generating (73 / 128 tokens) Generating (74 / 128 tokens) Generating (75 / 128 tokens) Generating (76 / 128 tokens) Generating (77 / 128 tokens) Generating (78 / 128 tokens) Generating (79 / 128 tokens) Generating (80 / 128 tokens) Generating (81 / 128 tokens) Generating (82 / 128 tokens) Generating (83 / 128 tokens) Generating (84 / 128 tokens) Generating (85 / 128 tokens) Generating (86 / 128 tokens) Generating (87 / 128 tokens) Generating (88 / 128 tokens) Generating (89 / 128 tokens) Generating (90 / 128 tokens) Generating (91 / 128 tokens) Generating (92 / 128 tokens) Generating (93 / 128 tokens) Generating (94 / 128 tokens) Generating (95 / 128 tokens) Generating (96 / 128 tokens) Generating (97 / 128 tokens) Generating (98 / 128 tokens) Generating (99 / 128 tokens) Generating (100 / 128 tokens) Generating (101 / 128 tokens) Generating (102 / 128 tokens) Generating (103 / 128 tokens) Generating (104 / 128 tokens) Generating (105 / 128 tokens) Generating (106 / 128 tokens) Generating (107 / 128 tokens) Generating (108 / 128 tokens) Generating (109 / 128 tokens) Generating (110 / 128 tokens) Generating (111 / 128 tokens) Generating (112 / 128 tokens) Generating (113 / 128 tokens) Generating (114 / 128 tokens) Generating (115 / 128 tokens) Generating (116 / 128 tokens) Generating (117 / 128 tokens) Generating (118 / 128 tokens) Generating (119 / 128 tokens) Generating (120 / 128 tokens) Generating (121 / 128 tokens) Generating (122 / 128 tokens) Generating (123 / 128 tokens) Generating (124 / 128 tokens) Generating (125 / 128 tokens) Generating (126 / 128 tokens) Generating (127 / 128 tokens) Generating (128 / 128 tokens) [12:46:32] CtxLimit:166/131072, Init:0.04s, Processed:38 in 0.31s (121.02T/s), Generated:128/128 in 2.75s (46.56T/s), Total:3.10s