*** Welcome to KoboldCpp - Version 1.124.2 Loading Chat Completions Adapter: /tmp/_MEIhU6cXR/kcpp_adapters/AutoGuess.json Chat Completions Adapter Loaded No GPU or CPU backend was selected. Trying to assign one for you automatically... Unable to detect VRAM. Auto Selected Default Backend (flag=0) Unable to detect VRAM. No GPU backend found, or could not automatically determine GPU layers. You may prefer to set layers manually. System: Linux #17~24.04.1-Ubuntu SMP PREEMPT Thu Sep 24 18:46:57 UTC 2026 x86_64 x86_64 Unable to determine GPU Memory Detected Available RAM: 14986 MB Initializing dynamic library: koboldcpp_default.so Loaded shared common library: koboldcpp_common.so ========== Namespace(admin=False, admindir='', adminpassword= adminunloadtimeout=0, agent=False, agent_api_key= agent_base_url=None, agentload='', allow_config_onready=False, analyze='', autofit=False, autofitpadding=1024, autoswapmode=False, autoswapthreshold=256, baseconfig='', batchsize=512, benchmark='results/ci-1/ubuntu-24.04-nocuda/bench-q8_0/bench.csv', blasthreads=0, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=4096, debugmode=0, defaultgenamt=2048, device='', downloaddir='', draftamount=4, draftgpulayers=999, draftgpusplit=None, draftmodel='', embeddingsgpu=False, embeddingsmaxctx=4096, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, ffncpu=0, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=0, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=False, jinja_kwargs='', jinjatemplate='', jinjathink='default', launch=False, lazy_mode='auto', lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mcpfile='', mmproj='', mmprojcpu=False, model=['models/qwen2.5-0.5b-instruct-q8_0.gguf'], model_param='models/qwen2.5-0.5b-instruct-q8_0.gguf', moe_cache_mib=0, moecpu=0, moeexperts=-1, multiplayer=False, multiuser=None, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken= nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nommq=False, nomodel=False, nopipelineparallel=False, noshift=False, noswa=False, onready='', overridekv='', overridenativecontext=0, overridetensors='', parallelrequests=1, paralleltotalcontext=0, password= pipelineparallel=False, port=5001, port_param=5001, preloadstory='', prompt='', proxy_port=None, quantkv='f16', queuelimit=10, quiet=False, ratelimit=0, reasoningeffort='default', remotetunnel=False, reqtimeout=600, ropeconfig=[0.0, 10000.0], routermode=False, rpccache='', rpcdevice='', rpchost='0.0.0.0', rpcmode='disabled', rpcport=5551, rpctargets='', run_bundled_agent=False, savedatafile='', sdaudiovae='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipdevice=-2, sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdllm='', sdlora=[], sdloramult=[1.0], sdmaingpu=None, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl=None, sdthreads=0, sdtiledvae=512, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, sdvaedevice=-1, sdvramlimit=0, showgui=False, singleinstance=False, skiplauncher=True, smartcache=0, smartcontext=False, splitmode='layer', ssl=None, swapadding=0, tensor_split=None, testmemory=False, threads=2, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', ubatchsize=-1, unpack='', usecpu=False, usecuda=None, usedirectio=False, usemlock=False, usemmap=False, usemtp=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, visionmaxtokens=-1, visionmintokens=-1, websearch=False, whispermodel='') ========== Loading Text Model: /home//work/test-KoboldCpp/test-KoboldCpp/models/qwen2.5-0.5b-instruct-q8_0.gguf The reported GGUF Arch is: qwen2 Arch Category: 24 --- Identified as GGUF model. Attempting to Load... --- Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead! System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | Number of Backends: 2 Backend 0: RPC Backend 1: CPU This model does not use SWA Automatic RoPE Scaling: Using model internal value. Threadpool set to 2 threads and 2 blasthreads... This architecture has explicitly disabled the BOS token - if you need it, you must add it manually. Starting model warm up, please wait a moment... Processing Prompt (0 / 3996 tokens) Processing Prompt (512 / 3996 tokens) Processing Prompt (1024 / 3996 tokens) Processing Prompt (1536 / 3996 tokens) Processing Prompt (2048 / 3996 tokens) Processing Prompt (2560 / 3996 tokens) Processing Prompt (3072 / 3996 tokens) Processing Prompt (3584 / 3996 tokens) Processing Prompt (3996 / 3996 tokens) Generating (1 / 100 tokens) Generating (2 / 100 tokens) Generating (3 / 100 tokens) Generating (4 / 100 tokens) Generating (5 / 100 tokens) Generating (6 / 100 tokens) Generating (7 / 100 tokens) Generating (8 / 100 tokens) Generating (9 / 100 tokens) Generating (10 / 100 tokens) Generating (11 / 100 tokens) Generating (12 / 100 tokens) Generating (13 / 100 tokens) Generating (14 / 100 tokens) Generating (15 / 100 tokens) Generating (16 / 100 tokens) Generating (17 / 100 tokens) Generating (18 / 100 tokens) Generating (19 / 100 tokens) Generating (20 / 100 tokens) Generating (21 / 100 tokens) Generating (22 / 100 tokens) Generating (23 / 100 tokens) Generating (24 / 100 tokens) Generating (25 / 100 tokens) Generating (26 / 100 tokens) Generating (27 / 100 tokens) Generating (28 / 100 tokens) Generating (29 / 100 tokens) Generating (30 / 100 tokens) Generating (31 / 100 tokens) Generating (32 / 100 tokens) Generating (33 / 100 tokens) Generating (34 / 100 tokens) Generating (35 / 100 tokens) Generating (36 / 100 tokens) Generating (37 / 100 tokens) Generating (38 / 100 tokens) Generating (39 / 100 tokens) Generating (40 / 100 tokens) Generating (41 / 100 tokens) Generating (42 / 100 tokens) Generating (43 / 100 tokens) Generating (44 / 100 tokens) Generating (45 / 100 tokens) Generating (46 / 100 tokens) Generating (47 / 100 tokens) Generating (48 / 100 tokens) Generating (49 / 100 tokens) Generating (50 / 100 tokens) Generating (51 / 100 tokens) Generating (52 / 100 tokens) Generating (53 / 100 tokens) Generating (54 / 100 tokens) Generating (55 / 100 tokens) Generating (56 / 100 tokens) Generating (57 / 100 tokens) Generating (58 / 100 tokens) Generating (59 / 100 tokens) Generating (60 / 100 tokens) Generating (61 / 100 tokens) Generating (62 / 100 tokens) Generating (63 / 100 tokens) Generating (64 / 100 tokens) Generating (65 / 100 tokens) Generating (66 / 100 tokens) Generating (67 / 100 tokens) Generating (68 / 100 tokens) Generating (69 / 100 tokens) Generating (70 / 100 tokens) Generating (71 / 100 tokens) Generating (72 / 100 tokens) Generating (73 / 100 tokens) Generating (74 / 100 tokens) Generating (75 / 100 tokens) Generating (76 / 100 tokens) Generating (77 / 100 tokens) Generating (78 / 100 tokens) Generating (79 / 100 tokens) Generating (80 / 100 tokens) Generating (81 / 100 tokens) Generating (82 / 100 tokens) Generating (83 / 100 tokens) Generating (84 / 100 tokens) Generating (85 / 100 tokens) Generating (86 / 100 tokens) Generating (87 / 100 tokens) Generating (88 / 100 tokens) Generating (89 / 100 tokens) Generating (90 / 100 tokens) Generating (91 / 100 tokens) Generating (92 / 100 tokens) Generating (93 / 100 tokens) Generating (94 / 100 tokens) Generating (95 / 100 tokens) Generating (96 / 100 tokens) Generating (97 / 100 tokens) Generating (98 / 100 tokens) Generating (99 / 100 tokens) Generating (100 / 100 tokens) [12:43:45] CtxLimit:4096/4096, Init:0.08s, Processed:3996 in 36.62s (109.12T/s), Generated:100/100 in 4.82s (20.76T/s), Total:41.52sLoad Text Model OK: True Chat completion heuristic: ChatML (Qwen 2.5 based) Embedded KoboldAI Lite loaded. Embedded API docs loaded. Llama.cpp UI loaded. ====== Active Modules: TextGeneration Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl MCPBridge MusicGen RouterMode Enabled APIs: KoboldCppApi OpenAiApi OllamaApi AnthropicApi Running benchmark (Save to File: results/ci-1/ubuntu-24.04-nocuda/bench-q8_0/bench.csv)... Benchmark Completed - v1.124.2 Results: ====== Flags: {'usecuda': None, 'usevulkan': None, 'usecpu': False, 'model': ['models/qwen2.5-0.5b-instruct-q8_0.gguf'], 'model_param': 'models/qwen2.5-0.5b-instruct-q8_0.gguf', 'config': None, 'contextsize': 4096, 'gpulayers': 0, 'host': '', 'launch': False, 'tensor_split': None, 'threads': 2, 'port': 5001, 'port_param': 5001, 'version': False, 'agent': False, 'agentload': '', 'analyze': '', 'autofit': False, 'autofitpadding': 1024, 'batchsize': 512, 'ubatchsize': -1, 'benchmark': 'results/ci-1/ubuntu-24.04-nocuda/bench-q8_0/bench.csv', 'blasthreads': 2, 'chatcompletionsadapter': 'AutoGuess', 'cli': False, 'debugmode': 0, 'defaultgenamt': 2048, 'device': '', 'downloaddir': '', 'draftamount': 4, 'draftgpulayers': 999, 'draftgpusplit': None, 'draftmodel': '', 'enableguidance': False, 'exportconfig': '', 'exporttemplate': '', 'failsafe': False, 'foreground': False, 'gendefaults': '', 'gendefaultsoverwrite': False, 'genlimit': 0, 'highpriority': False, 'ignoremissing': False, 'jinja': False, 'jinja_kwargs': '', 'jinjatemplate': '', 'jinjathink': 'default', 'lora': None, 'loramult': 1.0, 'lowvram': False, 'maingpu': -1, 'maxrequestsize': 32, 'mcpfile': '', 'mmproj': '', 'mmprojcpu': False, 'lazy_mode': 'auto', 'moe_cache_mib': 0, 'moecpu': 0, 'ffncpu': 0, 'moeexperts': -1, 'multiplayer': False, 'nocertify': False, 'nofastforward': False, 'noflashattention': False, 'noavx2': False, 'nobostoken': False, 'nommq': False, 'nomodel': False, 'noshift': False, 'noswa': False, 'onready': '', 'allow_config_onready': False, 'overridekv': '', 'overridenativecontext': 0, 'overridetensors': '', 'parallelrequests': 1, 'paralleltotalcontext': 0, 'password': None, 'preloadstory': '', 'prompt': '', 'quantkv': 'f16', 'queuelimit': 10, 'quiet': False, 'ratelimit': 0, 'reasoningeffort': 'default', 'remotetunnel': False, 'ropeconfig': [0.0, 10000.0], 'savedatafile': '', 'singleinstance': False, 'smartcache': 0, 'smartcontext': False, 'splitmode': 'layer', 'ssl': None, 'swapadding': 0, 'unpack': '', 'usemtp': False, 'usemlock': False, 'usemmap': False, 'usedirectio': False, 'visionmaxres': 1024, 'visionmaxtokens': -1, 'visionmintokens': -1, 'websearch': False, 'showgui': False, 'skiplauncher': True, 'hordemodelname': '', 'hordeworkername': '', 'hordekey': '', 'hordemaxctx': 0, 'hordegenlen': 0, 'sdaudiovae': '', 'sdclamped': 0, 'sdclampedsoft': 0, 'sdclip1': '', 'sdclip2': '', 'sdclipdevice': -2, 'sdconvdirect': 'off', 'sdflashattention': False, 'sdlora': [], 'sdloramult': [1.0], 'sdmaingpu': None, 'sdmodel': '', 'sdoffloadcpu': False, 'sdphotomaker': '', 'sdquant': 0, 'sdllm': '', 'sdthreads': 0, 'sdtiledvae': 512, 'sdupscaler': '', 'sdvae': '', 'sdvaeauto': False, 'sdvaedevice': -1, 'sdvramlimit': 0, 'whispermodel': '', 'ttsmodel': '', 'ttswavtokenizer': '', 'ttsgpu': False, 'ttsmaxlen': 4096, 'ttsthreads': 0, 'ttsdir': '', 'musicllm': '', 'musicembeddings': '', 'musicdiffusion': '', 'musicvae': '', 'musiclowvram': False, 'embeddingsmodel': '', 'embeddingsmaxctx': 4096, 'embeddingsgpu': False, 'rpcmode': 'disabled', 'rpcport': 5551, 'rpchost': '0.0.0.0', 'rpcdevice': '', 'rpccache': '', 'rpctargets': '', 'admin': False, 'adminpassword': None, 'admindir': '', 'adminunloadtimeout': 0, 'routermode': False, 'reqtimeout': 600, 'autoswapmode': False, 'autoswapthreshold': 256, 'baseconfig': '', 'hordeconfig': None, 'sdconfig': None, 'noblas': False, 'nommap': False, 'nopipelineparallel': False, 'pipelineparallel': False, 'sdnotile': False, 'sdt5xxl': None, 'sdvaecpu': False, 'sdclipgpu': False, 'forceversion': False, 'sdgendefaults': False, 'flashattention': False, 'useswa': False, 'multiuser': None, 'testmemory': False, 'run_bundled_agent': False, 'agent_base_url': None, 'agent_api_key': None, 'proxy_port': None} Timestamp: 2026-11-13 12:43:45.529946+00:00 Backend: koboldcpp_default.so Layers: 0 Model: qwen2.5-0.5b-instruct-q8_0 MaxCtx: 4096 GenAmount: 100 ----- ProcessingTime: 36.619s ProcessingSpeed: 109.12T/s GenerationTime: 4.816s GenerationSpeed: 20.76T/s TotalTime: 41.435s Output: 1 1 1 1 ----- Server was not started, main function complete. Idling. llama_model_loader: loaded meta data with 26 key-value pairs and 291 tensors from /home//work/test-KoboldCpp/test-KoboldCpp/models/qwen2.5-0.5b-instruct-q8_0.gguf (version GGUF V3 (latest)) print_info: file format = GGUF V3 (latest) print_info: file type = (guessed) unknown, may not work print_info: file size = 638.74 MiB (8.50 BPW) init_tokenizer: initializing tokenizer for type 2 load: 0 unused tokens load: control-looking token: '' was not control-type; this is probably a bug in the model. its type will be overridden load: printing all EOG tokens: load: - 128247 ('') load: - 151643 ('<|endoftext|>') load: - 151645 ('<|im_end|>') load: - 151662 ('<|fim_pad|>') load: - 151663 ('<|repo_name|>') load: - 151664 ('<|file_sep|>') load: special tokens cache size = 23 load: token to piece cache size = 0.9310 MB print_info: arch = qwen2 print_info: vocab_only = 0 print_info: no_alloc = 0 print_info: n_ctx_train = 32768 print_info: n_embd_inp = 896 print_info: n_embd = 896 print_info: n_embd_out = 896 print_info: n_layer = 24 print_info: n_layer_all = 24 print_info: n_head = 14 print_info: n_head_kv = 2 print_info: n_rot = 64 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: non_causal_type = 0 print_info: n_embd_head_k = 64 print_info: n_embd_head_v = 64 print_info: n_gqa = 7 print_info: n_embd_k_gqa = 128 print_info: n_embd_v_gqa = 128 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-06 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: f_attn_value_scale = 0.0000 print_info: n_ff = 4864 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: n_expert_groups = 0 print_info: n_group_used = 0 print_info: causal attn = 1 print_info: pooling type = -1 print_info: rope type = 2 print_info: rope scaling = linear print_info: freq_base_train = 1000000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 32768 print_info: rope_yarn_log_mul = 0.0000 print_info: rope_finetuned = unknown print_info: model type = 1B print_info: model params = 630.17 M print_info: general.name = qwen2.5-0.5b-instruct print_info: vocab type = BPE print_info: n_vocab = 151936 print_info: n_merges = 151387 print_info: BOS token = '<|endoftext|>' print_info: EOS token = '<|im_end|>' print_info: EOT token = '<|im_end|>' print_info: PAD token = '<|endoftext|>' print_info: LF token = 'Ċ' print_info: FIM PRE token = '<|fim_prefix|>' print_info: FIM SUF token = '<|fim_suffix|>' print_info: FIM MID token = '<|fim_middle|>' print_info: FIM PAD token = '<|fim_pad|>' print_info: FIM REP token = '<|repo_name|>' print_info: FIM SEP token = '<|file_sep|>' print_info: EOG token = '' print_info: EOG token = '<|endoftext|>' print_info: EOG token = '<|im_end|>' print_info: EOG token = '<|fim_pad|>' print_info: EOG token = '<|repo_name|>' print_info: EOG token = '<|file_sep|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (load_mode = none) done_getting_tensors: tensor 'token_embd.weight' (q8_0) (and 290 others) moved from CPU_REPACK, using CPU instead load_tensors: offloading 0 repeating layers to GPU load_tensors: offloaded 0/25 layers to GPU load_tensors: CPU model buffer size = 638.74 MiB ........................................................... llama_context: constructing llama_context llama_context: n_seq_max = 1 llama_context: n_ctx = 4352 llama_context: n_ctx_seq = 4352 llama_context: n_batch = 512 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = enabled llama_context: kv_unified = true llama_context: freq_base = 1000000.0 llama_context: freq_scale = 1 llama_context: n_rs_seq = 0 llama_context: n_outputs_max = 512 llama_context: n_outputs_max_per_seq = 1 llama_context: n_ctx_seq (4352) < n_ctx_train (32768) -- the full capacity of the model will not be utilized set_abort_callback: call llama_context: CPU output buffer size = 0.58 MiB llama_kv_cache: CPU KV buffer size = 51.00 MiB llama_kv_cache: size = 51.00 MiB ( 4352 cells, 24 layers, 1/1 seqs), K (f16): 25.50 MiB, V (f16): 25.50 MiB llama_kv_cache: n_rot_k = 0, n_embd_head_k_all = 64 llama_kv_cache: n_rot_v = 0, n_embd_head_k_all = 64 llama_context: enumerating backends llama_context: backend_ptrs.size() = 1 sched_reserve: reserving ... sched_reserve: max_nodes = 2328 resolve_fused_ops: resolving fused DeepSeek V4 HC support: resolve_fused_ops: fused DeepSeek V4 HC pre enabled resolve_fused_ops: fused DeepSeek V4 HC comb enabled resolve_fused_ops: fused DeepSeek V4 HC post enabled sched_reserve: CPU compute buffer size = 309.77 MiB sched_reserve: graph: nodes = 825, splits = 1, input objects = 4, input tensors = 9 sched_reserve: reserve took 2.45 ms, sched copies = 1 attach_threadpool: call