ggml-org/llama.cpp v0.4.1

on GitHub

Overview

llama.cpp 0.4.1 adds Maple 20B-A1B, Tencent Hy 4, and Spark2.5 support, improves JSON schema handling, chat parsing, logging, and server child-process management, and updates ggml to v0.24.0.

API changes

• Changed llama_sampler_chain_n() to return int32_t instead of int (#28631).
• Added server_subproc and waiter for router child-process monitoring in server-common.h (#28555).

New models

• Added Maple 20B-A1B ternary MoE architecture (CPU) (#27000).
• Added Tencent Hy 4 (hy_v4) preview architecture support (#28127).
• Added Spark2.5 support (#27868).

Core changes

• Added Kimi-K3 recurrent-state rollback support (#28466).
• Fixed MTP context KV cache allocation for DeepSeek2, GLM-MoE, and related models (#28630).
• Fixed GDN normalization from max to rsqrt for affected Qwen/Kimi/GLM models (#28068).
• Fixed Granite parameter counts and Granite3 MoE unknown parameter counts (#28643, #28632).
• Fixed MIMO2 SWA pattern loading and incorrect get_key_or_arr uses (#28865, #28868).
• Added --fuse-qkv conversion flag to fuse Q/K/V tensors during HF-to-GGUF conversion (#22780).
• Wrote explicit recurrent_layers metadata for Qwen3-Next/Qwen3.5 conversion (#28208).
• Refactored JSON schema handling with a common_schema internal representation (#28736).
• Split specialized chat parsers into common/parsers (#27764).
• Added structured JSONL logging via --log-jsonl and LOG_JSON (#28437, #28586).
• Removed deprecated --mmap/--mlock/--direct-io args in favor of --load-mode (#28334).
• Made mmproj and draft devices default to the global --device selection (#28390).
• Disabled lazy tensor loading by default on iGPUs (#28326).
• Fixed speculative decoding after multimodal input and DFlash mtmd chunk decode (#28715, #28587).
• Fixed Jinja dot-property integer literals and null in membership (#28817, #28620).
• Improved grammar insert/move handling and max repetition threshold (#26885, #28469).
• Improved qwen3-coder complex-type parsing and typed content detection (#28742, #28511).
• Added PCH and unity build support to improve build times (#28091).
• Updated cpp-httplib to 0.56.0 (#28787).

Multi-modality changes

• Propagated video hash to bitmap IDs ({hash}+{frame}) for cache identification (#28601).
• Fixed DFlash speculative decoding failures on mtmd chunks (#28587).

Server changes

• Refactored server child-process handling into a single monitor thread (#28555).
• Fixed LRU hang on multiple requests for the same model (#28539).
• Allow model downloads when --models-max is already reached (#28530).
• Fixed context checkpoint eviction on prompts shorter than checkpoint_min_step (#28302).
• Fixed speculative decoding after image input (#28715).
• Frame router child state commands as whole lines (#28747).
• Enabled --reasoning-preserve by default (#28437).

UI changes

• Improved chat message rendering performance with lazy mounting and compositor-friendly animations (#28460).
• Added UI asset cache for faster builds (#28802).
• Embedded UI assets directly with CMake, removing the build-time helper (#28445).
• Fixed MCP image attachments not displayed in tool blocks (#28089).
• Export conversations from the database instead of the cached store (#27432).

ggml changes

• Bumped ggml from v0.23.0 to v0.24.0 (release).

Version 0.24.0 focuses on expanding backend coverage and robustness, with a new precision-control API, major Vulkan/SYCL/Hexagon/OpenCL work, and numerous correctness and performance fixes across CPU, CUDA, Metal, and other backends.

Assets

Nightly build: b10964

More info

• Releases and versioning of ggml-org projects

Changelog since v0.4.0

b29c606 llama.cpp : bump version to 0.4.1 (#28900)
d9e03f1 sync : ggml
eeea731 ggml : bump version to 0.24.0 (ggml/1627)
bbdd9f2 tests : add fusion baseline README and broaden fusion CI triggers (#28893)
97e4ca7 models : fix incorrect uses of get_key_or_arr (#28868)
1aca1f9 models : fix mimo2 swa pattern load (#28865)
be2c6d7 tests(s390x): add non-vxe build to tests (#28776)
3d10bcd llama: add Maple 20B-A1B ternary MoE architecture (CPU) (#27000)
21f6b0d sycl: rfc: Use radix select for top_k (#28670)
2f53959 ggml-cpu : disable PCH and fix CACHE_LINE_SIZE ambiguity to fix heap corruption (#28882)
89fe242 ci : trigger self-hosted CI on changes to ci/run.sh (#28859)
15d8f2d ci : remove gg_sum summary logic (#28857)
661643e sycl : fix oneDNN scratchpad breaking the pool free order (#28704)
093a2f8 common : move llama_n_rs_seq to before llama_decode (#28749)
ad6c668 ggml-cuda: fallback to F32 on device without BF16 hardware acceleration (#28846)
7a16a6c grammar : coalesce find + insert into a single insert and adjust move/copy mechanics (#26885)
5f436dd tests : exclude HY_V4 from WebGPU test-llama-archs tests (#28855)
e49d2c2 models : guard the expert FFN size fallback in nemotron-h against a zero divisor (#28779)
6978052 ggml-cpu(s390x): guard VXE-only repack helpers (#28775)
243a308 tests : fix typo in test-quant-type-selection for nemotron 3 nano (#28835)
b6b003d sycl : Fix get mem error (#28227)
c95f8e4 ci : run editorconfig and code-style checks on ubuntu-slim (#28854)
bc52a12 pi : prefer PI_MODEL_NAME env var for model disclosure (#28853)
4a89937 tests : reduce FA test sizes (#28842)
37b3a9e ci : remove leftover command (#28839)
002a12a ci : cap test-backend-ops parallel jobs at 2 and add a 3600s timeout (#28833)
f1e44dc vulkan: workaround NV queuesubmit driver bug (#28830)
56b9eb2 opencl: apply the noshuffle row-alignment rule to q4_K, q5_K and q8_0, not just q6_K (#28575)
790cf51 chat : improve parsing of complex types in qwen3-coder (#28742)
8e33095 common: add LOG_JSON macro to log structured data (#28586)
acecd56 common : implement common_schema internal representation for JSON schemas (#28736)
ae9afff jinja : support dot property integer literals (#28817)
737e098 cmake: leave the timestamp out of precompiled headers on clang (#28816)
3057bb6 ui : add cache (#28802)
56381e4 server : allow model downloads at model limit fix issue #26809 (#28530)
c8edceb ggml-cuda: hip add specific config table for AMD GCN (#27841)
e192abb server : add missing headers (#28795)
718f7b4 vendor : update cpp-httplib to 0.56.0 (#28787)
2a3005c syscl : Handle (fail gracefully) unsupported tq1_0 quants (#28681)
f3a33df rpc : fix linking when compiling with BUILD_SHARED_LIBS=OFF (#28492)
c069aa7 server: frame the router child state command as a whole line (#28747)
8a56aed opencl: fix several bugs where the backend aborts (#27630)
07fc977 opencl: add bin kernel kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin (#28677)
3f5e94d webgpu: align tensor bindings to the type block size (#28382)
eafe15a hexagon: support for multi-device model split (aka row-split) (#28589)
d3146f2 ggml-webgpu: Update to a recent version of Dawn (#28683)
82d6bb2 server: refactor subproc handling (#28555)
8ea2902 cmake : skip PCH for llama-server PCH when using MSVC (#28763)
b78a39a ci : run test-backend-ops as a dedicated ci/run.sh test (#28740)
982937a tests: extend test-quantize-fns to test nrc=2 (i8mm) kernels (#16234)
8172e65 tests: tolerate a shared pool abort in test_completion_unified (#28759)
43f3dda ggml: skip 0-sized ids tensor when offloading selected experts (#28739)
5bda51b metal : skip the empty half of the mul_mm_id token tile (#28301)
3bcfeb7 cmake : add PCH and unity build to improve build times (#28091)
1dfe94e common : fix typo in speculative.cpp comment [no ci] (#28750)
a2878d3 metal : single-source fusion table + fusion debug rework (#28164)
aac8102 metal : fix idle threads in the remaining iq mul_mv kernels for ne00 < 1024 (#28692)
5cdd3d1 model : fix MTP context kv cache allocation for deepseek2, glm4moe, … (#28630)
b0dcb81 server: fix speculation after an image (#28715)
16378d9 CUDA/HIP: Flash Attention tuning (gfx1201) (#28102)
451b89b ci : key cache to sanitizer matrix (#28708)
481c65f vulkan: fix data race and OOB access in argsort(large) (#28705)
df03399 opencl: add A8 Q4_0 mm binary kernel support (#28268)
28ff095 vulkan: use CPU writes in ggml_backend_vk_cpy_tensor_async if the context is idle (#28618)
50182a5 vulkan: use add_alloc_dep to enable topk_moe fusion for prefill (#28422)
6788edb vulkan: small M matrix optimizations for qwen (#28457)
52d4268 ci : add self-hosted-gpu-cuda and server-sanitize to hf-jobs (#28693)
18c17b4 ci : Update WoA CUDA 13.4 release to use 13.4.1 GA redistributables (#28687)
fa67698 spec: fix failed to decode mtmd chunk with DFlash (#28587)
41fc758 scripts : use sed instead of grep for version parsing [no ci] (#28700)
d344123 models: clean up some dead switch branches in old models (#28669)
c32d1da tests : increase tolerance for Add fusion tests (#28691)
e5a8d43 tests : drop SYCL special-casing in test-backend-ops.cpp (#28688)
3ff67eb vulkan: fall back to shared-memory reduction for dmmv on PowerVR (#28341)
8c322d5 convert : expand Nemotron H conversion fix (#28689)
311d421 memory : avoid allocating V cache for indexer (it's not used) (#28330)
72797e8 vulkan : add command-buffer debug labels for GPU profilers (#28101)
4ea6d1b ggml-cpu(s390x): add repack support for q4_0 (#28667)
f1b6fbf ggml-cpu(s390x): add Q1_0 vector intrinsic support (#28606)
d7e8643 model: fix all granite family parameter counts (#28643)
434ddbb ci: fix sanitizer tests (#28583)
91f6a6c vulkan: use spec constant for matrix matrix multiplication A-type (#25773)
6d9c82e hexagon: rope updates (#28628)
22397c3 vulkan: Convert FILL to distribute workgroups in 2D to avoid exceeding maxComputeWorkGroupCount (#28592)
4b98ab8 py : lower numpy to 2.2.6 (#28654)
9cf3bf2 py : bump numpy to 2.4.6 (#28649)
4850c77 llama : use int32_t for llama_sampler_chain_n return type (#28631)
d4abd57 CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 (#28552)
5a4d0fe CUDA: replace GGML_FA_ALL_QUANTS with GGML_FA_QUANTS, more control over what is compiled (#28079)
14a9d09 args: officially deprecate --mmap|mlock|dio (#28334)
e2d2c0d model: fix granite3 moe unknown parameter count (#28632)
6de9cdb mtmd: propagate video ID to bitmap (#28601)
b31b71f jinja: treat a null left operand of in as a plain lookup (#28620)
df750f7 vulkan: add dedicated iq4_xs mat-vec shader (#28426)
1945e09 vulkan: add f16 B-type matmul pipelines and warp tile size tuning for Intel coopmat1 (#27471)
30b6a75 tests : use less threads for data initialization (#28325)
304665f Add IQ type handling for MoE (#28476)
f3f1a8f llama: disable lazy tensor loading by default on iGPUs (#28326)
9113cc1 ggml : fix msvc+clang ggml_vld1q_u32 (#28284)
d4389a4 Revert "ggml-cuda : restore prop.integrated on HIP builds (#24233)" (#28604)
5d806aa server : apply checkpoint min-step eviction only when the checkpoint list is full (#28302)
88ada91 metal : fix idle threads in mul_mv_iq3_xxs for ne00 < 1024 (#28086)
415e909 spec: single device drafter should create meta backend wrapper (#28390)
03fa73c ci : disable npm gha cache (#28600)
1744c6b ci : add PYTEST_WORKERS=1 to fix server-self-hosted job (#28603)
ca86fb2 llama : add missing headers (#28566)
64e9bce vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL (#27220)
f014bfe Fix Vulkan-Hpp handle usage on 32-bit targets. (#22892)
895c045 chat : split specialized parsers into common/parsers (#27764)
7d701b5 opencl: properly handle non-contiguous inputs to conv2d (#28503)
5a6caa0 ggml : update ggml_prec specification (#26675)
9dcf84e model : support Kimi-K3 recurrent-state rollback (#28466)
050dde5 hexagon: add RELU and LEAKY_RELU ops (#28585)
67672dc ci : bump ty to 0.0.78 (#28548)
f114f91 tests : initialize the L2_NORM batch array (#28553)
e71b805 Revert "CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 (#24546)" (#28551)
ccc3646 nix : update deprecated expressions (#28145)
c0b1871 webgpu: format the GET_ROWS case block (#28542)
160bd03 server: fix LRU hang on multiple requests same model (#28539)
dbeb375 sycl: add a batched L2_NORM kernel (#28222)
7a333e7 vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST) (#26578)
0c96345 CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 (#24546)
4735997 ggml: add gfx90c HIP support (#26454)
d23c47f convert : refactor Hy4-preview conversion - move HC tensor mapping to the global map (#28451)
73ab759 CUDA: branchless Q4_K/Q5_K unpack to speed up mmvq, L2 prefetch on DGX Spark (#26705)
0cae430 vulkan: support type-aligned GET_ROWS (#28253)
1173700 examples : print ggml_version and ggml_commit in test-cmake [no ci] (#28538)
5202104 caps : recheck typed content if template checks for string (#28511)
9a75705 convert : write explicit recurrent_layers for Qwen3-Next / Qwen3.5 (#28208)
b74f590 ggml-cuda: fix divergent barrier in f16 flash attention (#27870)
992cb50 ggml: allow backend inputs to not create another split (#28387)
9ac8c40 vulkan: rms_norm fusion opportunities (#28024)
2092353 ci : add container image checking and tagging (wip) (#28394)
8fe90e1 vulkan: add TQ1_0 support (mm, mat-vec, mat-vec-id, dequant, get_rows) (#27765)
465e49b convert : add --fuse-qkv flag to fuse Q/K/V into QKV during HF-to-GGUF conversion (#22780)
5fdfa62 models : fix GDN normalization from max to rsqrt (#28068)
3ad1ba7 [Model] Support for Spark2_5ForCausalLM implementation (#27868)
d03efa5 opencl: properly choose weights pack for q4_K, q5_K mul_mat (#28402)
73a43d1 cuda: fixes races in mmid and mmf (#28475)
9e0e220 grammar : fix max repetition threshold (#28469)
0afb805 ui: Improve Chat Messages rendering performance (#28460)
7620399 common: add --log-jsonl (#28437)
c457e3b ui : embed assets directly with CMake (#28445)
971595d metal : add remaining fa-vec tunings for M2 Max (#28458)
74a7c89 Github: limit blank issues to maintainers (#28435)
6a1a922 metal : fix memory leak in early return (#28399)
4d91760 sycl : fix test-backend-ops CI break && restore Kronecker product FWHT support (#28016) (#28254)
cd8cdf3 sycl: attribute device allocations by site (GGML_SYCL_MEMTRACE) (#27631)
427291b metal : add remaining fa-vec tunings for M3 (#28396)
85d5703 ui : fix MCP image attachments not displayed in tool block (#25789) (#28089)
1548a24 opencl: extend the elementwise and data‐movement op coverage (#27633)
4acf4a4 opencl: add Adreno xmem SDPA path (#26331)
8b4b355 ci : move more jobs to ccache-buckets (#28375)
1863ac0 ui: export conversations from database instead of cached store (#27432)
49c0dc8 model : add Tencent Hy 4 (hy_v4) preview architecture support (#28127)