ggml-org/llama.cpp v0.5.0

on GitHub

Overview

This release focuses on backend performance and correctness, broader model coverage, and more robust server/router operation. It adds HRM-Text (DFM Mimir 1B) support, MiMo-V2.6 and HunyuanOCR conversion support, ggml 0.25.0 backend improvements, multi-address HTTP binding, image outputs from function calls, and several chat parser/UI fixes.

Highlights

• Accelerate CUDA conv2d with implicit GEMM (#29135)
• Add Metal MoE and SSM_CONV fusion optimizations (#28948)
• Allow the server to bind to multiple addresses (#28690)

API changes

• Add llama_adapter_lora_init_from_file_ptr() for loading LoRA from an open FILE (#28993)
• Document llama_model_load_from_file_ptr() as reading from the current position and requiring aligned mmap (#28993)
• Add LLAMA_VOCAB_TYPE_TEST dummy tokenizer (#29084)
• Add input_image support to server function-call outputs (#22575)
• Allow --host to accept comma-separated TCP addresses and UNIX sockets (#28690)

New models

• Add HRM-Text / DFM Mimir 1B support (#27625)
• Add MiMo-V2.6 conversion support (#29257)
• Add DFlash support for HunyuanOCR (#28890)
• Extend Nemotron MTP and Nemotron-H model handling (#29018, #28989)
• Add Qwen4Exp hyper-connection ops and sparse flash attention (#28901, #28770)
• Add --fuse-qkv support for Muse Glimmer (#29203)

Core changes

• Add graph input/input-tensor diagnostics during scheduler reserve (#26625)
• Enable CUDA graphs for MTP drafting (#28549)
• Fix tensor-parallel split state/granularity for fused QKV models (#28965)
• Fix Mamba time-step projection input contiguity (#28832)
• Write the SWA pattern in the model saver and round-trip 15 more architectures (#29042)
• Add environment variables for temperature, top-p, min-p and penalties (#27380)
• Reduce the sampler backend probe size (#29285)
• Add Ling 3.0, DeepSeek V3.2/V4, qwen3-coder, Muse Glimmer and Gemma 4 parser fixes (#28682, #29008, #28869, #29242, #29115)
• Improve JSON Schema and PEG handling (#28518, #29127, #29161)
• Add ufakzeka pre-tokenizer and llama-bench --version (#29033, #28971)

Multi-modality changes

• Add sanity checks for mtmd layer indices, SAM layer counts, resize targets and graph allocation (#29276, #28149)
• Fix SigLIP bucket buffer overrun for tall/wide images (#29276)

Server changes

• Fix router eviction races and child process lifecycle handling (#29217)
• Do not pass log file or API key file to router-spawned children (#29212, #28938)
• Improve startup and model-source logging (#29125)
• Update vendored cpp-httplib to 0.57.1 (#29239)

UI changes

• Accept WEBM video files (#28622)
• Add close button to UI toasts (#28246)
• Fix mobile breakpoint and content overflow issues, including horizontal table scrolling (#29108)
• Restore the reasoning menu in single-model desktop mode (#27985)
• Stop re-probing a disabled /tools endpoint on every message (#28646)

ggml changes

• Updated ggml to v0.25.0 (release)
• The release expands hyper-connection, flash-attention, and fused MoE/SSM support across backends, with robustness, quantization, data-layout, and RPC/meta improvements.
• API changes include gated ggml_dsv4_hc_pre_gated(), optional ggml_dsv4_hc_post() comb, and RPC protocol major v7.

Assets

Nightly build: b11146

More info

• Releases and versioning of ggml-org projects

Changelog since v0.4.1

7fe450e llama.cpp : bump version to 0.5.0 (#29333)
177cd8c sync : ggml
e4e2f62 ggml : bump version to 0.25.1 (ggml/1637)
66fba63 CUDA: add a reserve to avoid spurious warning on older GCC builds (#29317)
bddf826 common : keep HF cache dir as path, expose UTF-8 only for logs (#29320)
9575389 metal: add the missing f32 x bf16 mul_mv variants (#28741)
dc9879c CUDA: enable sparse-fa for dsv4 prefill (again) (#29298)
42916d8 server: fix token counting API crash on sleep (#29309)
4e416ee jinja : parse unary +/- before variables (#29244)
ee3ecce metal : key the fa-vec tuned table by family instead of SKU (#29075)
057494f server: accept OpenAI video_url content type and data: video URIs (#27921)
bcbc936 server: Dedup the draft HF model via dedup-cache-models (#27934)
26758d3 ci : fix build-cmake runner target (#29299)
18f9f7b model-conversion : add causal-compare-logits recipe (#29305)
633733d model : support Gemma4 DSpark draft backbone (#29226)
86b2daa ci : run python (jinja) test (#29302)
183d2a0 make-release : update summary prompt
45062d4 sync : ggml
503549c ggml : bump version to 0.25.0 (ggml/1635)
e97545d sycl : fix compile warnings
b1ff4ca vulkan: add IQ4_XS MMQ/MMV matmul kernels (#28415)
9425611 ggml-meta: resolve multi buffer views (#29266)
1a67982 cuda: top-k MoE should always fire (#28432)
384a534 sycl : support new UT case for mul_mat_hadamard fp16 (#29218)
5e48b31 sycl: extend MMVQ GLU fusion, add rms_norm+scale and ssm_conv+silu fusions (#28931)
4d7d770 sycl : support op get_rows_back, only support fp32/fp16 (#25266)
08b1d2a vulkan: hide internal symbols to prevent duplicate-dlopen state destruction (#29139)
441df11 sampler: reduce the size of the probe (#29285)
e6ab7c1 hex-dma: introduce direct-mapped DMA cache that is better suited for HVX FA mask handling (#29282)
f46bc30 HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (#27962)
709fe75 jinja : fix dangling reference warning in for_statement (#29279)
d5f6649 opencl: add bin kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin (#29056)
9919911 server: fix router eviction races with the existing queue (#29217)
bbf99b1 server: do not pass log file to children (#29212)
4098fdc server: support input_image in function_call_output (#20663) (#22575)
4ceb171 vulkan: add Intel Xe flash attention optimization kernels (2/3, Xe-LPG Plus/Xe2/Xe3) (#24406)
73c941b mtmd: add various sanity checks (#29276)
0f8a414 metal : gate mul_mm_id src1 rescale behind ggml_prec (#29029)
f95b0d9 ggml : IQ1_M build prefix sums once per block (#28706)
c350a40 Performance tune for gemma4-26b-a4b flash attention shape. (#28450)
9b421fa ui : Accept WEBM video files (#28622)
348f853 jinja: use const for statement::execute and ::visit (#29271)
217f81c server: Add support for binding to multiple addresses (#28690)
828fdf2 spec : support DFlash for HunyuanOCR (#28890)
bfd73a8 convert: add MiMo-V2.6 support (#29257)
a60f9ae cmake : allow repeated find_package calls for llama (#29228)
7ab4ee7 chat : Fix Muse Glimmer tool-call first parser error (#29242)
0ee9435 ci : publish snapdragon builds in release workflow (#29007)
8cfc315 Add close button to UI toasts (#28246)
ec5a12b opencl: add A8 Q4_0 non-MoE dp4a binary kernel (#29055)
c550d2f ci : update Level Zero SDK to v1.33.1 and enable the L0/oneDNN CMake flags in the SYCL job (#29230)
5836771 hexagon: new HMX-optimized GATED_DELTA_NET (#29199)
ff0dbb9 vendor : update cpp-httplib to 0.57.1 (#29239)
fb34fc2 metal : fix mask bounds in flash attention block pre-pass (#29220)
c641dfa test-save-load-state : compare logits with NMSE and feed expected tokens (#29238)
9655061 llama-context : report graph inputs and input tensors during sched reserve (#26625)
b1c2863 cuda: fix sm_70 tile compilation error (#29224)
f4e276a ggml-cuda : convert contiguous tensors four elements at a time (#29155)
e6cef81 cuda : accelerate conv2d with implicit GEMM (#29135)
c21284c ggml : fix dimension and stride truncation in ggml_permute (#29227)
6f41ac5 vendor : update cpp-httplib to 0.57.0 (#29214)
ec91ab5 docker : bump cuda to 13.4.1 (#29207)
bb3c853 sycl : support gated DSV4_HC_PRE and optional HC_POST comb matrix (#29132)
af91114 sycl : pinned memory use right device context instead of 0 (#28895)
1884824 CUDA: Follow up of #25635, refactoring FA shared smem swizzle (#28536)
161755f test-llama-archs : make tensor data stdev configurable and improve help (#29133)
1d72b05 tests/test-backend-ops : allow regex entries in the -o filter (#29204)
542e920 ci : refactor build-self-hosted into backend-specific workflows (#28991)
e0dff58 args: add env vars for temperature, top-p, min-p and penalties (#27380)
982a332 server : do not forward --api-key-file to router-spawned child instances (#28938)
711f60b tests : remove stale comment (#29140)
335b21f ggml-metal : simplify fusion pattern op list declaration (#29206)
26394b4 json: Fixed json enum handling (#28518)
1aa2954 sycl : coalesce MKL-FA softmax loads instead of one work-item per row (#28918)
8034c1d ggml-cpu: ARM Repack kernels for Q1_0 (#23492)
6ad1af5 ci : Upgrade CUDA to 13.4 for Ubuntu CUDA Release Builds (#29202)
0c3626e hexagon: overhaul of buffer and DMA handling to support 64bit mappings + improvements (#29197)
68d9053 cuda : tune MMVQ to MMQ crossover for SM70 (Volta) (#28912)
8aa161b metal : fix deprecation warnings from macOS 27 SDK (#29136)
932a68e webgpu : add fused gdn + cpy (#28976)
62668d6 convert: enable --fuse-qkv for muse-glimmer (#29203)
ce8caa6 CUDA: tune FA for Gemma 4 on Ampere or newer (#29152)
a894dae metal : support arbitrary hc in dsv4_hc_pre (#29169)
3d82ef6 common/peg : handle invalid utf-8 sequences in the AST (#29161)
3cf0325 CUDA: enable sparse fa for qwen4 (#28770)
b23efaa ui: Fix mobile breakpoint + content overflow issues (#29108)
4260903 fix(mamba) : make time-step projection input contiguous (#28832)
9a9f939 metal: add F16 input to the FWHT (#29094)
f072b10 chat : fix gemma4 required tool grammar (#29115)
59657a6 chat : add dedicated Ling 3.0 (Bailing V3) parser (#28682)
e613ef2 hexagon: enable I32 GET_ROWS (#29116)
851cb34 hexagon: add support for GEGLU_QUICK (#29114)
7d4b92b hexagon: enable support for TOP_K op (#29113)
1af554f server : improve startup log messages (#29125)
eb1e1f4 json-schema : accept escaped hyphen in regex patterns (#29127)
5b59b83 metal : add MoE and SSM_CONV fusion optimizations (#28948)
60b06ab metal : fix FA support checks (#29122)
efa28e9 test-llama-archs : generate dummy test vocab (#29084)
59fc5a1 metal : support qwen4exp hc ops (#29000)
b23701f cuda : fix CUB argsort corruption caused by in-place keys (#28389)
60081bb opencl: add support for bin kernel flash_attn_f32_f16_bin (#29046)
2b18470 hexagon: add ROLL op support (#29105)
50631b3 hexagon: im2col update (#29103)
18a04f0 hexagon: HMX flash-attention head_dim padding (support DK=DV=72) (#26539)
ec92815 opencl: add bin kernel kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin (#28678)
4fea119 ggml-cpu: add F16 input to the FWHT (#27779)
5b335f4 ggml : check for allocation failures to prevent crashes (#28149)
542348a Model-Saver: Write the SWA pattern, 15 more architectures roundtrip (#29042)
d663dd3 ci: change ubuntu-latest to ubuntu-24.04 (#29079)
44be98f ggml-webgpu: fix supports_op condition for GET_ROWS (#28978)
911f6cd ggml : handle graph buffer reservation failure (#26070)
bbd488c vulkan: add IQ3_S MMQ matmul kernels (#28822)
dc85f89 vocab : add ufakzeka pre-tokenizer (#29033)
8ed1a55 cmake : fix build when GGML_CPU=OFF and GGML_CUDA=ON (#29026)
bb11ebb gguf-py: fix Q8_1 block size in GGML_QUANT_SIZES (2+2+32) (#29036)
f03cf3e ci : disable GHA cache for copilot (#29068)
bdcbaaf ci : bump android-actions/setup-android to 4.0.4 (#29065)
5c53396 vulkan: raise the hoisted row-id limit for mul_mat_id from 256 to 512 experts (#28501)
972d231 ci : add missing evict-old-files (#29041)
c77ae69 rpc : skip ACCEL devices (#29020)
b49650a model : skip gate_up_exps if TENSOR_SKIP is set (#29014)
7076180 model : extend Nemotron MTP support (#29018)
ebbb185 openvino : Update OpenVINO to 2026.4;fix clangd,MSVC warnings; (#29009)
4ff829e ui: fix removed reasoning menu in single model mode on desktop (#27985)
f172be7 vulkan: split buffers and debug code into separate files, add shared headers (#28732)
87f9c82 ci : add API/ABI check to make-release workflow [no ci] (#28947)
7f6f0c2 chat : add message delimiters to the DeepSeek V3.2/V4 parser (#29008)
81aeaeb gguf : align the data section relative to the GGUF start, not the file (#28993)
c9a5eee sycl : fix the B70 mem allocate error when >19.3GB (#28953)
7490357 vulkan: skip unneeded MoE work in mul_mm coopmat1 path (#25483)
817e5f8 sycl: ssm_conv: fuse the SiLU epilogue into the ssm_conv kernel (#28929)
c57da6f opencl: fix various warnings (#28984)
79bfc1d docs: remove JG as CODEOWNER for test-llama-archs (#29003)
05f2dcf vulkan: fix buffer_reference alignment in im2col shaders (#28996)
35822af vulkan: support qwen4exp hc ops (#28988)
aa39d7a [SYCL] Fix function signature for ggml_backend_sycl_split_buffer_type (#28981)
4bc272f vulkan: work around NV bug with argsort_large.comp (#28975)
fb27a52 TP: fix split state and granularity for fused QKV gemma4, qwen35 (#28965)
c6824a9 ci: switch fast jobs back to github (#28959)
2f3fd02 Enable CUDA graph for MTP draft (#28549)
1ec8188 hexagon: Support for K-Quants Q4_K and Q6_K (#28994)
82324fc hexagon: accept the zeroed rope probe in supports_op (#28995)
7ceed87 models : allow Nemotron-H models to only define layer_norm_epsilon (#28989)
7d6f5d0 model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625)
83078fe CUDA/HIP: improve access patterns in im2col (#28013)
f266648 spacemit : fix wrong transpose function for int16 data (#25161)
6019933 rpc : invalidate cached compute graph when a referenced buffer is freed (#24292)
b04d4e5 Change max context length for auto-fitting with unified KV (#28849)
37b53fd qwen4exp: add hc ops (#28901)
fccf716 HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935)
0bec16e chat : force \n</think> on reasoning budget end for qwen3-coder (#28869)
d4365d9 vulkan: make MUL_MAT_ID BN/2 tail unconditional (#28923)
0a8b29a metal: fix NaN in mul_mm_id when activations exceed f16 range (#26223)
583926e ci : add self-hosted webgpu to hf-jobs (#28712)
e13469a llama-bench: support --version to print build info (#28971)
930e2fa hexagon: add back missing contiguous fast-path and hvx_copy_uu for each run (#28886)
72b590d hex-cpy: use dma if src and dst are contiguous (#28906)
38a5b42 HIP: Enable AllReduce for ROCm (#27825)
9f31776 opencl: choose the MoE expert matmul by batch size for speculative decoding/MTP (#27637)
d1d3c33 ci: build MUSA for only 1 arch (#28944)
6011c34 docs: Rule of thumb for AI review time [no ci] (#28945)
7609846 rpc : hash-cache only weights (#28789)
5431581 cuda: support row-contiguous SUM_ROWS (#26308)
9e71716 models : move build_arch_graph() after graph() template specialization (#28934)
fc82583 vulkan: support sparse Flash Attention (#28105)
77d554b OpenVINO: optimize stateful decode and GPU MoE inference (#28638)
6ec1a7e opencl: add generic ssm_scan (#28881)
1af6c65 ci: bump kleidiai runners from 22.04 to 24.04 (#28885)
1e7bcf3 metal : add FA kernels for HSK=96, HSV=64 (MiniCPM3) (#28599)
0ecb159 ci: Bump CUDA Windows x64 builds to 13.4.1 (#28930)
987498f ci : fix android release (#28936)
4c9233c cuda : enable i16 and i32 for DUP (#28897)
69eb250 cmake : use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR (#28771)
1bc7a5a webui: stop re-probing disabled /tools endpoint on every message (#28646)
7cf1c54 ci : reuse build tag name when used instead of safe one (#28911)
96ffdc4 CI: hip-quality-check: ignore spill added in bfdc321 (#28909)
bfdc321 HIP: fattn-mma: use fp32 accumulation on MFMA devices (#28576)
391fac1 ci : add ubuntu-cuda builds to release (#28186)
41abbfd qwen4exp: enable rms_norm + mul fusion (#28896)
b4fa47d release : added gfx1103 to ubuntu rocm build (#28423)
f3a184b cmake : remove precompiled headers (#28892)
dfe4516 scripts: Add script to verify API/ABI compatibility (#28579)