ggml-org/llama.cpp v0.9.0

on GitHub

Overview

This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. ggml is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.

Highlights

• Quant: OCP FP8 E4M3 tensor type (saturating mode) with conversion helpers and a new ftype (#27512)

• llama: flash_attn_ext_rows - more efficient unified KV for multi-seq, restoring non-unified KV prefill performance (#29510)

• ggml: batched tensor copy API (ggml_backend_tensor_copy_n/set_n/get_n) superseding the _2d API, implemented for Metal and CUDA (#30881)

• For a 3rd year in a row, llama.cpp is a featured repo in the Github Octoverse report:

  

API changes

• llama.h: add LLAMA_FTYPE_MOSTLY_F8_E4M3 (#27512)
• mtmd.h: add MTMD_GEN_AUDIO_TYPE_CHATTERBOX and n_voices to mtmd_gen_audio_info (#30993)
• mtmd.h: add predefined-voice APIs: mtmd_gen_audio_voice_name(), mtmd_gen_audio_get_voice(), mtmd_gen_state_set_ref() (#30993)
• mtmd.h: add mtmd_bitmap_is_voice() and mtmd_gen_audio_read_tensor() (#30993)
• mtmd.h: add MTMD_VIDEO_TEMPLATE_QWEN3VL video prompt template (#31619)
• server: GET /props reports the model's predefined TTS voices; the TTS endpoint accepts predefined voice names (#30993)
• server: /v1/responses now merges extra_params (seed, min_p, ...) into the converted chat-completions response (#30959)

New models

• Clover-35B-A3B-Preview diffusion model, AR mode (#31306)
• Limite 1B - Violetto (#29433)
• Metis-1.6 (Praxis ~2.9B and Logos ~10.2B): hybrid Mamba-2 / GQA attention with fixed-depth recurrence and LatentMoE (#31134)
• Mistral Large 4 (#31376)

Core changes

• gemma4: graph optimizations with massive prefill gains for E2B and E4B (#31356); create token_embd before the tied output in gemma3n and gemma4 (#31394)
• LFM3: enable NVFP4 GGUFs (#31608), enable tensor split (#31431), quantize the audio codec decoder (#31386)
• spec dec: save and restore MTP hidden state in context checkpoints (#30141) and record LFM3 layer inputs (#31598)
• dflash: copy hidden features via device copies (#31506); qwen4exp: write the conv state tail without an intermediate cont (#31443)
• model loading: reuse the tensor staging buffer and skip the direct-IO bounce buffer (#31321)
• kv-cache: keep Hadamard rotation constants on the GPU (#31073) and fix attention tensor rotation in layers with shared KV (#31406)
• kv-cache: never return 0 kv_rows for sequences without cells (#31570) and refuse rollbacks past the snapshots of the last ubatch (#31368)
• Gemma4 MTP assistant: follow ctx_other devices and flash-attention setting (#31635); DSV4: remove simple batch splitting (#31300)
• chat: strip the qwen3-coder think block with reasoning_format none (#31380) and end reasoning budget forcing on a prompt end tag (#31374)
• chat: allow a constraint type after the gpt-oss final channel (#31556); common: fix -hf tag matching inside longer quant names (#31620) and bound control vector layer indices (#31336)
• tools: reject mismatched shards in GGUF split merge (#31579), fix imatrix --nextn crash (#31522), fix gguf-py np_roundf dropping the sign (#31310)
• vendor: miniaudio 0.11.26 (#31459) and cpp-httplib 0.64.0 (#31400); embed an app manifest in all Windows executables (#31329)

Multi-modality changes

• add Chatterbox TTS support (#30993)
• add Nemotron 3.5 Transcribe support: offline transcription in 40 languages with prompt context and speaker attribution (#31471)
• Pocket TTS: support predefined voices without voice cloning (#31429)
• accurate support for the qwen3-vl video prompt format (#31619)
• resize Gemma 4 images without letterbox padding (#30922) and fix InternVL CLS token order (#30886)

Server changes

• cap shared decision prefixes at the first output (#30710)
• restore the speculative checkpoint for drafts longer than n_rs_seq (#31350)

UI changes

• add MCP group checkboxes in the tools settings (#31303) and request persistent storage on startup (#31299)
• keep chats at the bottom until the reader scrolls up (#31311) and show the submit button when the input has content (#31324)
• reveal the collapsed sidebar from its logo (#31307) and align alert dialog styling (#31312)
• keep queued steering messages in their own conversation (#31304) and show an interrupted state for tool calls that never returned (#31314)
• disable model deletion the server would refuse (#31305) and prefer a loading model in pre-selection (#31326)
• add a Playwright harness for recording demo videos (#31318) and run local browser tests in installed Chrome (#31596)

ggml changes

• v0.29.0: ggml v0.29.0 adds an OCP FP8 E4M3 tensor type, a new ggml_flash_attn_ext_rows op where each query slice attends to its own list of K/V rows, a batched tensor copy API, and initial support for multi-node tensor parallelism through a new multi-process communicator interface. Core improvements include bounding tensor views by the bytes their strides reach, zero-copy GGUF export of host-buffer tensors, and vm_allocate-based allocations on Apple platforms. Performance work spans all major backends: new SVE kernels in the CPU backend, DPAS flash attention for quantized KV in SYCL, fused MoE MMQ in CUDA, faster MXFP4 mat-vec in Metal, split-K tuning in Vulkan, and a large expansion of quantized GEMM/GEMV coverage in OpenCL. The Hexagon backend gains support for several new ops, including the lightning indexer, top-k mask, conv2d and upscale.

Assets

Nightly build: b12110

More info

• Releases and versioning of ggml-org projects

Changelog since v0.8.0

cd959ba llama.cpp : bump version to 0.9.0 (#31646)
13ec8f6 sync : ggml
cca14f6 ggml : bump version to 0.29.0 (ggml/1680)
e853ec9 llama : follow ctx_other devices and flash attention in Gemma4 assistant (#31635)
feaf800 ggml : Gate RVV code on the target extension (#31607)
4c4ca5b common : do not allow _ in sidecar (#31639)
162e2bb ggml : fix bf16 (#31609)
6900fb5 opencl: improve q2_0 GEMV for Adreno (#31577)
bbb02c9 ggml: bound views by the bytes their strides reach (#31584)
960b1d5 ggml : write host-buffer tensor data directly in gguf writer (#31144)
4b29d02 common : fix -hf tag matching inside longer quant names (#31620)
bee2332 hexagon: add LIGHTNING_INDEXER support (#31289)
149f564 mtmd: accurate support for qwen3-vl video prompt format (#31619)
b976279 model, mtmd: add chatterbox TTS support (#30993)
7071ba3 hexagon: add TOP_K_MASK op (#31552)
6cb3e67 gemma4: graph optimizations (massive gains on pp for E2B and E4B) (#31356)
6b9c0cf hexagon: faster TOP_K for the backend sampler (#31279)
d1c4e5b ui: run local browser tests in installed Chrome with its sandbox on (#31596)
20a51f2 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
0f35d6d spec : save and restore MTP hidden state (#30141)
a464960 ggml-zdnn: fix fattn test failure (#31583)
9309bf1 model : Add support for Metis-1.6 models (#31134)
d1044bc model : enable NVFP4 LFM3 ggufs (#31608)
28b812d mtmd : add Nemotron 3.5 Transcribe support (#31471)
749db7f qwen4exp: write the conv state tail without an intermediate cont (#31443)
0943ca5 ui: add Playwright harness for recording demo videos (#31318)
f9448cd model : record layer inputs in LFM3 for spec dec (#31598)
0978a6e ggml: avoid signed overflow in ggml_roll bounds check (#30375)
0ef2730 conversion : use original model tensor names in _flush_nvfp4_experts (#31427)
8fc0376 cuda: potential fix for Q8_1 correction sum (#31247)
740251a cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)
71efc55 Add nccl support for multi-node tensor parallelism (#28967)
19e8a86 tools : reject mismatched shards in GGUF split merge (#31579)
259d917 meta: support GLM-5.3 next sm tensor (#30499)
36ee3ca metal : fix cpy kernel for dst non-contiguous in dim 0 (#31581)
fac3a89 vulkan: skip maxStorageBufferRange pre-check for non-compute tensors (#31114)
b74298c delta_net_base: refactor conv state (#30878)
37b0f9c vulkan: fix out-of-bounds NVFP4 UE4M3 scale lookup (#31079)
ca8daad llama : drop the redundant cnv from llm_attn_mask (#31558)
0f663d2 dsv4.1: fix llama-context tests with partial offload (#31292)
00f113e cuda : prevent premature eviction of graphs about to be used (#30662)
e6192de opencl: add initial iq4_xs support (#31339)
4402e65 chat : allow constraint type after gpt-oss final channel (#31556)
74101b3 ci : update the HIP VGPR spill ignore list for the kv_rows FA kernel argument (#31550)
713319e webgpu: add ARANGE support (#31455)
ee17f29 kv-cache : never return 0 kv_rows for sequences without cells (#31570)
51fe42c hexagon: tiled hvx permute (#31446)
2cbca38 dflash: copy hidden features via device copies (#31506)
29c720e docs: update s390x build docs (#31546)
d02b4c0 mtmd: Pocket tts - support for voices without voice cloning (#31429)
e4b6c8c model: add Clover support (#31306)
1b6935b llama : flash_attn_ext_rows - more efficient unified kv for multi-seq (#29510)
3f8af15 imatrix : fix --nextn crash (#31522)
cd8427d context : simplify setters (#29730)
6b00d56 readme : fix links in README.md for maintainer PRs (#31487)
458fcfc docs: CANN backend supports BF16 on 910B, document GGML_CANN_NO_PINNED (#31249)
6ed8120 ggml-zdnn: share buffers with host (#31481)
ea24b95 ggml : use vm_allocate instead of posix_memalign on all Apple platforms (#31437)
84d9ce0 opencl: add topk_mask (#31441)
dcbb091 vulkan: add tensor-parallel allreduce function (#25051)
07ec7c8 vulkan: improve small-n split-K selection on RDNA3 and RDNA4 (#30146)
cb02250 vulkan: avoid redundant scans when zeroing skipped rows (#31266)
2c93f0b sycl : fast ggml_sycl_comm_exchange() on Intel Arc Pro B70s (#30956)
5e18eb0 sycl: dmmv code cleanup (#31325)
5c2ea1a ui: disable model deletion the server would refuse (#31305)
23426d7 ui: align alert dialog styling with other overlay surfaces (#31312)
90080ce ui: keep queued steering messages in their own conversation (#31304)
64be95b ui: show interrupted state for tool calls that never returned (#31314)
b12b405 ui: don't reload a loading model on select, prefer it in pre-selection (#31326)
aac6da8 docs : fix Vulkan example command in build.md (#31530)
00ef136 tests : merge identical if branches (#31523)
5969566 jinja : implement map('filter') (#24033)
a7a490a ggml : fix clang-cl build with AVX512 BF16 (#31276)
f72dc30 docs: fix Huawei ascend brand (#31463)
fcfe4b5 requirements : minor adjustments for python 3.14 wheels (#31534)
4e910a8 cmake : add explicit thread linkage (#31414)
1c5ed70 webgpu : fix CI by keeping scratch alloc sizes monotonic in batch length (#31422)
785322d requirements : fix numpy dependencies (#31508)
dde879b sycl: Try PkgConfig for LevelZero includes / libs (#31076)
060b2fe opencl: add initial iq3 support (#31338)
a91eab6 hexagon : fix DMA transfers with strides > 24 bits (#31442)
d5ef408 mtmd : fix InternVL CLS token order (#30886)
79bce0c tests : enable Gemma 4 in test-llama-archs (#31401)
bc428e4 llama : enable tensor split for LFM3 (#31431)
f6312ad vendor : update miniaudio to 0.11.26 (#31459)
0138937 llama : fix rotation of attention tensors in layers with shared KV (#31406)
ddf48f8 reshaping so mm on 2d instead of 3d tensor for lfm sc (#30831)
f3586e6 server: cap shared decision prefixes at the first output (#30710)
2871678 server : fix /v1/responses endpoint to accept extra_params (#30959)
7bd1808 mtmd : resize Gemma 4 images without letterbox padding (#30922)
0ec4027 opencl: add fwht (#31434)
1052f7e chat : strip the think block in the qwen3-coder with reasoning_format none (#31380)
8956c76 sycl: fix f32 to q2_0/q4_0/q5_0 rounding in CPY (#31120)
5047748 rpc : delete fs_path_to_utf8() (#31424)
792fcf4 opencl: add initial iq1 support (#31341)
a34de86 common : delete fs_path_to_utf8() (#31423)
aa588a3 common : drop the Windows argv reconstruction (#31417)
889f2ec quant : allow quantization for LFM3-Audio codec decoder (#31386)
1ab23c3 vendor : update cpp-httplib to 0.64.0 (#31400)
27b8464 ci : add missing ccache-clear to windows behavior check (#31413)
d63fbfe cmake : embed an app manifest in all Windows exes (#31329)
e216f4e ci : extract most apple jobs into daily workflow (#31408)
344846b gemma3n, gemma4 : create token_embd before the tied output (#31394)
fb59972 kv-cache : keep rotation constants on the GPU (#31073)
ab3768c memory: refuse rollbacks past the snapshots of the last ubatch (#31368)
e0f142d opencl: add initial q2_0 support (#31340)
3c809f4 common : end reasoning budget forcing on an end tag from the prompt (#31374)
36a0e2f sycl: FLASH_ATTN_EXT failures (#30761)
418f905 ggml : add batched tensor copy API (#30881)
eb4e244 ROCm/HIP: add RDNA4_M family for gfx1170/gfx1171 (#31373)
68d20b8 Quant: OCP FP8 E4M3 support (#27512)
0893ec9 skills: specify rule about ggml_build_forward_expand (#31375)
78feec0 model: mistral le chonk (#31376)
0ac1a7f llama : reuse the tensor staging buffer and skip the direct-io bounce buffer (#31321)
7df8b87 server : restore the spec checkpoint for drafts longer than n_rs_seq (#31350)
8ab29fa ui: keep chats at the bottom until the reader scrolls up (#31311)
f5d4d75 ui: show submit button when the chat input has content (#31324)
3ed915e ui: reveal the collapsed sidebar from its logo and toggle it there (#31307)
f8fcef2 ui: add MCP group checkboxes in tools settings and fix indeterminate checkbox colors (#31303)
b7cde58 ui: request persistent storage on startup (#31299)
3dd0f60 doc: update ops for opencl (#31347)
78febba HIP/CUDA: enable sparse attn on CDNA (#31259)
abb76ab model : Support Limite 1B - Violetto (#29433)
41be0bd common: select A100 cores for SpaceMiT default threads (#30284)
22183e5 opencl: initial support for iq2 (#31267)
6d19453 metal : faster MXFP4 mat-vec, full threadgroups for unary/binary/GLU (#31075)
fade717 tests : remove duplicate FA test cases added in #31125 (#31357)
c9daee1 ggml-metal : fix unary kernels compile on older Metal toolchains (#31360)
8a199ad hexagon: support for UPSCALE op (#31331)
da6c30a hexagon: improve DDR bw utilization in transposed CONCAT (#31277)
f7c01c5 [SYCL] add DPAS flash attention kernel for quantized KV cache (#31125)
bf48103 sycl : vectorize reordered-layout dequantization stores (Q4_0, K-quants, IQ3, MXFP4) (#30976)
6d5ac3c tests : fix comp_padding CUDA graph reservation (#31344)
7ac5727 ggml-openvino: slice view inputs of SCALE (#31334)
2e0d86b Hexagon: IM2COL improvement - block based HVX parallelization (#31050)
e09a78a tests: fix CI flakes in server cancel and sleep tests (#31330)
369dcf3 ggml-cpu: add SVE implementation of ggml_gemv_q8_0_4x8_q8_0 (#30585)
d0ff631 docs : fix DSpark confidence threshold flag name (--spec-draft-p-min) (#31178)
cf24317 common, mtmd: bound control vector layer indices and M-RoPE position loops (#31336)
74a4fb0 opencl: add tiled q4_k GEMM for adreno (#30412)
5220213 hexagon: add conv2d support (#30995)
c3df4b7 Speed up IQ4 quant creation with pre-computed halfway points (#31201)
ac8e169 ggml-cpu: SVE 256 bit implementation of gemm_q6_K_8x8_q8_K kernel (#27472)
c02b5c6 ggml-cpu: SVE 256-bit Implementation of mxfp4 gemv kernel (#28141)
ff0d396 ci(s390x): switch to Ubuntu 26.04 runners (#31317)
838093f llama : remove simple batch splitting for DSV4 (#31300)
604cc43 codeowners : refresh teams and maintainer PRs link (#31316)
39febd2 sycl: work-group argmax for short TOP_K rows (#31110)
f5222f2 cuda: fuse MoE (MUL_MAT_ID) gate/up and GLU in MMQ (#31188)
28cb47c gguf-py: fix np_roundf dropping the sign (#31310)