github ggml-org/llama.cpp b11636

latest release: b11639
pre-release2 hours ago
Details

sycl: IQ3 code reorder (#29107)

  • sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill

Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights
are reordered once at load time; the reordered MMVQ dot products then
walk dense super-blocks with dp4a and SWAR sign expansion instead of
gathering bits from the interleaved standard layout. Reorder-aware
dequantization feeds the batched prefill GEMMs so prompt processing
sees the same benefit.

Also harden the oneDNN build check: a missing DNNL package or a
CPU-only DNNL build now fails at configure time instead of silently
producing a degraded backend.

Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP),
-ngl 999 -b 4096 -ub 1024 -fa on:

tg256 @ 0 ctx 12.92 -> 23.54 tok/s (+82%)
tg256 @ 9,216 12.33 -> 21.69 tok/s (+76%)
tg256 @ 64,000 9.76 -> 14.91 tok/s (+53%)
pp8192 833.9 -> 833.6 tok/s (flat)

Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta
-0.5% pp, tg flat), so standard quants are not affected.

The MMVQ GLU fusion widening and rms_norm+scale fusion are
intentionally not part of this change; master already carries them
via #28931.

  • removing wait as per review request

  • sycl : fix fusion.cpp line ending

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.