ggml-org/llama.cpp b11463

on GitHub

sycl: accelerate GLM MLA prefill with MKL flash attention (#29171)

• sycl: accelerate GLM MLA prefill with MKL flash attention

GLM-4.7 Flash uses an MLA shape with 576-wide Q/K heads, a 512-wide
V head, GQA 20, and F16 KV. The SYCL dispatcher rejects this shape
because the normal MKL flash-attention gate requires matching K/V
widths and caps the head dimension at 512, so prompt processing falls
back to the substantially slower TILE kernel.

Admit only the validated 576/576/512, GQA-20 F16 shape to the existing
MKL pipeline. Keep all other mismatched K/V shapes on their current
fallback paths.

Handle GLM's V cache as a narrower strided view of K rows. Select the
strided F16 descriptor when row stride is padded, and alias K/V
dequantization buffers only when their logical widths match. Restrict
the stride exception to a real V view sharing K's row stride.

Add the exact 576/512, GQA-20 prompt-path backend test.

On an Intel Arc Pro B70 at master e613ef2, pp8192 improves from
432.80 to 1292.29 tok/s (2.99x, +198.6%). tg256 remains unchanged
within noise at 45.67 versus 45.65 tok/s. The exact MLA test passes
and debug output confirms MKL dispatch.

• sycl: store MKL flash attention scores in F16

Keep the QK GEMM output in F16 instead of F32. The online softmax still
converts each score to F32 for its max, exponent, and sum, so the
per-element math is unchanged apart from score rounding, and the F32
matrix was being written only to be consumed as F16 probabilities.

The F32 score matrix is the largest flash-attention intermediate on this
path; storing it as F16 halves its size and traffic. This builds on the
coalesced softmax loads from 1aa2954, which read each score row
cooperatively, so the smaller dtype pays off.

Measured on an Intel Arc Pro B70 with the dispatch from the previous
commit, -ngl 999 -b 4096 -ub 1024 -ctk f16 -ctv f16 -fa on:

pp8192 1583.9 -> 1657.1 tok/s (+4.6%)
pp64000 610.0 -> 684.5 tok/s (+12.2%)
pp131072 ~354 -> 402.1 tok/s (+13.5%)

tg256 at 8k context is unchanged (32.64), and the FLASH_ATTN_EXT suite
shows no new failures. The exact GLM MLA backend cases pass against CPU.
Adjust the ~354 baseline figure if you prefer citing only measured pairs (the 131k dispatch-only point came from the equivalent maintained build). Optionally add Assisted-by: per the contribution guidelines since AI contributed to the change.

• Revert "sycl: store MKL flash attention scores in F16"

This reverts commit 265f974.