ggml-org/llama.cpp b12342

on GitHub

Title: CUDA: few-row MMQ for BS 2-24 (#31533)

• CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

• nit fixes

• CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

• few-row quantize: DS4 sum from the quantized values, as in #31247

• few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

• few-row MMQ: dispatch order MMVQ, few-row, MMQ