ggml-org/llama.cpp b11879

on GitHub

CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows (#29948)

• cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows

For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.

• The loaders take the gate weights as a second pointer and fill the weight
  tile with up and gate rows interleaved in chunks of 8, so each thread holds
  an up value and the matching gate value (rows r and r + 8 of an MMA tile).
• The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
  per thread. The fused write-back computes the GLU from registers and writes
  half the rows.
• The fused path uses the regular tiling (no stream-k) with the column tiles
  innermost so that neighbouring blocks share weight tiles.
• Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.

• cuda: assert the fused MMQ gate argument and trim redundant comments

• cuda: move the fused MMQ hint check to should_fuse and add TODOs

• cuda: pass prec_src1 to the fused MMQ write-back

• cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back

• cuda: require has_fusion explicitly, drop the fusion-only src1 precision check

• cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile

• cuda: inline the MMQ tiling-efficiency check

• tests: one fused gate/up MMQ case per type and GLU op

• cuda: always launch the fused MMQ kernel with one block per tile

• tests: shorten the fused gate/up MMQ test comment

• cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs

• cuda: run the fused MMQ kernel on the plain tiling path

• cuda: put the column tiles innermost in the MMQ plain tiling path