github ggml-org/llama.cpp b12543

latest releases: b12547, b12546
pre-releaseone hour ago
Details

metal : CONV_2D tensor API for M5, split-K for all chips (#32542)

  • metal: add a tensor API conv_2d kernel and split long reductions

The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.

Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.

Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.

test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.

CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 19008 -> 5235
[16,16,128,8] k [3,3,128,512] 333 -> 124
[1536,1536,32,1] k [3,3,32,64] 12086 -> 6512
[33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33
[2054,1,768,1] k [7,1,768,768] 3161 -> 699

  • metal: walk the conv_2d patches incrementally, size splits by output

The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.

The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.

CONV_2D, F16 kernel, us/run on an M5 Max:
[19,19,256,16] k [4,4,256,4096] 4998 -> 4512
[1536,1536,32,1] k [3,3,32,64] 6721 -> 5354
[4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236
[2054,1,768,1] k [7,1,768,768] 1050 -> 871
with GGML_METAL_TENSOR_DISABLE:
[4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283

  • nit: share the CONV_2D simdgroup count between host and kernel

kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.

  • metal: fall back to the generic conv_2d kernel for images past 4 GiB

The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.

  • cont : comments

Co-authored-by: Georgi Gerganov ggerganov@gmail.com


Co-authored-by: Georgi Gerganov ggerganov@gmail.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.