ggml-org/llama.cpp b12064

on GitHub

hexagon: tiled hvx permute (#31446)

• hexagon: add tiled HVX transpose path for permuted CONT/CPY

Permuted copies where dst dim 0 is strided in src, but some other src dim
is contiguous, used to fall back to HTP_COPY_KERNEL_RESHAPE. That issues
one DMA descriptor per element. The channel-first <-> channel-last
permutes around per-channel norms (e.g. [W,H,T,C] <-> [C,W,H,T] in video
VAEs) ht this path, and these copies ran at about 45 MB/s.

Add HTP_COPY_KERNEL_TRANSPOSE for same-type, same-shape f32/f16 copies
into a contiguous dst.

Host:

• drop extent-1 dims and merge dims that stay adjacent in src into a
  canonical view, stored in the kernel params. There are no minimum
  size limits.
• the copy is treated as a 2D transpose between dst dim 0 and the
  contiguous src dim, batched over the remaining dims.
• tiles are sized to 64 KB per buffer and halved until they fit in
  session VTCM. blkk is capped so the halfword vgather offsets fit in
  16 bits.

Kernel, per tile:

• 2D DMA of blk0 strided src rows (blkk contiguous elements each)
  into VTCM
• vgather transpose in 32x32 (f32) or 64x64 (f16) blocks
• 2D DMA of blkk contiguous rows out to dst

Staged and transposed buffers are double-buffered, so the next tile's
input DMA overlaps the current transpose and output DMA. Tiles are split
across threads and mdev sessions.

Small dims:

• packed mode: when dim k follows dim 0 and dim 0 divides the HVX
  lanes (f32: 2/4/8/16, f16: 2/4/.../32), one vgather writes several
  whole dst rows, and each tile goes out as one contiguous DMA.
• when the contiguous src dim fits in one vector, it is staged exactly
  and the tile is made long along dim 0.

All other copies select the same kernels as before.

Tests: add test_cont cases for perms {1,2,3,0} and {3,0,1,2}, f32 and
f16, at various sizes.

• hexagon: cpy transpose cleanups

• fix vgather halfword offset comments: offsets are unsigned 16-bit, caps are kept lower for speed
• set packed directly in the packed case
• add dim_k == 3 CONT tests (full reversal)

Assisted-by: Claude Opus 5.5

• hexagon: use cpy transpose path for ne[0] == 1 permutes

• drop the nb[0] check in the transpose solver: size-1 dims are skipped, and the canonical tp.nb[0] check covers packed dim 0
• try the transpose path before SAMESHAPE_SAMETYPE, which did one DMA per element for these
• add ne[0] == 1 permute CPY tests

Assisted-by: Claude Opus 5.5

• hex-cpy: further improve cpy

Bring in improvements from aizip branch/PR.

Cleanup kparams and replace divs with fastdiv.

Co-authored-by: Yiwei Shao yiwei@aizip.ai

• hex-cpy: avoid vtcm bank conflicts

• hex-cpy: trim down the comments

• cpy-test: trim down new tests that cover permute scenarios

Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com
Co-authored-by: Yiwei Shao yiwei@aizip.ai