ggml-org/llama.cpp b11574

on GitHub

hexagon: optimize CPY/CONCAT — round 3, short-rows edition (#30341)

• hexagon: move the converting CPY in blocks of rows per DMA descriptor so short rows are not DMA-latency bound

• hex-cpy: cleanup cpy cursor handling

• hex-cpy-concat: introduce threaded dma/hvx gather-row helper

Co-authored-by: Ruslan Karymov 67752306+karusrus@users.noreply.github.com

• hex-cpy: remove leftover rejections for extended buffers

• hex-cpy: relax the gate for fast-path

• hex-cpy: remove scalar divs and vtcm access and use explicit gather_sync

• hex-cpy: add proper support for mdev work splitting to fastpath

• hex-cpy: optimize dense gather-row tasks

• hex-cpy: unroll gather-rows loops

• hex-cpy: add fastpath for f16

• hex-cpy-concat: support for colapsing 3/4D dims

• hex-cpy: double-buffer support

• hex-cpy: factor out gather-rows into more of an op by itself

• hex-cpy-concat: dedup kernel-params

• hex-cpy-concat: step-down nr to fit instead of hard fail

• hex-gather: replace div with a mask

• hex-gather-rows: cleanup dma pipeline and remove the stalls

• hex-gather: specialize compute funcs

• hex-gather: reduce scalar reg.spills and func calls

• hex-gather: do not recompute rows-per-thread on the npu

• hex-build: add support for building/installing just hexagon

• hex-cpy-concat: remove redundant barrier in concat and move cpy barrier under fence

• hex-gather: annotations for dma-out

• hex-mdev: make all key vtcm_acquire sync state atomic (fixes random hangs in mdev setups)

Co-authored-by: Yiwei Shao yiwei@aizip.ai
Co-authored-by: Ruslan Karymov 67752306+karusrus@users.noreply.github.com