ggml-org/llama.cpp b10643

on GitHub

hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501)

• hexagon: use non-host bufs by default and make the backend fully async

• hex-hb: remove optional hostbuf support and fix async copy

• hex-unary: relax supported unary check

• hex-bufs: use same get_alignment for host bufs

• snapdragon: bump android_platform to 34

• hex-rows: super hacky get/set rows for q8_0

• hex-get-rows: fix q8_0

• hex-get-rows: supprot for f16 and cleanup for q8_0

• hex-get-rows: generic macros and specialized thread funcs

• hex-get-rows: add DMA pipeline, vtcm_layout and kernel params

• hex-set-rows: fix q8_0 support, add dma and tracing

• hex-tests: override nmse threshold for HTP of Q8_0 quants

• hex-fa: add support for Q8_0 with inplace dequantizers

• hex-get-rows: simplify type dispatch

• hex-rows: simplify GET/SET_ROWS DMA pipeline

• hex-async: add events, set/get-tensor-async and rest of the async api support

• hex-repack: use slice instead of expert in repack functions

• hex-cpy: update event/async-cpy logging

• hex-set-rows: optimize smaller tensors

• hex-geglu: fix perf regression with larger tensors

• hex-get-rows: add missing header

• hex-set-rows: add missing header

• hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default

• hexagon: do not reject ops with non-heaxon buffers

• hex-get-rows: apply >=32 restriction only for q8_0

• hex-res: bump vtcm acquire timeout to 10 seconds

• hex-bufs: add support for cloning buffers between sessions to speed up tensor copies

• hex-async: rework event recording and batch flushing and integrate with meta backend

• hex-bufs: improved handling of repacked tensors

• hex-repack: handle get_tensor_2d offsets

• hex-dev: add support for devices with multiple NPUs

• hex-sync: add support for sync tokens to synchronize npu devices for async splits

• hex-mmap: cleanup mmap calls and add a retry for robustness

• hex-sync: add failsafe if sync wait gets stuck

• hex-sync: use sync_seq to check for completed events

• hex-sync: rotate tokens for extra robustness

• hex-devs: add supprot for legacy device names for now

• hex-bufs: add support for auto-cloning buffers from diff sessions

• hex-fusion: simplify and optimize htp-opnode fusion handling

• hex-sync: override opnode name so that it shows up in the profiles

• hex-trace: update scripts to handle multiple devices

• hex-sync: bump the size of the opbatch queue and number of sync tokens

• hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma

• hex-sync: add graph-flush threshold to avoid single op batches

• hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops

• hex-bufs: introduce tensor->extra and shadow_bufs for repacking

• hex-l2: flush tiny tensors inline

• hex-sync: use explicit l2flush for sync tokens

• hex-extra: track weight flags via tensor extra

• hex-fence: rename sync to fence

• hex-repack: proper handling of set-tensor-2d in the shadow_buf

• hex-trace: remove obsolete opstage mask that we used for profiling

• hex-env: remove obsolete use_hmx variable

• hexagon: new unified run.py and build.py and updated docs

• snapdragon: update run script to auto-escapt test-backend-op -p argument

• hex-scripts: fix trailing spaces

• hex-scripts: fix flake8 warnings

• snapdragon: cleanup dst lib/bin dirs before copying new build

• hex-ops: add support for allreduce

• hex-ar: improved allreduce with dma pipeline

• hex-ar: align macros

• hex-ar: consistent use of fence_seq

• hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback

• hex-ar: add proper synchronize handling for ALLREDUCE

• hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold

• hex-ar: bump block size to improve dma efficiency

• hex-ar: fused ALLREDUCE+ADD

• hex-ar: cleaner fence buffer management

• hex-ar: futher allreduce tweaking to remove race conditions

• hex-ar: add simple solver and remove non-dma kernels

• hex-ar: add row-broadcast to fuse with bias ADD

• hex-fence: pass seq numbers via op_params

• hex-ar: allow for both entry/exit seq for completing entry wait

• hex-ar: align macros

• hex-ar: do not refetch broadcast row

• hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things

• hex-fusion: fix incorrect MUL_MAT reordering

• hex-mm: make fused 2x and 3x matmuls more generic

• hex-fusion: move tensor fusion tagging to graph_compute

• hexagon: make sure to copy tensor->extra by value

• hex-get-rows: fix offset calc with row-chunking

• hex-repack: get_tensor_2d fixes for non-zero offsets

• snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default

• hex-devices: use legacy device nameing by default to ease the transition

• hex-devices: hardcode CDSP domain IDs for current devices for now

• hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values

• hex-optrace: more robust handling of the fence events