ggml-org/llama.cpp b12080

on GitHub

cuda : gather batched tensor reads into mapped host memory (PR to master) (#31391)

• cuda: gather batched tensor reads into mapped host memory

Assisted-by: Codex

• add comments

Assisted-by: Codex

• simplify the implementation based on checks that exist earlier in the flow

• simplify batched tensor gather indexing

• add conditional dispatch

• address comments - make host registration raii, better test mindful of ci time, assert instead of return false

• cuda : gather small reads and use pinned DMA for large reads

• cuda : use std::vector for host gather descriptors

• cuda : add 8-byte gather copies, generalize tensor test and log registration failures

• common buffer for descriptors

• simplify stream sync + increase to 128k descriptor pinned buf