ggml-org/llama.cpp b12328

on GitHub

ggml-backend : prefetch only inputs computed by earlier splits (#32100)

• ggml-backend : prefetch only inputs computed by earlier splits

The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.

• llama : skip the MoE cache when its device cannot run the cached mul_mat_id

Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.