ggml-org/llama.cpp b10782

on GitHub

CUDA: Allow concurrent streams per split for multi-GPU (#28198)

• CUDA: Allow CUDA optimization per split for multi-GPU.

Previous guard caused multi-GPU to skip the graph optimization. The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.

The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.

• Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta amangupta052@gmail.com

Co-authored-by: tannerbruhn tannerbruhn@users.noreply.github.com
Co-authored-by: Aman Gupta amangupta052@gmail.com