ggml-org/llama.cpp b12284

on GitHub

CUDA: Add OCP FP8 support (#28413)

• Initial CUDA WIP version

• Use fp8 intrinsics

• Do W8A16 for FP8 in GEMV case

(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB

• Optimize GEMV kernel for FP8

• Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

Fixes #28413 (comment)

• Remove trailing whitespace

• Add supported for strided copies to/from FP8 in CUDA

• Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

• Fix odd-rows for convert/getrows, possible with FP8

• Fix FP8 fallback in cuBlas

• Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

• Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

• Add todo about accumulator/precision honoring

This is beyond the scope of the stacked PR

• Avoid races in fp8_fallback

• Allow Hopper and other BW SKUs for FP8

• Use intrinsic over explicit manual conversions

• Resolve TODO

As per spec, we don't have to preserve signs

• Add commend on cublaslt workspaces

• Use convert_unary_cont_cuda over block dequant in convert.cu

• Use get_rows_cuda_float for fp8 kv cache

• Make FP8 copies work on all IHV's

• More cleanup

• Simplify by falling back to shared cuBLAS helper

• Fuse weight_scale multiplication for FP8 also

• More code-cleanup

• Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9 was incorrect fix

• Use F32 activations in fallback mmvf path

• Explicitly link against cublaslt

• Move FP8 conversion intrinsics into ggml_cuda_cast

Cleaner this way

• WIP src1/prec for fp8 in MMVF

• Remove comment

• Keep only f8 x f8 KV cache for now

• Just do FP8 x FP32 x FP32 in MMVF as its fast enough

• Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta amangupta052@gmail.com

• Add batched FP8 GEMMS in cublas as suggested by code review

#28413 (comment)

• Extract fp8x2 -> float2 load into helper

• Searapte copy checks to/from fp8

• Compiler massaging to avoid repeated address offsets

Co-authored-by: Aman Gupta amangupta052@gmail.com