Details
llama : add llama_prec_policy + model-driven W4A4 path (#24364)
- Rebase and update based on #26675
Signed-off-by: ynankani ynankani@nvidia.com
- CI failure fix(launh_bounds overload on HIP) and cleanup
Signed-off-by: ynankani ynankani@nvidia.com
- Address review comments
Signed-off-by: ynankani ynankani@nvidia.com
- Use ggml tensor instead of name in act policy map
Signed-off-by: ynankani ynankani@nvidia.com
- Address review comments and cleanup
Signed-off-by: ynankani ynankani@nvidia.com
- Address review comments
Signed-off-by: ynankani ynankani@nvidia.com
- Rename changes
Signed-off-by: ynankani ynankani@nvidia.com
- Update ggml/src/ggml-cuda/mmq.cu
Co-authored-by: Georgi Gerganov ggerganov@gmail.com
- MXFP4 dispatch changes for higher src prec
Signed-off-by: ynankani ynankani@nvidia.com
- Refactor and address review comments
Signed-off-by: ynankani ynankani@nvidia.com
- Updates based on review comments
Signed-off-by: ynankani ynankani@nvidia.com
- Apply batched suggestions from code review
Co-authored-by: Johannes Gäßler johannesg@5d6.de
- Address review comments
Signed-off-by: ynankani ynankani@nvidia.com
- Apply patch from review
Signed-off-by: ynankani ynankani@nvidia.com
Signed-off-by: ynankani ynankani@nvidia.com
Co-authored-by: Georgi Gerganov ggerganov@gmail.com
Co-authored-by: Johannes Gäßler johannesg@5d6.de
Website:
Attestations:
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu s390x (CPU)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
- Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries
- Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries
- Ubuntu x64 (ROCm 10.0)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
- Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.4 DLLs
- Windows arm64 (CUDA 13) - CUDA 13.4 DLLs
- Windows x64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (ROCm 10.0)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: