ggml-org/llama.cpp b11195

on GitHub

ggml-cpu: tiled mul_mat for k-quants (#27851)

• Added tiled mul_mat.

For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8,
one routine per quent. Then microkernel computes 16x16 tiles before writing out
256x256 float reults to main memory.

Tests/benches in tests/test-tiled-mulmat.cpp. 3-6x speed improvement
for large matmul, break even at 4096x64 * 64x4096, 80% performance (net
loss) for GEMV. Error rates trivial (order of 1-e04 max, 1-e05 rmse).

• Fixes for ARM/windows builds

• more windows fixes, ggml-cpu.h isn't visible in MSVC for some reason

• unified iqp + tiled on the Q5_K, IQ4_XS set for benchmarking, updated benchmark

• Fixed accidental removal of llama_build_and_test(test-backend-ops.cpp)

• First integration of iqp code

Co-authored-by Bartowski 3266127+bartowski1182@users.noreply.github.com

• Cleaning up declaration of iq unpacking helpers to align with the bit unpackers

• Removed iqp path

• Fix cross-platform warnings

• Disabling benchmarks unless explicitly enabled

• Fix backend_init for DLL-based builds, add self and bartowski to CODEOWNERS for tiled

• Put benchmarks behind a flag

• kernel fix for AVX2, iq quants

• Fix for asan, leaking memory in test-tiled-mulmat and avoid stack use after return

• guarding env flags with std::call_once

• Simplified repacking for VNNI to a single call per macrotile

• No threadlocals anymore, aligned wdata access

• Doing aligned reads since we ensure alignment with padding in wdata

• Eliminated per-thread gather of Q8_K rows in mul_mat_id, we now gather/repack in a single pass. Repack method now takes pointer array to support both dense/normal and mmid paths. Interface with ggml-cpu.c simplified as a result

• Unified/simplified dispatch and support checks. Put details on wdata needed inside the kernel.h body, simplified interactions with ggml-cpu.c.

• Cleanup includes and whitespace, update src1_repack to return false if we don't need a special repack, so the common case is handled by driver

• Better detection of win32 and additional whitespace fixes

• Gating fuzz tests behind a parameter and some extra prints to try and fix slow CI hosts

• Optimized AVX2 kernel

• Changed interleave format and added ability to interleave in-place after dequant

• Repacks now happen in-place, 16x64 microtiles are independent of each other

• Only repack rows in groups of 16 as they're needed. Save work in low n_rows cases and optimize L1 usage in other cases

• Use long panels for memory-bound regime (M <= 16), reintroduce IQP path for benchmarks

• Fix unused warnings and cleanup. Improved IQ dequantization speed.

• Removed separate process benchmarks

• Revert "Removed separate process benchmarks"

This reverts commit 0688cf4.

• AVX2 optimizations and guards for tests on windows

• Removed temp perf harness

• Remove perf-mulmat from build

• Removed IQP path, simplified tests to not use sub processes

• Cleaning up alignment of wdata

• Whitespace fixes and aligning L2 workspace to clean 512kb boundaries

• Update ggml/src/ggml-cpu/tiled/tiled-kernel.cpp

Co-authored-by: Georgi Gerganov ggerganov@gmail.com

• Cleanup merge-duplicated declaration of test-backend-ops target

• Undo accidental line deletion in ggml.c

Co-authored-by: Georgi Gerganov ggerganov@gmail.com