Anbeeld/beellama.cpp v0.4.7 on GitHub

Changelog

Added true rectangular KVarN storage and attention for 64-dimensional K/V heads, including 64x128 K records, 128x64 V records, WHT64 transforms, target and owned draft caches, full-context and iSWA routing, and F16/BF16 precision tails. Existing D128, D256, and D512 record layouts and state formats remain compatible, while unsupported backend placements continue to fail closed.
Optimized CUDA D64 KVarN with block-parallel quantization, batched workspace staging and commit, grouped materialization, split portable/GQA attention, and generated split-MMA decode kernels. Decode consumes compressed records directly at every KV length; prompt batches above the native rotated-query limit use transient tiled materialization. Packed K2/K4 decoding now handles odd indirect and SWA offsets without slowing aligned pairs.
Optimized CUDA D128 KVarN decode for low GQA with exact GQA2/GQA4 geometries, an eight-warp split-128 route with a measured shallow-context crossover, cooperative staging of complete packed records, and shared-word unpacking of adjacent 3-, 5-, and 6-bit values, raising occupancy and halving deep-context split/combine work.
Improved HIP/ROCm KVarN correctness and performance. Qualified gfx1100 D256 prompt prefill uses F32-accumulator WMMA, larger prompt batches publish complete softmax metadata, and portable attention sizes scratch storage for the full head dimension. RDNA4 D256+ remains on the portable route until its F32 tiles qualify. HIP also gains wave32 top-k handling, compact MoE MMQ tiles, quantized vec-dot and dequant-matvec kernels, strict dequant controls, and a fix for the top-k radix upper boundary; the unsafe APU host-mapped placement is disabled.
Added independent physical ubatch sizing for model-backed drafts with --spec-draft-ubatch-size (-ubd), defaulting to 128 or, for parallel DFlash/DSpark, the capacity needed for all slots' noise blocks. Explicit -ubd values remain unchanged; draft contexts inherit the target -b logical capacity without changing target -ub. Owned draft KVarN contexts also gain --spec-draft-kvarn-window-chunk, defaulting to 2048, so their transient prefill workspace can be bounded independently of the target context.
Added shared Qwen4Exp MTP sidecars. They borrow the target embedding and output projection while retaining their own draft block and output mixer; legacy self-contained sidecars remain supported, and conversion maps the MTP mixer into the shared layout.
Improved server prompt-cache and checkpoint restoration. Long unstructured prompts gain periodic stable anchors, retained cache bodies can be restored transactionally without unnecessary clearing and rebuilding, and response timings now separate slot selection, RAM-cache work, and checkpoint prepare, commit, and restore phases.
Preserved input marking through chat-template tokenization so special-token text supplied by users or tools is treated as content rather than template control syntax. The same protection applies to text-only, multimodal, continuation, and token-counting paths, fixing output stalls caused by stray message tags.
Fixed compact precision-tail updates when a batch reuses a tail slot. Only the globally last write is committed across tokens and sequence owners, avoiding duplicate-destination races on CPU and GPU backends.
Fixed tensor-split speculative graph reuse after Meta context growth. Resetting the rotating graph context now recreates every reserved subgraph slot, preventing stale graph pointers when speculative topology changes. Attention planning uses the query-token dimension, and the original four-argument KVarN planner symbol remains available for binary compatibility.
Fixed Qwen4Exp QSA serving on KVarN caches. Index-mirroring hybrid memories keep one attention row per physical cache cell, KVarN-only stage slots no longer reach the plain index cache, and state save/restore selects stage rows with the mapping the store actually uses.
Enabled direct-record CUDA attention for qualified owned DFlash1/DFlash2 non-causal KVarN draft caches, including supported full-context, SWA, and exact-tail layouts. Admission checks the actual layer backend and final attention operation; unqualified shapes and backends retain materialization or fail closed. Multi-stream SWA remains materialized after a native target-token mismatch, while tested single-stream and unified multi-slot routes matched the materialized oracle. Persistent cache and checkpoint formats are unchanged.
Made CUDA exact-tail attention materialize unsupported KVarN body layouts through the standard fallback instead of aborting, and made CUDA graph and dequant optimizations honor each device's controls rather than a single global decision.
Fixed llama-perplexity KLD processing to honor --batch-size directly, so compressed-logit working memory scales with the requested batch instead of a vocabulary-derived row limit. Baseline and candidate comparisons should use matching --batch-size and --ubatch-size values.
Fixed a first-decode abort from stale scheduler hash entries after graph reserve and tensor probing: inserting into the hash set now invalidates the reset state, so the next reset always clears it.

macOS:

macOS Apple Silicon

Linux:

Windows:

Docker:

CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.4.7
CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.4.7
CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.4.7
CUDA 13.3: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.7
ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.4.7
Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.4.7
SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.4.7

Browse all container images