Anbeeld/beellama.cpp v0.4.5 on GitHub

Changelog

Updated the llama.cpp base from 6fdd0ac89 to 57291f264 (b10689). Notable inherited changes include Qwen3.8 Flash Next (qwen4exp) target support with HyperConnections, QSA sparse attention, PLE n-gram embeddings, GGUF metadata and tensor mappings, conversion, multi-GPU placement, and graph lifecycle integration, plus --kv-unified-per-slot context limits, OpenVINO 2026.3.1, and upstream server, Web UI, conversion, model, and backend updates through that merge point. The public lazy-loading API now uses llama_lazy_mode and llama_model_params::lazy_mode instead of llama_tensor_read_lazy and tensor_read_lazy.
Added owned draft KVarN caches through --spec-draft-type-k/v for draft-simple, EAGLE3, audited Qwen MTP (Qwen3.5/Qwen3.6 dense and MoE), DFlash1/DFlash2, and non-MLA DSpark. All six KVarN widths are selectable with the intrinsic 128-token exact suffix and no draft-tail option, target and draft settings stay isolated, capability checks use the actual MTP cache layers, DSV4/MLA DSpark fails closed because its latent cache is not dense K/V, and unsafe draft-memory fit probes fail with guidance to use -fit off rather than silently omit draft allocations. DFlash attention semantics resolve from the draft model's metadata before context construction; non-causal blocks use materialized K/V with the validated native exact-tail merge, avoiding a context-by-ubatch F32 score matrix.
Extended KVarN to shared Gemma 4 MTP and multi-stream SWA target caches. Gemma 4 MTP reads the target cache's KVarN records and exact-tail rows (configure target --cache-type-k/v kvarn*; explicit draft selections fail closed), the position-addressed SWA ring encodes an independent KV stream per parallel slot with per-stream staging, records, and store/materialization routing on CPU, CUDA, and Vulkan, multi-stream SWA uses materialized attention while single-stream keeps native attention, and server checkpoint, restore, and rollback no longer treat shared-cache MTP drafts as owning draft state.
Added standalone Qwen3.8 MTP sidecars. The draft loader follows the configured draft path, device, and offload settings; the Qwen4Exp MTP graph consumes target hidden states and applies the output HyperConnection mixer, and the converter exports complete sidecars.
Extended standard quantized caches, KVarN, and F16/BF16 precision tails to Qwen3.8 QSA. Sparse selection composes with generic attention, native KVarN routing, and tail handling, with a validated mirrored index-cache slot layout. Prompt checkpoints preserve transactional hybrid attention/index state, rebuild mirrored precision-tail slots, and restore PLE recurrent rows into the selected destination.
Fixed Windows CUDA tensor-parallel speculative decoding for MTP, DFlash1, and DFlash2, covering meta-graph headroom, GQA head partitions, mirrored softmax inputs, CPU scheduler boundaries, Windows graph-projection stack size, DFlash1 backend-sampling fallback, and DFlash2 vocabulary-logit gathering.
Optimized CUDA KVarN decode and speculative verification: wider descriptor reduction with division-free single-stream scan, masked multi-query batches through eight rows with skipped empty splits and K/V reuse across query tiles, occupancy-based 64/128-token geometry, packed fragment loads, half-width quantization axes, and chunked windowed materialization. Fast tiles use padded MMA query/probability storage, record-boundary checks, synchronized shared-memory reductions, and query-count-independent split partitioning. Unsupported shapes retain descriptor-native MMA fallback, and GGML_KVARN_DEBUG_ROUTES reports the selected wide MMA path (#132).
Improved KVarN structured cache allocation. Single-slot decode carries its planned stage assignment into cache mutation instead of rescanning the full context per token, and general stage-liveness scans use a flat sequence/group matrix instead of per-record tree allocations. Unified-cache planning reserves complete monotonic physical segments for interleaved sequences, preventing record-tile aliasing.
Fixed historical KVarN SWA checkpoint restoration: partial SWA checkpoints own their ring records so later prefill cannot evict live anchors, incomplete legacy SWA partial frames and shared-stream ring replacement fail closed, and repeated historical-prefix reuse no longer reprocesses the whole prefix (issue #140).
Fixed DFlash image position validation and SWA capacity (issue #140). Overlapping image positions and the logical gaps after them are preserved for feature injection. Draft generation and rollback use the target's next position rather than the image-expanded token count, preventing the post-image draft-acceptance and decode-speed collapse. Owned, unshared DFlash SWA caches start compact and grow on demand while preserving state, up to the full-context limit, so image feature rows can exceed their logical position extent without forcing full-capacity startup allocation. This applies to ordinary and KVarN caches; explicit --swa-full remains available.
Reset per-sequence MTP hidden-state carry-over before processing a prompt from position zero, while preserving restored carry for nonzero continuations and the final prefill state needed by the first draft.
Corrected compact-tail and KVarN suffix-rollback capability reporting so legal speculative rollback no longer triggers unnecessary checkpoints; tail-state serialization scans once per stream, MTP draft rollback follows the draft token axis with widened-removal and full-clear recovery, KV-tail placement follows the realized cache device, and projector thread counts reach every compatible scheduler backend.
Declined the KV precision tail when no device advertises a native KV-tail attention entry point: the context warns and resolves the request to zero instead of paying the several-times-slower generic tail route, and LLAMA_KV_TAIL_ALLOW_GENERIC=1 keeps it anyway.
Fixed Metal compilation for standard Q2 cache quantization and explicit F16/BF16 conversions; embedded Metal assembly now carries a content digest so sccache cannot serve stale kernels.
Added native Vulkan FlashAttention decoding for standard q2_0, q2_1, q3_0, q3_1, q6_0, and q6_1 K/V caches, removing the CPU fallback that collapsed decode throughput.
Updated release and validation tooling: ROCm runtime images bypass registry build-cache import/export, shared CUDA KVarN kernels are deduplicated with a binary-equivalence verification script, and regression coverage now exercises draft-cache validation, Qwen3.8 MTP conversion and state reuse, DFlash image/SWA growth, and historical checkpoints.

macOS:

macOS Apple Silicon

Linux:

Windows:

Docker:

CPU: docker pull ghcr.io/anbeeld/beellama.cpp:server-cpu-v0.4.5
CUDA: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.4.5
CUDA 12: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda12-v0.4.5
CUDA 13.3: docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.5
ROCm: docker pull ghcr.io/anbeeld/beellama.cpp:server-rocm-v0.4.5
Vulkan: docker pull ghcr.io/anbeeld/beellama.cpp:server-vulkan-v0.4.5
SYCL: docker pull ghcr.io/anbeeld/beellama.cpp:server-sycl-v0.4.5

Browse all container images