ggml-org/llama.cpp b11709

on GitHub

model : add Kolibri-1 (Aleph-Alpha) support (#30673)

• model : add Kolibri-1 (Aleph-Alpha) support

Kolibri-1 is a bilingual German/English MoE reasoning model with 78B
parameters and 3.46B active per token: 50 layers, 384 routed experts plus
one shared expert.

Attention interleaves four sliding layers at a 513 window with one full
layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE.
GQA runs 48 query heads against 4 KV heads at head_dim 128 with a
QK-RMSNorm, and every block carries four RMSNorms, so the attention and the
MLP output are normalized before they rejoin the residual stream. Routing
picks the experts from logits + expert_bias and weights them with the
sigmoid of the unbiased logits, without renorm.

The converter reuses AfmoeModel for the per-expert checkpoint layout and
the expert selection bias, and writes the shared expert and the per layer
sliding window pattern. The released FP8 checkpoint needs no special code,
it goes through the generic weight_scale_inv path. Three tensor names are
added to the mapping (post_attn_norm, post_ffn_norm,
moe.router.expert_bias) and the 128k vocabulary registers the qwen2
pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA
and the sliding window pattern.

Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash)

• Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Add fusion records for kolibri1

• Update src/models/kolibri1.cpp

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co