ggml-org/llama.cpp b11671

on GitHub

model: add Muse Spark Open (text, vision and audio) (#30616)

• model: add Muse Spark Open (text, vision and audio)

Muse Spark Open is a latent space MoE: the residual stream stays at 8192
while the 256 experts live in 4096, with a projection down and up around
them, top-8 sigmoid routing with a correction bias. Attention interleaves
three sliding layers at a 2048 window with rope for one global layer
without rope, with a sigmoid output gate and a weightless QK-norm.
Residuals are norm preserving, their gate is folded at conversion time.

Vision reuses the Muse Glimmer tower. Audio is a causal convolutional
pyramid with no attention and no spectrogram, four stages folding 16, 6,
5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.

The quantizer keeps the latent projections and the attention gate at
Q8_0, a little over 1% of the weights, since the whole residual stream
passes through them and the low-bit types turned the output into a
repeating loop. The model saver writes the post norm epsilon, and the
arch is registered in test-llama-archs.

Co-authored-by: Xuan Son Nguyen son@huggingface.co
Co-authored-by: Pedro Cuenca pedro@huggingface.co

• Muse Spark: drop the redundant forward expand

• Muse Spark: drop the redundant forward expand and tighten the comments

• Update src/llama-quant.cpp

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Update src/llama-graph.cpp

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

• Update conversion/muse_spark.py

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

Co-authored-by: Xuan Son Nguyen son@huggingface.co
Co-authored-by: Pedro Cuenca pedro@huggingface.co
Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co