github ggml-org/llama.cpp b12552

latest releases: b12555, b12556, b12554...
pre-releaseone hour ago
Details

ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)

  • ggml-openvino: round the GPU KV length up to a power of two

Review notes (follow-ups, not blocking):

  • convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
    copies it into the ov::Tensor. Writing the padded rows straight into
    padded.data() avoids the second allocation and copy (up to 16 MB per
    call at a 16K context).
  • The padding is skipped when the mask is not contiguous, and the KV
    inputs are then sliced to more rows than the mask has columns. Masks
    are always contiguous today; a GGML_ASSERT would make that explicit.
  • compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
    is_stateful_enabled() is local to ggml-openvino.cpp.
  • ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

  • ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

  • ggml-openvino: Remove solution C and keep A as default

  • ggml-openvino: match the output count before reusing a compiled NPU prefill

The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.

  • ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding

  • ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix

  • compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
    ran before the new one and still exported result_norm, so the
    result_norm exclusion had no effect. Keep only the new block.
  • compute_node_dynamic_dims checked is_activation_input twice in a row.
    Remove the second copy.
  • process_view_input_new created the dynamic Split on split_dim. With
    stateful execution the input drops the leading size-1 axis, so use
    split_axis, which already accounts for the rank difference.
  • ggml-openvino: hash the output flags into the model-cache fingerprint

The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.

  • ggml-openvino: support f16 SCALE and f16/bf16 TRI

test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.

SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.

  • ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
    layer

The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.

  • enable mamba based models on OV CPU-GPU

  • revert changes in tests

  • docs : add Falcon-H1 to the OpenVINO validated models

Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.

  • ggml-openvino: simplify the SSM_SCAN build check and clean up
  • Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
    always provides the op. Drop the no-op LANGUAGE CXX source property.
  • Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
    with -Wextra-semi, which fails builds with fatal warnings.
  • Add ggml_openvino_is_static() for the repeated NPU or
    GGML_OPENVINO_FORCE_STATIC check.
  • Share the f16 and f32 KQ mask padding code.
  • Drop the redundant inline on in-class static functions.

Co-authored-by: Suhas Somashekar suhas.somashekar@intel.com
Co-authored-by: yzhou51 yaowei.zhou@intel.com
Co-authored-by: Ananth, Swathi swathi.ananth@intel.com
Co-authored-by: Beheshti nazanin.beheshti@intel.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.