github ggml-org/llama.cpp b12136

latest releases: b12138, b12137
pre-release2 hours ago
Details

fit : account for skipped NextN layers when setting n_gpu_layers (#31569)

  • fit : account for skipped NextN layers when setting n_gpu_layers

Without MTP, fit counts n_layer + 1 layers while the model loader counts
n_layer_all + 1 for n_gpu_layers, including the NextN layers which hold no
weights. A fit result that offloads all layers then leaves layer 0 on the CPU
("offloaded 65/66 layers" for Qwen3.8-27B on two GPUs).

Add the skipped NextN layers to n_gpu_layers and to the tensor split of the
device that holds the output layer so the loader assigns the layers that
fit measured.

Fixes #31531

  • fit : pass hp_ngl_skip by reference like the other hparams

  • fit : add hp_ngl_skip to common_get_device_memory_data

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.