vLLM Metal v0.31.0

v0.31.0 tracks the vLLM v0.31.0 core. The release contains 194 merged pull requests from 24 contributors since vLLM Metal v0.30.0.

Highlights

• vLLM 0.31.0 core: the plugin moves to vLLM 0.31.0 and adapts to its moved multimodal-input check and renamed hybrid-backend platform hook (#1022).
• More KV cache, less resident memory: startup profiling reserves logits only for rows a step can sample, image steps included, and KV pool memory is committed only as blocks are written, except for hybrid, MLA, and draft-model setups (#897, #898, #1000, #1009, #931, #959).
• GQA decode kernels: long-context FP16/BF16 decode on supported grouped-query layouts routes to new split-KV kernels, for single requests and batches; VLLM_METAL_DISABLE_GQA_DECODE=1 opts out (#715, #960, #991).
• Gemma 4 image input: vision-capable Gemma 4 checkpoints, including E2B, serve images through an mlx-vlm vision sidecar, with bidirectional attention inside each image block (#820, #830, #844, #992, #749, #813).
• GGUF K-quants: Q2_K to Q6_K and Q5_0/Q5_1 files for dense Qwen2, Qwen3, Llama, and Mistral load quantized, and loading a 7B file peaks at about 5 to 8 GiB instead of over 30 GiB (#870, #995, #1002, #808, #849, #929).
• KV cache offloading: --kv-offloading-size keeps evicted KV blocks in a host RAM pool taken from the --gpu-memory-utilization budget, with an optional fs disk tier that survives restarts (#998).
• New experimental models: Ling-3.0 Tiny (Bailing V3, hybrid MLA + KDA) and DiffusionGemma block diffusion (--diffusion-config) serve on the paged Metal path (#607, #994).

Installation

Requirements:

• Apple Silicon with macOS 15 or newer.
• Native arm64 Python 3.12.

Stable channel:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash -s -- --stable

Homebrew:

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

The installer downloads prebuilt vllm-metal and vLLM wheels. It does not invoke a C++ or Metal compiler.

Release Artifacts

The release attaches a native Python 3.12 wheel for macOS 15 arm64:

vllm_metal-0.31.0-cp312-cp312-macosx_15_0_arm64.whl

The compatible upstream core is the prebuilt vLLM v0.31.0 macOS arm64 wheel.

Engine, Cache, and Performance

• The startup profile measures the same overhead on every launch, so the KV budget is no longer sometimes too large (#836).
• Draft-model speculative decoding counts the draft in the startup memory profile and keeps lookahead KV in scheduler-allocated blocks instead of a private pool (#993, #843).
• Sliding-window decode skips KV blocks before the window, which speeds up long-context decode on models such as Gemma 4 (#785, #838, #867).
• Outside M5 NAX, TurboQuant, and speculative decoding, mixed prefill/decode batches run decode rows with 4,096 or more tokens of context on the faster split-KV kernel (#851).
• MLA models such as DeepSeek-V3 and GLM-4.7-Flash prefill large cached-context chunks faster through MLX SDPA (#833, #846, #848).
• Concurrent short-context decode on MLA models such as DeepSeek-V3 and GLM-4.7-Flash runs faster by sharing attention calls across rows (#908, #913, #927, #974).
• Long TurboQuant prefills run faster on the NAX or tiled prefill kernel through a workspace taken from the KV budget, by default on M5 or with VLLM_METAL_TQ_PREFILL=1; VLLM_METAL_TQ_PREFILL=0 turns it off (#853, #895, #990).
• TurboQuant no longer turns a K block of 32 identical values into NaN (#891).
• XLM-R embedding models such as BAAI/bge-m3 run attention through MLX SDPA in their FP16/BF16 dtype, lowering latency (#811).
• Requests resumed after preemption get correct sampling penalties, M-RoPE positions, and draft-model catch-up for their first new token (#892).

Models and Serving Features

• Gemma 4 12B (gemma4_unified checkpoints such as mlx-community/gemma-4-12B-it-4bit) now generates coherent text (#845).
• Qwen3-ASR no longer returns noise for repeated audio, and Qwen3-VL and PaddleOCR-VL under --dtype float16 no longer corrupt cached images, both regressions introduced in v0.30.0 (#810).
• Speech-to-text loads quantized Qwen3-ASR checkpoints, follows --dtype instead of always float16, and turns prefix caching off so an identical repeated Qwen3-ASR request no longer kills the engine (#946, #944, #818).
• GGUF files downloaded with huggingface_hub 1.32 or newer load again, BF16 tensors load exactly, and GGUF models compute in the requested --dtype instead of float32 (#828, #823).
• Remote GGUF references accept the new quant tags, such as Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M, prefer a single .gguf over its own shards, also under HF_HUB_OFFLINE=1, and reject unsupported families before downloading (#1007, #997, #1019, #1054, #1073).
• Seeded requests no longer spend random draws on unfinished prompt chunks, which made their output depend on what else was scheduled (#917).
• Gated DeltaNet layers in Qwen3.5/3.6/3.8 and Qwen3-Next scale the q/k normalization epsilon by the head dimension, as mlx-lm does (#919).
• Experimental DFlash and DSpark block drafters run through --speculative-config for plain greedy requests on a single-device Qwen3 target, with prefix caching (#928, #986, #1044, #834, #900, #910, #915, #940, #977).

Compatibility and Operational Changes

• vllm-metal and vLLM core are both versioned 0.31.0 for this release.
• MLX stays pinned to exactly 0.32.1, and startup now stops with a clear error when a different MLX is installed (#925, #947).
• transformers requires 5.10.4 or newer and is now capped below 5.18.0, following vLLM 0.31.0; huggingface_hub requires 1.31.0 or newer (#1022).
• With vLLM 0.31.0, --enable-return-routed-experts is rejected at startup, and pipeline parallelism no longer needs --no-async-scheduling, though an explicit --async-scheduling still fails (#1022).
• Vision-capable Gemma 4 checkpoints that are local or fully cached, E2B and E4B included, now start the vision sidecar by default; VLLM_METAL_MULTIMODAL_MODE=text-only or --language-model-only keeps text-only serving (#820, #992, #845).
• In sidecar mode, startup fails with TurboQuant or with --max-num-batched-tokens below 282, and the multimodal profile pass can leave no KV budget at a low --gpu-memory-utilization (#830, #992).
• A new default-on startup probe shrinks the KV pool to free memory less a safety reserve, with a warning, when macOS would compress or swap to back it, and startup fails if too little is left for the KV cache; VLLM_METAL_KV_COMMIT_PROBE=0 skips it (#930, #1006, #1034, #1083).
• Invalid non-boolean vllm-metal environment variables, such as a VLLM_MLX_DEVICE other than gpu/cpu (once treated as CPU), now stop startup with one error naming each (#873, #869, #861).
• vLLM no longer crashes at shutdown on PyTorch 2.13.0 (#802).
• Docs: recipes for 16, 32, and 64 GB Macs, a macOS benchmarking guide, the gguf extra and GGUF support scope, Gemma 4 checkpoints, and Qwen3.5 image input (#1071, #989, #815, #911, #879, #937, #827).

Known Boundaries

• Hybrid GDN prefix caching remains experimental: cache-on versus cache-off parity still diverges on a subset of divergent-suffix prompts, unchanged from v0.30.0.
• Nemotron-H and Granite 4.0 hybrid keep one private recurrent-state slot per request, so prefix caching is off for them.
• --watermark-config requires Model Runner V2 and is not supported on Metal.
• KV offloading supports only uniform full-attention (GQA, MHA, or MQA) models without draft-model, DFlash, or DSpark drafting, pooling, speech-to-text, or pipeline and data parallelism, and can fail startup on a machine already paging (#998, #1048, #1046).
• LoRA adapters that use vLLM 0.31.0's new modules_to_save for classifier or score heads are not supported on Metal yet (#1022).
• DFlash and DSpark are experimental: LoRA and TurboQuant are refused, serving losslessness is not established, and they can be slower than target-only decoding (#928, #999).
• DiffusionGemma is experimental and text-only; speculative decoding, pipeline parallelism, LoRA, and TurboQuant fail at startup, and top_k, top_p, prompt logprobs, and per-request diffusion options are refused (#994, #1076).
• The Gemma 4 vision sidecar handles images only, and stays off under speculative decoding, when Gemma4Processor cannot be built, or for a repo id that is not fully cached; run hf download <repo> first (#820).
• On M5, VLLM_METAL_MM_PREFIX_PATH=recompute can prefill Gemma 4 images faster, because the default kernel path takes image-block steps off NAX (#863).
• GGUF does not support LoRA, IQ or T-quants, MoE, SSM, hybrid or vision models, fused-QKV or sharded files, or Q2_K/Q3_K exports with IQ4_NL rows (#1066, #1019).
• prompt_logprobs steps project logits for every prompt row, which startup profiling does not reserve; a lower --gpu-memory-utilization leaves room (#897).

Contributors

@FENP, @HanningLin, @Lavmee, @Luan-Fuzi, @LxYuan0420, @RobbieJ, @SanjanaB123, @StevenWang-CY, @TheVerdman, @Tialo, @WindChimeRan, @abhijithneilabraham, @b-wu8, @begcdn, @blueberry808, @luke-edward, @marcomd, @mhdimo, @nathanzhu144, @pythongiant, @ricky-chaoju, @suntp, @tak-bro, @varad-more

Full changelog: v0.30.0...v0.31.0