vLLM Metal v0.32.0

v0.32.0 tracks the vLLM v0.32.0 core. The release contains 51 merged pull requests from 14 contributors since vLLM Metal v0.31.0.

Highlights

• vLLM 0.32.0 core: the plugin moves to vLLM 0.32.0 and adapts to its removed mamba block-size field, its on-device image normalization, its renamed block-table sizes, and its dropped check on narrower diffusion canvases (#1158).
• Experimental PyTorch MPS backend: VLLM_METAL_BACKEND=mps runs vLLM's own model definitions on Model Runner V2 through PyTorch MPS and the paged Metal attention kernels, for greedy requests on unquantized FP16/BF16 text models such as Qwen3, Hunyuan dense, OLMo 2, and LFM2/LFM2.5, with output logprobs and a fused Q/K RoPE kernel; MLX stays the default (#858, #880, #1001, #1052, #859).
• Sleep mode: --enable-sleep-mode now works at level 1 on the MLX backend; sleeping frees the KV cache while the weights stay in unified memory and also returns a KV offloading host pool to macOS, and wake_up rebuilds the cache with an empty prefix cache (#1102, #1118, #1166).
• GGUF and speech-to-text in every install: gguf, librosa, and numba are now regular dependencies, so GGUF models and Whisper or Qwen3-ASR transcription work from the installer and Homebrew without extras (#1161).
• DiffusionGemma structured reads: DiffusionGemma applies the structured-read request options (diffusion_seed_canvas, diffusion_pinned, diffusion_max_steps, diffusion_read_only, diffusion_constrained) instead of refusing them, so the upstream structured-decisions server answers yes/no, choice, and score questions with probabilities; docs/structured_decisions.md shows the setup (#1077).
• Speculative decoding on more targets: n-gram drafting now runs on hybrid Gated DeltaNet targets such as Qwen3.5, Qwen3.6, and Qwen3.8, and experimental DSpark loads self-contained Qwen3 Speculators checkpoints such as RedHatAI/Qwen3-4B-speculator.dspark (#1072, #1092).
• LoRA for vision-language models: --enable-tower-connector-lora (experimental, off by default) also adapts the Qwen3-VL and Qwen3.5 vision tower and merger, and Hugging Face PEFT adapters with model.language_model.* names load both on the native multimodal path and when Gemma 4 or Qwen3.5 runs on the mlx-lm text backbone (#1107, #1168).

Installation

Requirements:

• Apple Silicon with macOS 15 or newer.
• Native arm64 Python 3.12.

Stable channel:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash -s -- --stable

Homebrew:

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

The installer downloads prebuilt vllm-metal and vLLM wheels. It does not invoke a C++ or Metal compiler.

Release Artifacts

The release attaches a native Python 3.12 wheel for macOS 15 arm64:

vllm_metal-0.32.0-cp312-cp312-macosx_15_0_arm64.whl

The compatible upstream core is the prebuilt vLLM v0.32.0 macOS arm64 wheel.

Engine, Cache, and Performance

• Standalone MLA models such as DeepSeek-V2/V3, GLM-4.7-Flash, Kimi, and MiniCPM3 now commit KV pool memory only as blocks are written instead of zero-filling the whole pool at startup, except under draft-model speculation, so MiniCPM3-4B-4bit idles at about 4.5 GB instead of 23 GB at --gpu-memory-utilization 0.6 on a 48 GB M5 Pro (#1080).
• LoRA decode groups rows by adapter, as prefill already did, so decode rows batched with long prefill chunks no longer kill the engine with an oversized Metal allocation, and batched LoRA decode runs faster (#1106).
• On M5, mixed prefill/decode batches whose longest decode row has 4,096 or more tokens of context now also split their decode rows out of the NAX prefill kernel, and split-out decode rows, on M5 and on the tiled path elsewhere, run on the GQA decode kernels where the layout allows; VLLM_METAL_DISABLE_GQA_DECODE=1 keeps them on the per-token kernel (#1060, #1061).
• Batched GQA decode sizes its scratch by each request's own context instead of the longest one, so one long request among many short ones needs far less scratch, and the GQA decode kernels also take the 24-query-head, 4-KV-head, head-dim-256 layout of models such as Qwen3.8-27B with a 32-token kernel block (#1129, #1089).
• On M5, long prefills run faster: full NAX attention tiles skip their bounds checks, and Gemma 4's head-dim-512 layers, which already ran on NAX, now take that path too by splitting their output into two 256-column slices; VLLM_METAL_DISABLE_NAX=1 still turns NAX off (#1088, #1170).
• DFlash, DSpark, and n-gram drafting cap each draft at the request's remaining max_tokens less one, so requests close to their output limit no longer draft and verify tokens they cannot return (#1078, #1139).
• With async scheduling, requests preempted by reset_prefix_cache(reset_running_requests=True) no longer resume one token ahead and skip a token of output (#1100).

Models and Serving Features

• Qwen3-VL and dense Qwen3.5 checkpoints accept video input, where a video_url request used to stop the engine for every client (#1113).
• Mistral Small 3.1 checkpoints (mistral3) now load and serve text through mlx-lm by default, with image requests refused by a 400, instead of failing at load (#1131).
• A GGUF file can take its config and tokenizer from a quantized checkpoint, such as an mlx-community 4-bit, FP8, or AWQ repo given as --tokenizer, which used to fail with a quantization mismatch or a wrong-shaped model (#1132, #1142).
• Pooling requests with options Metal cannot serve, such as "use_activation": false on /v1/embeddings, get a 400 naming the parameter instead of killing the engine (#1134).

Compatibility and Operational Changes

• vllm-metal and vLLM core are both versioned 0.32.0 for this release, and the installer and the Homebrew updater now fetch vLLM's macOS wheel under its new cp38-abi3 tag (#1158).
• MLX stays pinned to an exact version, now 0.32.2, with mlx-vlm pinned to 0.7.6 and nanobind to 2.15.0 (#1116, #1155).
• transformers requires 5.16.1 or newer and is capped below 5.20.0, following vLLM 0.32.0, which also admits huggingface_hub 2.x; huggingface_hub still requires 1.31.0 or newer (#1158).
• The gguf, stt, and all extras were removed, since their packages now install by default; only dev remains (#1161).
• Metal keeps host-side pixel normalization for Qwen3-VL and Qwen3.5 by turning vLLM 0.32's mm_device_do_normalize off, because the MLX vision towers expect normalized pixels (#1158).
• thinking_token_budget, which the MLX backend accepted and then ignored, now gets a 400 naming the parameter, as on the MPS backend (#1162).
• With KV offloading's automatic host pool (no --kv-offloading-size) on a Mac that is already paging at startup, the pool now shrinks to fit, and when it cannot, the error says the machine is paging instead of advising a flag that was never set (#1063, #1115).
• Docs: the structured-decisions guide for DiffusionGemma, a PyTorch MPS column in the supported-models table, and the README news (#1077, #859, #1164).

Known Boundaries

• The PyTorch MPS backend is experimental: it serves unquantized FP16/BF16 dense text decoders on one device, with LFM2 ShortConv as the only state layer and head sizes of 64, 96, 128, 256, or 512; MoE, MLA, multimodal, pooling, diffusion, and other hybrid models fail at startup, as do LoRA, speculative decoding, KV offloading, TurboQuant, parallelism, and sleep mode (#858, #880, #1102).
• MPS requests must be greedy, and a request that leaves temperature unset gets a 400, because vLLM's default of 1.0 or a checkpoint default such as Qwen3's 0.6 applies; penalties, allowed_token_ids, bad_words, prompt logprobs, logprob_token_ids, and structured outputs are refused too, so Hunyuan also needs --generation-config vllm to drop its default repetition_penalty (#858, #859).
• Sleep mode supports level 1 on the MLX backend only: a level-2 request gets a 500 and, like any refused sleep, leaves the engine paused until /wake_up; the MPS backend and speculative methods other than n-gram fail at startup with --enable-sleep-mode, and freeing the KV cache is refused for standalone MLA and speech-to-text models (#1102, #1166).
• N-gram drafting on hybrid GDN targets needs float32 recurrent state (--mamba-ssm-cache-dtype float32, which Qwen3.5 and Qwen3.6 configs already resolve to), the lazy GDN kernels, and an attention head size of at most 256, runs on the text-only path, and can be slower than plain decoding on diverse long contexts; other speculative methods on hybrid targets, and n-gram on Mamba-2, KDA, or ShortConv hybrids, fail at startup (#1072).
• DFlash and DSpark are experimental: LoRA and TurboQuant are refused, serving losslessness is not established, and they can be slower than target-only decoding; Speculators checkpoints must be Qwen3 drafts with the full target vocabulary and their own embeddings and output head (#928, #999, #1092).
• Hybrid GDN prefix caching remains experimental: cache-on versus cache-off parity still diverges on a subset of divergent-suffix prompts, unchanged from v0.31.0.
• Nemotron-H and Granite 4.0 hybrid keep one private recurrent-state slot per request, so prefix caching is off for them; an explicit --mamba-block-size equal to the KV block size is now dropped with it instead of rejected, because vLLM 0.32 no longer records that it was set (#1158).
• --watermark-config requires Model Runner V2, so it fails at startup on the default MLX backend, and the opt-in MPS backend serves only greedy requests, which vLLM does not watermark.
• KV offloading supports only uniform full-attention (GQA, MHA, or MQA) models on the MLX backend, without draft-model, DFlash, or DSpark drafting, pooling, speech-to-text, or pipeline and data parallelism; on a Mac already paging, an automatic pool now shrinks to fit and startup fails only when one request and one host chunk do not fit, while an explicit --kv-offloading-size can still fail it (#998, #1046, #1063).
• LoRA adapters that use vLLM's modules_to_save for classifier or score heads are not supported on Metal yet (#1022).
• --enable-tower-connector-lora adapts only the Qwen3-VL and Qwen3.5 vision encoders, and other multimodal models ignore it with a warning (#1107).
• vLLM 0.32's per-module LoRA scaling for rank_pattern and alpha_pattern (vllm#59801) and its per-replica reseed for data parallelism (vllm#59788) do not reach Metal yet (#1158).
• DiffusionGemma is experimental, text-only, and MLX-only; speculative decoding, pipeline parallelism, LoRA, and TurboQuant fail at startup, and top_k, top_p, prompt logprobs, logprobs=-1, and a diffusion_canvas_length smaller than the served canvas are refused (#994, #1076, #1158).
• vLLM 0.32's built-in /v1/systemone reads DiffusionGemma with a 16-token canvas, so on Metal it answers only when the served canvas_length is 16 and returns a 400 for larger canvases; with the 32-token canvas in docs/structured_decisions.md, use the upstream example server as shown there (#1077, #1158).
• Video input covers Qwen3-VL and dense Qwen3.5 only: --video-pruning-rate fails at startup, and PaddleOCR-VL and Gemma 4 stay image-only (#1113).
• Mistral Small 3.1 serves text only, an image request to it under VLLM_METAL_MULTIMODAL_MODE=multimodal-native stops the engine, and Ministral 3 and Devstral Small 2 still fail at load (#1131, #1130).
• The Gemma 4 vision sidecar handles images only, and stays off under speculative decoding, when Gemma4Processor cannot be built, or for a repo id that is not fully cached; run hf download <repo> first (#820).
• On M5, VLLM_METAL_MM_PREFIX_PATH=recompute can prefill Gemma 4 images faster, because the default kernel path takes image-block steps off NAX (#863).
• GGUF does not support LoRA, IQ or T-quants, MoE, SSM, hybrid or vision models, fused-QKV or sharded files, or Q2_K/Q3_K exports with IQ4_NL rows (#1066, #1019).
• prompt_logprobs steps project logits for every prompt row, which startup profiling does not reserve; a lower --gpu-memory-utilization leaves room (#897).

Contributors

@JC011118, @JWriter20, @Luan-Fuzi, @LxYuan0420, @RobbieJ, @StevenWang-CY, @WindChimeRan, @XXXXRT666, @abhijithneilabraham, @marcomd, @ricky-chaoju, @suntp, @syang-genai, @varad-more

Full changelog: v0.31.0...v0.32.0