vLLM Metal v0.33.0

v0.33.0 tracks the vLLM v0.33.0 core. The release contains 70 merged pull requests from 23 contributors since vLLM Metal v0.32.0.

Highlights

• vLLM 0.33.0 core: the plugin moves to vLLM 0.33.0. vLLM published its 0.33.0 release on the v0.33.0rc3 tag (the same commit), so the installer and the Homebrew updater download the vLLM wheel from there (#1249).
• Model Runner V2 by default on MLX: the MLX backend now runs under vLLM's Model Runner V2 configuration and scheduler contract by default, still through MetalModelRunner; VLLM_USE_V2_MODEL_RUNNER=0 opts back out (#1199, #1238).
• Native MLX sampling by default: random sampling runs in MLX instead of torch, including prefill, mixed batches, and seeded requests, which draw from per-request keys; logit_bias and min_tokens are now accepted on the MLX backend instead of refused, and VLLM_METAL_NATIVE_SAMPLING=0 goes back to the torch sampler (#1196, #1201, #1233, #1239, #1096).
• More multimodal input: Gemma 4 E2B/E4B take audio and video, MLX-quantized Qwen3.5/3.6 MoE checkpoints take images and video, and LFM2-VL/LFM2.5-VL and LFM3-VL take images (#1175, #1180, #1193, #1182, #1232).
• New text models: LFM3, Recursive Flint, Muse Glimmer, EmbeddingGemma 2 embeddings, and MiniCPM4 MLX checkpoints serve, and Ministral 3 no longer fails at warm-up (#1212, #1200, #1120, #1214, #1086, #1156).
• Qwen3.5 and IQ quants in GGUF: text-only Qwen3.5 GGUF files serve, including files that embed an MTP block, and IQ4_NL, IQ4_XS, and IQ3_S weights load, so llama.cpp IQ3_M and unsloth UD-Q3/Q4 exports work locally and as remote repo:TAG references (#1181, #1202, #1228, #1230, #1240).
• Experimental PyTorch MPS backend grows: VLLM_METAL_BACKEND=mps adds Qwen3.5 Gated DeltaNet text models, LFM2-VL image input, greedy EAGLE3 speculative decoding, a fused residual-add RMSNorm kernel, and Qwen3-TTS through vLLM-Omni, and installs pin a PyTorch nightly that submits MPS work earlier (#1241, #1243, #1179, #1194, #1109, #1117).
• MoE expert streaming: --additional-config '{"moe_expert_offload": {"cache_gb": <GiB>}}' keeps MoE experts on disk and streams the routed ones through a bounded cache, so MoE models larger than memory can serve, slower than resident; docs/moe_offload.md has the details (#1104).

Installation

Requirements:

• Apple Silicon with macOS 15 or newer.
• Native arm64 Python 3.12.

Stable channel:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash -s -- --stable

Homebrew:

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

The installer downloads prebuilt vllm-metal and vLLM wheels. It does not invoke a C++ or Metal compiler.

Release Artifacts

The release attaches a native Python 3.12 wheel for macOS 15 arm64:

vllm_metal-0.33.0-cp312-cp312-macosx_15_0_arm64.whl

The compatible upstream core is the prebuilt vLLM v0.33.0 macOS arm64 wheel, attached to vLLM's v0.33.0rc3 tag.

Engine, Cache, and Performance

• Checkpoints whose weights are float16 but whose config says bfloat16, such as many mlx-community 4-bit Llama, Mistral, Qwen2.5, and OLMoE conversions, no longer run every layer in float32, which made them up to about twice as fast on an M3 Pro (#1186).
• In mixed prefill/decode batches, decode rows move to the decode kernel once their contexts add up to 512 tokens instead of when one row reaches 4,096, raising prefill-heavy serving throughput about 19% on M5 and 11% on M1 Max (#1159).
• Qwen3-VL and Qwen3.5 rotate packed M-RoPE batches in one fused call, about doubling Qwen3.5-0.8B throughput at concurrency 48 (#1114).
• Qwen3.5 on the native mlx-vlm path and Muse Glimmer project logits only for the rows they sample again, instead of the whole vocabulary on every prefill row (#1217, #1220).
• On M5, eligible BF16 head-dim-256 prefills with six query heads per KV head, such as Qwen3.8-27B, use a grouped NAX kernel that cuts cached long-context TTFT by 11-19%; VLLM_METAL_NAX_GQA_PREFILL=0 turns it off (#1225).
• TurboQuant decode at P256 runs on the GQA producer kernels, with VLLM_METAL_DISABLE_GQA_DECODE=1 as the switch back (#1174, #1185).
• Hybrid Gated DeltaNet models keep the decay gate in float32 through the lazy kernels, use a packed-row kernel for eligible recurrent prefills, and stash and rescan less state during n-gram verification (#1101, #1143, #1224).
• N-gram, draft-model, DFlash, and DSpark speculation stop drafting for a request that has just emitted EOS or a stop token (#1203).
• --kv-cache-memory-bytes now sets the MLX KV budget instead of being ignored, and --num-gpu-blocks-override sizes the KV pools before allocation instead of only limiting the scheduler (#803, #1145).
• The PyTorch MPS backend sizes its KV cache to fit one Metal buffer, so the documented command no longer fails at startup on a 48 GB Mac, and Qwen3.5 there no longer crashes on resumed prefill with a large pool (#1141, #1244).
• Metal workers no longer build tokenizers and processors that vLLM already builds, and replaying lazily loaded paged-attention inputs no longer hangs (#1207, #1206).

Models and Serving Features

• Gemma 4 E2B/E4B accept audio, keeping the first 30 seconds of longer clips as vLLM's GPU runner does, and video; 26B and 31B keep video off (#1175, #1180).
• MLX-quantized Qwen3.5/3.6 MoE checkpoints such as mlx-community/Qwen3.6-35B-A3B-4bit serve image and video input in auto mode instead of returning a 400; --language-model-only keeps the text-only load and its larger KV budget (#1193).
• LFM2-VL/LFM2.5-VL serve image input on the paged ShortConv path, where images used to be dropped, on both backends, and LFM3-VL MLX checkpoints such as LiquidAI/LFM3-VL-2B-MLX-4bit serve images with --trust-remote-code (#1182, #1243, #1232).
• LFM3 text models such as LiquidAI/LFM3-1.7B-MLX-4bit serve with --trust-remote-code (#1212).
• RecursiveSI/flint-int4 serves on the paged path with its sliding-window layers and learned attention sinks (#1200).
• Muse Glimmer checkpoints such as mlx-community/Muse-Glimmer-30B-4bit serve text, where they used to produce garbage, and image requests get a 400 (#1120).
• google/embeddinggemma-2 serves text embeddings with --runner pooling --language-model-only (#1214).
• openbmb MiniCPM4 MLX checkpoints that declare model_type: minicpm4 load with --trust-remote-code (#1086).
• Ministral 3 serves text with its position-dependent query scale instead of failing at warm-up (#1156).
• Remote GGUF references such as unsloth/Qwen3.5-4B-GGUF:BF16 skip mmproj-*.gguf projector files instead of failing on multiple matches (#1211).
• With VLLM_METAL_BACKEND=mps, EAGLE3 drafts such as RedHatAI/Qwen3-8B-speculator.eagle3 speed up greedy decoding of dense text targets, and Qwen3-TTS serves /v1/audio/speech through vllm-omni serve with the new omni extra; docs/pytorch-mps.md and docs/qwen3-tts-mps.md show the setup (#1179, #1109).

Compatibility and Operational Changes

• vllm-metal and vLLM core are both versioned 0.33.0 for this release; vLLM's wheel comes from its v0.33.0rc3 tag because vLLM cannot reuse the v0.33.0 tag (#1249).
• MLX stays pinned to an exact version, now 0.32.4, with nanobind 3.1.0; mlx-lm and mlx-vlm are pinned to git commits until releases carry the Flint and LFM3-VL fixes (#1125, #1200, #1223, #1232).
• transformers requires 5.16.1 or newer and is capped below 5.21.0, following vLLM 0.33.0; huggingface_hub still requires 1.31.0 or newer (#1249).
• The installer and Homebrew install a pinned PyTorch nightly (torch==2.16.0.dev20261013) after vLLM, for every backend; a plain pip install of the wheel keeps vLLM's torch (#1117).
• New omni and build extras, and all is back as build,omni,dev; source builds now go through MLX's MetalExtension and need CMake and the Xcode Metal toolchain (#1109, #1125).
• The installer no longer falls back to vLLM's cp312-cp312 wheel name (#1176).
• prompt_logprob_token_ids, enable_trace_replay, return_sampling_mask, --watermark-config, and prefill context parallelism are refused at startup on Metal, and with pipeline parallelism under Model Runner V2, async scheduling is turned off with a warning (#1199).
• KV offloading refuses canonical_layout in kv_connector_extra_config at startup instead of writing Metal's own layout under a canonical label (#1256).
• Under the new Model Runner V2 default, a KV offloading disk tier written by v0.32.0 is not reused and fills again (#1199, #1238).
• XLM-R and BGE-M3 pooling now runs mlx-vlm's encoder, so reduced-precision embeddings are not elementwise identical to v0.32.0's (#1147).
• Docs: Mac recipes generated from YAML and fitted to a real 16 GB MacBook Pro, the GLM rows of the supported-models table, and NAX dispatch tracing in docs/profiling.md (#1146, #1213, #1208, #1257, #1226).

Known Boundaries

• The PyTorch MPS backend is experimental: it serves unquantized FP16/BF16 causal text decoders on one device, with LFM2 ShortConv and Qwen3.5 Gated DeltaNet (float32 state) as the only state layers and head sizes of 64, 96, 128, 256, or 512; MoE, MLA, pooling, diffusion, other hybrid models, and multimodal prefix LMs such as Gemma 4 vision fail at startup, as do LoRA, KV offloading, TurboQuant, parallelism, and sleep mode. LFM2-VL is the verified multimodal model (#858, #1241, #1243).
• MPS requests must be greedy, and a request that leaves temperature unset gets a 400; penalties, allowed_token_ids, bad_words, logit_bias, min_tokens, prompt logprobs, logprob_token_ids, and structured outputs are refused too. EAGLE3 is the only speculative method there, without parallel drafting, adaptive verification, or hybrid targets (#858, #1096, #1179).
• vLLM 0.33 refuses --enable-lora for models whose vLLM class lacks LoRA support, so StableLM, Flint, Ling, and PaddleOCR-VL now fail at startup with LoRA (vllm#61884, #1249).
• Native sampling gives different random draws than torch for the same seed, and seeded output is reproducible only for the same batch composition; sub-batches with penalties, sample logprobs, allowed_token_ids, bad_words, logit_bias, or a pending min_tokens fall back to torch, as do steps with structured output or speculative decoding (#1196, #1233, #1096).
• MoE expert streaming is MLX-only and fails at startup with pipeline parallelism, LoRA, or speculative decoding, and at load for GGUF, AWQ, multimodal checkpoints, and Mixtral-style or fused expert layouts (#1104).
• Sleep mode supports level 1 on the MLX backend only: a level-2 request gets a 500 and, like any refused sleep, leaves the engine paused until /wake_up; the MPS backend and speculative methods other than n-gram fail at startup with --enable-sleep-mode, and freeing the KV cache is refused for standalone MLA and speech-to-text models (#1102, #1166).
• N-gram drafting on hybrid GDN targets needs float32 recurrent state, the lazy GDN kernels, and an attention head size of at most 256, runs on the text-only path, and can be slower than plain decoding on diverse long contexts; other speculative methods on hybrid targets, and n-gram on Mamba-2, KDA, or ShortConv hybrids, fail at startup (#1072).
• DFlash and DSpark are experimental: LoRA, TurboQuant, and adaptive verification are refused, serving losslessness is not established, and they can be slower than target-only decoding; Speculators checkpoints must be Qwen3 drafts with the full target vocabulary and their own embeddings and output head (#928, #1092, #1199).
• Hybrid GDN prefix caching remains experimental: cache-on versus cache-off parity still diverges on a subset of divergent-suffix prompts, unchanged since v0.31.0.
• Nemotron-H and Granite 4.0 hybrid keep one private recurrent-state slot per request, so prefix caching is off for them (#1158).
• KV offloading supports only uniform full-attention (GQA, MHA, or MQA) models on the MLX backend, without draft-model, DFlash, or DSpark drafting, pooling, speech-to-text, or pipeline and data parallelism, and refuses canonical_layout; on a Mac already paging, an automatic pool shrinks to fit, while an explicit --kv-offloading-size can still fail startup (#998, #1063, #1256).
• LoRA adapters that use vLLM's modules_to_save for classifier or score heads are not supported on Metal yet, and --enable-tower-connector-lora adapts only the Qwen3-VL and Qwen3.5 vision encoders (#1022, #1107).
• vLLM's per-module LoRA scaling for rank_pattern and alpha_pattern (vllm#59801) and its per-replica reseed for data parallelism (vllm#59788) do not reach Metal yet (#1158).
• DiffusionGemma is experimental, text-only, and MLX-only; speculative decoding, pipeline parallelism, LoRA, and TurboQuant fail at startup, and top_k, top_p, logit_bias, min_tokens, prompt logprobs, logprobs=-1, and a diffusion_canvas_length smaller than the served canvas are refused (#994, #1096, #1158).
• vLLM's built-in /v1/systemone reads DiffusionGemma with a 16-token canvas, so on Metal it answers only when the served canvas_length is 16 and returns a 400 for larger canvases; with the 32-token canvas in docs/structured_decisions.md, use the upstream example server as shown there (#1077).
• Video input covers Qwen3-VL, Qwen3.5/3.6, and Gemma 4 E2B/E4B; Gemma 4 26B/31B, PaddleOCR-VL, LFM2-VL, and LFM3-VL are image-only, FP8 and GGUF Qwen3.5 checkpoints stay text-only, and --video-pruning-rate fails at startup (#1113, #1180, #1193).
• Mistral Small 3.1 serves text only, and an image request to it under VLLM_METAL_MULTIMODAL_MODE=multimodal-native stops the engine; Devstral Small 2 takes the Ministral 3 path but is unverified (#1131, #1156).
• The Gemma 4 sidecar handles images, plus audio and video on E2B/E4B, and stays off under speculative decoding, when Gemma4Processor cannot be built, or for a repo id that is not fully cached; run hf download <repo> first (#820, #1175, #1180).
• On M5, VLLM_METAL_MM_PREFIX_PATH=recompute can prefill Gemma 4 images faster, because the default kernel path takes image-block steps off NAX (#863).
• GGUF does not support LoRA, IQ1, IQ2, IQ3_XXS, or T-quants, MoE, SSM or hybrid models other than text-only Qwen3.5, vision input, MTP speculation, or sharded files, and IQ4_NL/IQ4_XS weights need about twice their file size in memory (#1066, #1202, #1228).
• prompt_logprobs steps project logits for every prompt row, which startup profiling does not reserve; a lower --gpu-memory-utilization leaves room (#897).

Contributors

@JWriter20, @JarJarFran, @Jarvis636431, @LxYuan0420, @MeKaustubh07, @RobbieJ, @StevenWang-CY, @TheVerdman, @WindChimeRan, @XXXXRT666, @YidaWeng, @abhijithneilabraham, @aeilot, @b-wu8, @ischencheng, @jimpablo, @junqiu-lei, @ricky-chaoju, @suntp, @syang-genai, @the-cross-art, @varad-more, @xuchen-w

Full changelog: v0.32.0...v0.33.0