github vllm-project/vllm-omni v0.28.0
on GitHub

3 hours ago

Highlights

This release features 397 merged changes from 126 contributors, including 44 new contributors.

vLLM-Omni v0.28.0 is led by three major advances: 1) a new Host Weight Runtime that turns distributed layerwise offload into a reusable, mmap-backed weight lifecycle, 2) scheduler-managed paged KV cache for diffusion, including HunyuanImage3 support on GPU and NPU, and 3) a broader full-duplex and realtime speech stack spanning MiniCPM-o 4.5, PersonaPlex, and NVIDIA Nemotron VoiceChat. The release rebases onto vLLM 0.28, significantly expands MiniMax H3 deployment and optimization options, and adds a wide range of speech, music, image, video, world-model, and VLA integrations.

Key Improvements

  • Introduced the Host Weight Runtime (HWR), a neutral runtime for building, validating, publishing, mmap-loading, leasing, and recovering reusable host-weight artifacts. Its first end-to-end diffusion integration supports final-layout BF16 artifacts for MiniMax H3, warm-start restoration, no-AllGather distributed layerwise offload, bounded pinned staging, and registered mmap-to-GPU transfer. (#6419, #6427, #6445, #6486, #6591)
  • Added scheduler-managed paged KV cache for diffusion, with native cache initialization, block allocation, worker RPC contracts, and a diffusion-native paged-KV backend. HunyuanImage3 DiT is the first complete model integration, supported on both GPU and NPU. (#5541, #5550, #6094, #6102, #6563)
  • Expanded full-duplex speech serving with a native vLLM port of PersonaPlex, native full-duplex NVIDIA Nemotron VoiceChat serving, configurable concurrent MiniCPM-o sessions, and substantial improvements to barge-in, playback state, handoffs, cancellation, and bounded long-running context. (#4771, #6089, #6021, #6170, #6529, #6626, #6772)
  • Advanced MiniMax H3 from initial support to a much broader deployment stack, adding modular pipelines, continuous batching, DLO on consumer GPUs, global FP8 and online INT8 paths, text-encoder disaggregation, TeaCache/Cache-DiT, Turbo and FlashGen LoRA variants, a fused four-step FastH3 adapter, and extensive GPU, NPU, ROCm, and MUSA optimizations. (#5720, #5810, #5764, #5910, #6573, #5885, #5840, #6550, #6666, #6714)

Core Architecture & Runtime

  • Rebased the project onto vLLM 0.28.0, including corresponding NPU integration updates. (#6606, #6674)
  • Added the Host Weight Runtime foundation with exact artifact identity, atomic local publication, corruption detection, crash recovery, mmap leases, capacity policy, and explicit preferred/required resolution modes. Post-load publication can warm later startups without mutating the model serving the current startup. (#6419, #6427)
  • Integrated final-layout HWR artifacts with no-AllGather distributed layerwise offload. Warm starts can skip ordinary DiT materialization, preserve a bounded two-slot staging path, or directly register mapped host-weight regions for asynchronous H2D transfer when supported. (#6445, #6486, #6591)
  • Added an opt-in event-driven orchestration loop for autoregressive stages, replacing millisecond polling with awaitable stage readers and a blocking final-output drain while preserving the legacy loop as the default. (#5221)
  • Added AR-stage pause/resume and sleep/wake control to AsyncOmni, including admission gating and acknowledged aborts for colocated serving, RLHF/weight-sync workflows, and idle memory reclamation. (#6084)
  • Continued the scheduler cleanup with shared AR/generation lifecycle contracts, explicit stage transport capabilities, refined diffusion admission waiting, per-replica fault isolation, and pipeline validation for missing terminal output stages. (#5461, #6149, #5843, #4583, #6291)
  • Consolidated configuration around VllmOmniConfig and reused native vLLM configuration objects, while removing the legacy stage-config loading path and its internal plumbing. (#5678, #6050, #5647, #5741, #6200)

Model Support

  • Added PersonaPlex, a Moshi-based full-duplex speech-to-speech model, with a native vLLM port and duplex serving. (#4771)
  • Added NVIDIA-NemotronLabs/VoiceChat-11B for offline speech-to-speech inference and native full-duplex serving. (#5842, #6089)
  • Added new TTS integrations for IndexTTS 2.5, dots.tts continuous-AR 48 kHz synthesis, and Gepard 1.0 native-AR FSQ/NanoCodec offline inference. (#5957, #4765, #5666)
  • Added MiniMax Music 3 text-to-music generation. (#6186)
  • Added HiDream-O1-Image, SANA-Video 2B T2V/I2V, and Stage-1 support for the SANA world model. (#5194, #5508, #4061)
  • Added LongCat-Video-Avatar-1.5 audio/image-to-video and audio/text-to-video support, and expanded LingBot Video with T2I and TI2V generation modes. (#4099, #5311)
  • Added LTX-2.5, including its diffusion pipeline and VAE decoder, plus a standard two-stage LTX execution path. (#6070, #6189, #5500)
  • Added the pi0 vision-language-action model, extending vLLM-Omni beyond media generation into VLA inference. (#4222)
  • Added a Qwen3-Omni thinker-only pipeline for Instruct serving and a Ming Flash Omni TTS adapter. (#6284, #5746)

Audio, Speech & Realtime Serving

  • Matured the MiniCPM-o 4.5 duplex runtime with configurable concurrent sessions, native deploy configurations, resilient Stage-1 handoffs, barge-in isolation, playback checkpoints, camera/video unit binding, bounded auto-response, and sliding context recomputation. (#6021, #6619, #6529, #6170, #6821, #6404, #6630, #6626)
  • Accelerated MiniCPM-o audio generation with TensorRT Code2Wav execution, CUDA Graphs for HiFT and CFM DiT, NPU Graph replay, batched codec sampling, and lower concurrent first-packet latency. (#5638, #5869, #6082, #5604, #5792, #6767)
  • Improved TTS streaming through cached incremental Qwen3-TTS decoding, fused code-predictor projections, adaptive buffer-feedback chunk ramping, asynchronous MOSS-TTS scheduling, and hot-path optimizations for Voxtral TTS, Step-Audio2, GLM-TTS, and OmniVoice. (#5202, #5791, #6001, #6241, #5175, #5067, #5068, #5174)
  • Added optional TTS timestamps through forced-aligner pooling and speech token-usage response headers. (#4795, #4499)
  • Moved TTS detection, sampling overrides, and model capability metadata into adapters, reducing model-name special cases in shared serving code. (#5682, #5272, #6138)

Diffusion, Image & Video Generation

  • Added a complete scheduler-managed paged-KV architecture for diffusion, covering cache initialization, block allocation, worker contracts, RPC plumbing, and a native diffusion backend. (#5541, #5550, #6094, #6102)
  • Enabled paged KV cache for HunyuanImage3 DiT on both GPU and NPU. (#6563)
  • Added UniProc diffusion execution for single-GPU deployments and native SymmMem Fast Ulysses transport for distributed sequence parallelism. (#6308, #6340)
  • Expanded MiniMax H3 execution with modular pipelines, scheduler-level continuous batching, packed and sparse attention paths, faster MP4/frame conversion, optimized output transfer, and dedicated VAE decoder operators. (#5720, #5810, #5891, #6518, #6499, #6824, #6607)
  • Added request-level batching for Wan2.2, FastVideo VSA attention, distilled diffusion LoRA support, and FLUX.2-klein Host Weight Runtime contracts. (#5676, #4820, #2783, #6651)
  • Improved distributed diffusion kernels with fused Q/K RMSNorm plus RoPE, a mask-free TensorRT-LLM packed-padding path, quantized FlashInfer attention for Blackwell, and a device-correct dedicated VAE communication group. (#5990, #6542, #5344, #6401)

Quantization & Memory Efficiency

  • Added Qwen2.5-Omni thinker-only ModelOpt NVFP4 W4A4 checkpoint support and AutoRound MXFP4 offline quantized-model support. (#5073, #5544)
  • Added offline SVDQuant W4A4 support for diffusion models. (#6162)
  • Extended distributed layerwise offload with MiniMax H3 global FP8, generic online FP8 over DLO AllGather, and online INT8 over DLO AllGather. (#5910, #6279, #6573)
  • Added MiniMax H3 online FP8, NPU RainFusion plus online INT8, and Qwen3-Omni ModelOpt FP8 inference on MUSA. (#5737, #5706, #5671)

Serving, Frontend & API Behavior

  • Added batched Chat Completions and speech token-usage headers. (#5317, #4499)
  • Added ComfyUI reference-to-video integration, with MiniMax H3 as the initial example. (#5756)
  • Added first-class diffusion metrics and returned them from image-edit serving as well as generation paths. (#4755, #5999)
  • Improved model-tag synchronization, TTS validation for models without uploaded speakers, object-storage model resolution, media redirect policy, request overflow handling, and online profiler stage selection. (#3805, #5878, #5036, #6122, #6598, #6609)

Platforms & Hardware Coverage

  • Expanded Ascend NPU support for MiniMax H3 with RainFusion, online INT8, packed mask-free attention, distilled four-step schedules, sparse reference/target attention, fused encoder/DiT kernels, and paged KV cache for HunyuanImage3. (#5706, #5891, #5991, #6518, #6040, #6410, #6563)
  • Completed the vLLM-Omni platform interfaces for Moore Threads MUSA, and expanded MiniMax H3 and Qwen-Image kernel compatibility alongside Qwen3-Omni ModelOpt FP8. (#6058, #5881, #6110, #5671)
  • Added verified ROCm recipes for MiniMax H3 and Cosmos3 Nano, moved ROCm CI to MI300X, and refreshed AMD coverage for the v0.28 line. (#5723, #5634, #6207, #5886, #6830)
  • Made autoregressive asynchronous output and image D2H synchronization more device-agnostic on XPU, and moved XPU CI onto the vLLM base image. (#5569, #5571, #6727)

Breaking Changes

  • The legacy stage-configuration path has been removed. --stage-configs-path, the internal stage_configs_path plumbing, and the legacy stage_args YAML loader are no longer supported. Deployments should use registered pipelines and deploy configurations through the unified vllm serve --omni flow. (#5647, #5741, #6200, #6221)
  • OmniRequestOutput now directly inherits vLLM RequestOutput. Code using the removed nested request_output accessor must read the inherited fields directly. (#5146, #6172)
  • Support was removed for DreamID-Omni, MagiHuman, SoulX-Singer, and AudioX. MammothModa2 was temporarily removed in the same cleanup series but restored before v0.28.0. (#6357, #6362, #6353, #6694)

Note

  • The event-driven orchestrator is opt-in through VLLM_OMNI_EVENT_DRIVEN_ORCH=1; the legacy polling loop remains the default in v0.28.0. (#5221)
  • Host Weight Runtime is a general foundation, but the first concrete final-layout producer and DLO consumer in this release targets MiniMax H3 BF16 no-AllGather deployments. Registered mmap H2D is used only when supported and otherwise falls back to bounded pinned staging. (#6445, #6486, #6591)
  • MiniMax H3 continuous batching provides scheduler-level control and request co-batching, but its dense DiT compute scales roughly with the number of packed requests; it should not be assumed to improve throughput for every workload. (#5810)
  • Full-duplex serving continues to evolve. Users should validate session concurrency, barge-in policy, long-duration context behavior, and client/API compatibility for their target model and deployment. (#4771, #6021, #6089)

What's Changed

New Contributors

Full Changelog: v0.26.0...v0.28.0

Check out latest releases or
releases around vllm-project/vllm-omni v0.28.0

Don't miss a new vllm-omni release

NewReleases is sending notifications on new releases.

Get notifications