github vllm-project/vllm-omni v0.24.0
on GitHub

4 hours ago

Highlights

This release features 285 commits from 112 contributors, including 40 new contributors.

vLLM-Omni v0.24.0 brings a large set of improvements across TTS, speech, diffusion, image generation, video generation, robot-policy serving, quantization, runtime architecture, request batching, cache behavior, CI, and hardware backend support.

This release expands production-ready multimodal serving coverage with new and improved models such as Higgs Audio V3, Qwen3-TTS, Ming-omni-tts, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, SDXL, Cosmos3, GR00T-N1.7, HunyuanImage3, DreamZero, Wan2.2 / Wan S2V, SenseNova-U1, JoyAI-VL-Interaction, and more. It also includes the vLLM 0.23 and 0.24 rebases, significant Omni stage runtime refactoring, diffusion request-level batching, async output materialization, improved streaming paths, and broad CUDA / ROCm / XPU / NPU coverage.

Key Improvements

  • Aligned with the vLLM 0.24 release line, including the vLLM 0.23.0 and v0.24.0 rebases, release-branch updates, and compatibility fixes across model loading, graph behavior, quantization, and serving paths. (#4286, #4709)
  • Expanded TTS and audio model support, adding or improving Higgs Audio V3, Qwen3-TTS, CosyVoice, Fish Speech, VoxCPM2, Voxtral, Ming-omni-tts, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, soulx-singer, Aura, and Stable-Audio-Open. (#4169, #4204, #3838, #4664, #4341, #464, #3862, #4257, #3664)
  • Made speech serving more production-ready, with SSE audio streaming, word-level timestamps, Qwen3-TTS correctness fixes, token accounting, WebSocket input handling, streaming-tail fixes, cross-request audio-corruption fixes, and better voice-clone reference-audio behavior. (#4034, #4490, #4679, #4673, #4731, #4706, #4415, #4429)
  • Strengthened diffusion, image, and video generation, including major improvements for Cosmos3, HunyuanImage3, DreamZero, BAGEL, Wan2.2, Wan S2V, LTX-2.3, SDXL, Qwen-Image, Qwen-Image-Edit, MammothModa2, GLM-Image, and SenseNova-U1. (#4331, #4379, #4514, #4627, #4041, #4213, #4098, #4667, #4739, #4474, #4293, #4299, #3956, #4188)
  • Improved core Omni runtime and serving infrastructure, including the Omni stage runtime and distributed replica control plane, orchestrator output-path optimization, async output materialization, diffusion request-level batching, HF-config-based pipeline resolution, structured config classes, and configurable video storage backends. (#3855, #4527, #4476, #4079, #3760, #4425, #2531)
  • Expanded quantization and memory-efficiency coverage, including ModelOpt FP8 for video-generation models, Qwen3-Omni NVFP4 W4A4 on Blackwell, Qwen-Image AutoRound W4A16, ROCm MXFP4 online quantization, HSDP + FP8 compatibility, CacheDiT cleanup, prefix-cache fixes, and improved memory metrics. (#3305, #4025, #4528, #3588, #4104, #4494, #2527, #4449, #4106, #4477)
  • Improved hardware backend support, with fixes and features for CUDA, ROCm, XPU, and Ascend NPU, including NPU graph behavior, custom-op registration, diffusion-worker initialization, XPU CI fixes, platform-agnostic device selection, and ROCm test stability. (#4454, #4712, #4386, #4674, #4451, #4496, #4380, #4574, #4821, #4848)

Core Architecture & Runtime

  • Refactored the Omni stage runtime and distributed replica control plane, providing a stronger foundation for multistage omni serving and distributed stage coordination. (#3855)
  • Reduced orchestrator bottlenecks by separating inter-stage outputs from client-facing outputs. (#4527)
  • Added async Omni output materialization so generated multimodal artifacts can be surfaced through more flexible serving flows. (#4476)
  • Added request-level batching for diffusion pipelines, improving throughput and serving efficiency for compatible diffusion workloads. (#4079)
  • Introduced HF-config-based pipeline resolution and eager pipeline registry behavior, improving automatic pipeline selection and startup behavior. (#3760)
  • Added structured VllmOmniConfig classes and restored deploy-config-based pipeline resolution in StageConfigFactory. (#4425, #4729)
  • Migrated existing pipelines to SupportsComponentDiscovery / ModuleDiscovery, improving offload discovery and nested HSDP DiT sharding support. (#3076, #3456)
  • Extracted diffusion output formatting boundaries and removed dead custom stage input hooks to simplify stage-processing internals. (#4407, #4531)
  • Fixed multi-replica stage identity and cleaned up redundant deploy configuration. (#4410)
  • Aligned Omni request input signatures with vLLM and fixed multimodal generation-stage output construction. (#4568, #4579)

Model Support

  • Added support for bosonai/higgs-audio-v3-tts-4b and optimized Higgs Audio V3 serving paths. (#4169, #4204)
  • Added or improved TTS model families including Ming-omni-tts dense 0.5B, Ming-omni-tts MoE 16.8B-A3B, IndexTTS2, MOSS-TTS-local-v1.5, Step-Audio2, soulx-singer, Aura, and fine-tuned Qwen3-TTS language support. (#2906, #4341, #3838, #4664, #4804, #464, #3862, #4257, #4210)
  • Added or enabled diffusion and image/video model coverage for SDXL, GR00T-N1.7, Cosmos3, HunyuanImage3, DreamZero, Wan2.2 / Wan S2V, LTX-2.3 I2V, JoyAI-VL-Interaction, MammothModa2, and SenseNova-U1. (#4331, #3798, #4266, #4041, #4154, #4276, #4458, #4739, #4575, #4299, #4188)
  • Added DROID policy server support for Cosmos3 OpenPI and GR00T-N1.7 OpenPI serving, expanding robot-policy serving coverage. (#4282, #3798)
  • Migrated or refreshed examples for Cosmos3, Helios, Magi human, VACE, MammothModa2-Preview, AudioX, Qwen Image, Stable-Diffusion-3.5, Voxtral TTS, and Qwen3-TTS. (#4581, #4569, #4572, #4648, #4691, #4607, #4236, #4052, #4051, #4026)

Audio, Speech & TTS Serving

  • Improved Qwen3-TTS serving correctness across custom voice mode, WebSocket input handling, no-async-chunk payload splitting, multi-token replay on resume, seeded residual MTP sampling, token accounting, Code2Wav CUDA Graph output length, and full-payload completion behavior. (#4650, #4731, #4870, #4559, #4869, #4889, #4673, #4525, #4465)
  • Added streaming TTS word-level timestamps via forced alignment. (#4034)
  • Added SSE stream_format support for audio speech streaming and made speech streaming default to SSE. (#4490, #4679)
  • Fixed Code2Wav async-chunk streaming tail drop, reducing boundary clicks and time-compression artifacts. (#4706)
  • Improved CosyVoice and CosyVoice3 paths, including TensorRT optimization, reference-text instruction templating, and prompt handling fixes. (#4168, #4756)
  • Fixed Fish Speech serving issues including Gradio default voice handling, prefix-cache collision, and KV-cache unpacking after vLLM compatibility changes. (#3941, #4008, #4428)
  • Improved MOSS-TTS stability and performance with eager initialization, CUDA Graph support, cross-request audio-corruption fixes, talker micro-optimizations, loading shims, streaming support, and WER accuracy fixes. (#3230, #4157, #4415, #4230, #4398, #4804, #4859)
  • Improved VoxCPM2 quality, hardware adaptation, KV-cache settings, unified decode graph optimization, and test stability. (#4319, #4310, #4374, #4279, #4606, #4878)
  • Updated audio and TTS recipes, baselines, and docs for Qwen3-TTS, VoxCPM, Voxtral, Stable-Audio-Open, AudioX, and Ming-omni-tts. (#4026, #4521, #4051, #4774, #3664, #4607, #4324)

Diffusion, Image & Video Generation

  • Improved Cosmos3 generation, transfer, quality, distributed behavior, testing, and documentation, including v2v support, transfer paths, regression fixes, I2V conditioning optimization, skipped unused conditioning latents, regional compile, sequence-parallel sound latent padding, OpenPI serving, and L2 serving tests. (#4266, #4379, #4514, #4627, #4467, #4614, #4485, #4678, #4282, #4535)
  • Improved HunyuanImage3 correctness and performance with DiT grouped step batching, more resolution support, AR RGB conversion alignment, CoT truncation fixes, streaming CoT display, sync removal, timestep scalar-sync avoidance, KV prefetch, MoE fixes, and bridge request batching fixes. (#4041, #4004, #4502, #4260, #4148, #4401, #4363, #4448, #4893, #4849)
  • Advanced DreamZero serving with TP and cross-attention cache fixes, CUDA Graph support, torch.compile, DiT caching, and engine-level KV-cache management groundwork. (#4154, #4213, #4534)
  • Improved BAGEL generation with denoising schedule fixes, batched CFG forward reimplementation, CFG-parallel mixin reuse, memory metrics, and SP denoise fixes. (#4509, #4098, #4768, #4477, #4328)
  • Improved Wan2.2 / Wan S2V with graph-break fixes, CacheDiT + Ulysses fixes, S2V API support, SP and HSDP support, Wan VAE spatially sharded decode, and VACE cache / lifecycle fixes. (#4053, #3927, #3394, #4276, #4458, #4620, #4667)
  • Added LTX-2.3 I2V support, VAE decode parallelism, resident auxiliary modules, and tensor-parallel gated-attention fixes. (#4739, #4277, #4144, #4439)
  • Improved image-generation serving with file-style image response_format, default image-generation limits, Qwen-Image RoPE fixes, Qwen-Image-Edit performance fixes, Qwen-Image-Layered compatibility, and Ming-flash-omni image-generation batching fixes. (#1673, #3381, #4474, #4293, #4777, #4837)
  • Improved SenseNova-U1 with LoRA support, CFG parallelism, fused RMSNorm + 3D RoPE kernels, TeaCache, and pipeline fixes. (#3971, #4188, #4669, #4164, #4326)

Quantization, Cache & Memory Efficiency

  • Added ModelOpt FP8 support for Wan2.2 and HunyuanVideo-1.5 video generation, and propagated quantization configuration into HunyuanVideo-1.5 I2V transformer layers. (#3305, #4245)
  • Added Qwen3-Omni NVFP4 W4A4 serving on Blackwell and fixed Thinker lm_head prefix handling so NVFP4 exclude lists are honored. (#4025, #4528)
  • Added Qwen-Image AutoRound W4A16 support and fixed Qwen2.5-Omni AutoRound loading. (#3588, #4781)
  • Added ROCm support for MXFP4 W4A4 online quantization. (#4104)
  • Improved HSDP + FP8 online quantization compatibility and gated quack FP8 GEMM to datacenter Blackwell GPUs. (#4494, #4817)
  • Simplified CacheDiT integration and improved cache behavior across DFlash, Qwen3-TTS, Fish Speech, DreamZero, Wan2.2, and OmniTensorPrefix. (#2527, #4449, #3689, #4317, #4008, #4534, #3927, #4106)
  • Added memory metrics fixes and reporting improvements for BAGEL, Qwen-Image, non-text requests, and audio RTF metrics. (#4477, #4517, #4482, #4794)

Serving, Frontend & API Behavior

  • Added configurable video storage backends with TTL. (#2531)
  • Returned proper HTTP status codes for audio voice endpoints and refactored guardrail error handling with 400 error codes. (#3969, #4297)
  • Added async output materialization and streaming diffusion video generation output. (#4476, #3737)
  • Added support for image-generation responses with response_format: file. (#1673)
  • Fixed request batching regressions and output construction issues across diffusion and multimodal serving paths. (#4079, #4579, #4837, #4910)
  • Added support for pipeline resolution based on HF config and eager pipeline registry initialization. (#3760)
  • Replaced hardcoded CUDA device selection with platform-agnostic APIs. (#4496)
  • Reverted the MediaConnector image/video URL-fetching change after follow-up validation needs. (#4751)

Platforms, Distributed Execution & Hardware Coverage

  • Improved Ascend NPU support with VoxCPM2 support, Qwen3-TTS 310P support, model-runner fixes, OmniMRotaryEmbedding support, diffusion-worker Ascend config initialization, custom-op registration, graph behavior restoration, HunyuanImage3 MoE EP+SP mapping, and v0.24.0 upgrade work. (#4310, #4283, #4454, #3609, #4386, #4712, #4674, #4344, #4848)
  • Improved ROCm stability with CI grouping, diffusion-test fixes, Voxtral fixes, AMD CI updates after the v0.24.0 rebase, and MXFP4 online quantization support. (#4208, #4574, #4380, #4821, #4104)
  • Improved XPU coverage with sage-attn backend support, Cosmos3 CUDA-hardcode removal, DreamZero any-hardware support, docker-build improvements, marker fixes, SDXL enablement, and v0.24 CI updates. (#3785, #4360, #4399, #4402, #4451, #4331, #4805)
  • Improved distributed diffusion paths with HSDP/SP fixes and features across Cosmos3, Wan S2V, Wan VAE, nested DiT sharding, and CFG-parallel serving. (#4452, #4678, #4276, #4458, #4620, #3456, #4900)

CI, Testing, Benchmarks & Documentation

  • Improved ready/merge, rebase, and nightly pipelines with diff-aware L2/L3 upload, full E2E rebase coverage, nightly L2/L3 E2E groups, split diffusion X2V tests, Cosmos3 L2 tests, and reduced ready-CI duration. (#4291, #4313, #4365, #4478, #4532, #4693, #4734, #4744, #4535, #4354)
  • Added reliability and stability tests for HunyuanImage3 and VoxCPM2, plus targeted skip/unskip changes for unstable or OOM-prone test cases. (#4600, #4311, #4701, #4758, #4759, #4391)
  • Updated performance baselines for VoxCPM, Qwen3-TTS, Cosmos3, and perf JSONs. (#4289, #4521)
  • Added and refined Claude / agent skills for precheck, quantization, vLLM-Omni test generation, and diffusion-model contribution workflows. (#4216, #4252, #4434, #4697, #4110)
  • Updated docs and recipes for supported models, Qwen Image, Qwen3-TTS, Voxtral TTS, Stable-Diffusion-3.5, Stable-Audio-Open, Cosmos3 FP8, CUDA custom Docker builds, failure modes, and release metadata. (#4233, #4300, #4236, #4026, #4051, #4774, #4052, #3664, #4393, #4584, #1386, #3926)

What's Changed

New Contributors

Full Changelog: v0.22.0...v0.24.0

Check out latest releases or
releases around vllm-project/vllm-omni v0.24.0

Don't miss a new vllm-omni release

NewReleases is sending notifications on new releases.

Get notifications