What's Changed
🚀 Features
- FP8 kv cache quantization by @CUHKSZzxy in #4563
- Support Qwen3 Omni by @CUHKSZzxy in #4411
- support qwen3.5(vit) inference in turbomind backend by @irexyc in #4602
- Add OpenAI Responses-compatible endpoint by @CUHKSZzxy in #4582
- Add /get_ppl endpoint by @irexyc in #4679
💥 Improvements
- Update turbomind modeling infrastructure by @lzhangzz in #4557
- refactor(turbomind): consolidate CUDA error handling and add manual stacktracing by @lzhangzz in #4565
- Add Qwen3.5 Moe lite awq by @43758726 in #4561
- [Improve]: Drain queues when sleep engine by @RunningLeon in #4577
- Extend chat completions by introducing token-in/out and returning routed experts by @lvhan028 in #4593
- Follow openai's spec to add "AllowedToolChoice" and report 400 when parsing request failed by @lvhan028 in #4585
- Improve health endpoint by @lvhan028 in #4615
- Remove state init by @grimoire in #4604
- Include spec stats in metrics by @RunningLeon in #4625
- Add raw chat completion logprob output by @lvhan028 in #4637
- fix(pytorch): offload guided decoding CPU ops to thread pool to prevent event loop blocking by @windreamer in #4590
- update gated delta rule state layout by @grimoire in #4636
- Improve kernel dispatch for dp>1 by @RunningLeon in #4653
- Extend v1/messages by introducing token-in/out and returning routed experts by @lvhan028 in #4642
- Fuse gdr preprocess by @grimoire in #4656
- refactor: simplify multimodal preprocessing expansion by @CUHKSZzxy in #4663
- feat: configure cudagraph capture batch sizes by @CUHKSZzxy in #4573
- Refactor prefix caching for pytorch engine by @grimoire in #4618
- Pading one more block for fa3 prefill by @RunningLeon in #4674
- Add usage.prompt_tokens_details.cached_tokens for prefix caching by @lvhan028 in #4670
- Optimize XML tool parsers with incremental streaming and fast-path buffering by @lvhan028 in #4664
🐞 Bug fixes
- fix the anthropic adapter by @lvhan028 in #4578
- Fix Structured Output for GPT-OSS Models by @windreamer in #4386
- Allow W8A8Linear to accept dtype during initialization instead of hard code by @43758726 in #4586
- fix: compact split multimodal tensors by @CUHKSZzxy in #4583
- Fix legacy VLM preprocessors for normalized image data by @CUHKSZzxy in #4584
- fix dockerfile which missing common.txt by @lvhan028 in #4608
- fix: enable FA3 for SM80+ GPUs and fix CUDA version comparison by @windreamer in #4591
- flatten_kv_cache zero padding by @grimoire in #4613
- align streaming usage chunks with OpenAI spec by @lvhan028 in #4616
- fix(vl): reduce multimodal feature memory use by @CUHKSZzxy in #4603
- fix memleak when input contain large image data by @grimoire in #4610
- fix(turbomind): map Intern-S1 HF checkpoint keys by @lvhan028 in #4617
- fix(serve): emit all stream_chunk deltas to fix concurrent tool-call streaming by @lvhan028 in #4622
- fix cp inference by @irexyc in #4619
- refactor(serve): avoid per-request tokenizer work in parsers by @lvhan028 in #4633
- Bring MixtralForCausalLM back to Turbomind by @43758726 in #4623
- fix model loading on windows by @irexyc in #4626
- Fix mtp cudagraph when no warmup in RL by @RunningLeon in #4641
- fix: remove hard CUDA_PATH assert on Windows, search DLL paths from multiple sources by @windreamer in #4628
- Fix unit test by removing latest-transformers-unsupported models by @lvhan028 in #4649
- Fix qwen3.5 mtp by @RunningLeon in #4652
- fix gdr kernel for tilelang>=0.1.9 by @grimoire in #4660
- [Fix]: Revert the reuse of cudagraph buffer for mtp by @RunningLeon in #4661
- Fix client-disconnect session leaks in PyTorch MP engine by @grimoire in #4655
- fix cancel stopped seq by @RunningLeon in #4654
- feat: support num_experts_per_tok=10 in turbomind backend by @irexyc in #4665
- fix batched seqs with different stop words by @RunningLeon in #4671
- Move warmup inside wakeup by @RunningLeon in #4667
- Fix dequant_mixed by @irexyc in #4657
- Improve engine health monitoring by @lvhan028 in #4645
- fix qwen3.5 27b gdr preprocess by @grimoire in #4676
- Fix dequant mixed for qwen3.5 quantized model made by vllm/llm-compressor by @irexyc in #4675
- [Bugfix] Fix double-counted max_q_seqlen in decode delta kv_seqlens by @waynehacking8 in #4685
- Fix scheduler for ssm by @RunningLeon in #4691
- fix(serve): avoid parallel tool-call argument leakage in XML parsers by @lvhan028 in #4692
- fix prefix caching by @grimoire in #4700
📚 Documentations
- docs: update multimodal model support docs by @CUHKSZzxy in #4643
🌐 Other
- chore: gate request logs behind request level by @CUHKSZzxy in #4581
- miss rdkit for intern-s models by @lvhan028 in #4587
- extract common deps into requirements/common.txt by @lvhan028 in #4595
- Remove staled cli arg in vlmevalkit docs by @CUHKSZzxy in #4598
- log reponse for debugging by @lvhan028 in #4592
- cancel in-progress runs when PR is updated or merged by @lvhan028 in #4609
- TEST: update qwen3.5 397b test by @littlegy in #4607
- TEST: update video test by @littlegy in #4606
- Validate final chat response structure by @lvhan028 in #4621
- Support dp for qwen35 mtp by @RunningLeon in #4611
- [ci] refactor testcoverage config by @zhulinJulia24 in #4630
- TEST: update ascend and mtp test config by @littlegy in #4659
- TEST: update FP8 processing logic and remove duplicate MTP tests by @littlegy in #4668
- freeze tilelang version by @grimoire in #4669
- fix windows ci by @irexyc in #4672
- [ci] add mtp test config in pr_test by @zhulinJulia24 in #4651
- support disaggregated weight update by @irexyc in #4638
- bump version to v0.14.0 by @lvhan028 in #4689
New Contributors
- @waynehacking8 made their first contribution in #4685
Full Changelog: v0.13.0...v0.14.0