What's new in 3.4.0 (2026-09-11)
These are the changes in inference v3.4.0.
New features
- feat(vllm): support native multiprocessing executor routing by @m199369309 in #5447
- feat: Add Gemma-4 Transformers backend with batching support by @maoyuehui in #5430
- feat(model): add download-only flow by @Minamiyama in #5463
- feat(model): support dots ocr model by @llyycchhee in #5468
- feat(model): add engine registration hooks by @amumu96 in #5465
- feat(cache): add resumable cache management by @Minamiyama in #5471
- feat(cache): open cached models for launch by @Minamiyama in #5476
- feat(ui): select sole available model engine by @Minamiyama in #5480
- FEAT: support oci:// model URIs via llmman serve by @ericcurtin in #5457
- feat(model): support MonkeyOCR by @llyycchhee in #5475
- feat(model): Add Fish Audio S1-mini and S2-Pro support by @Minamiyama in #5490
- feat: support multi-worker replica scaling by @leslie2046 in #5491
- feat(models): add embedding and rerank multimodal abilities by @Minamiyama in #5496
- feat(ui): prefer system download source by @Minamiyama in #5498
- feat: support prefill/decode separation by @amumu96 in #5505
- feat(audio): add multilingual TTS language controls by @Minamiyama in #5500
- feat(i18n): add zh-TW (Taiwan Traditional Chinese) locale by @leonoxo in #5509
- feat(llm): add MiniCPM5-2B support by @Minamiyama in #5506
- feat(webui): reuse launch history across models by @m199369309 in #5517
Enhancements
- ENH(auth): Add granular system settings permissions by @Minamiyama in #5445
- ENH: support JoyAI image editing models by @Minamiyama in #5458
- ENH: Add TTS streaming playback to Web UI by @Minamiyama in #5487
- ENH: Complete SD WebUI image workflow compatibility by @amumu96 in #5508
- ENH: update frontend language order and product links by @qinxuye in #5514
- BLD: support Python 3.14 and require xoscar 0.10.0 by @qinxuye in #5515
Bug fixes
- fix(venv): require xoscar native find-links support by @m199369309 in #5448
- fix(embedding): align Jina flash-attn version by @m199369309 in #5452
- fix(venv): protect virtual environments in use by @m199369309 in #5449
- fix(llm): emit streaming tool call metadata once by @m199369309 in #5446
- FIX: Restore supported vLLM and SGLang model engines by @qinxuye in #5461
- FIX: apply download worker environment to snapshots by @amumu96 in #5460
- fix(llm): gate Hy-MT2 remote code execution by @m199369309 in #5474
- fix(transformers): support legacy cache conversion by @m199369309 in #5477
- fix(scheduler): isolate batch inference failures by @m199369309 in #5478
- fix(transformers): handle multi-device cache reduction by @m199369309 in #5479
- fix(security): report bans lifted, not records dropped, from bulk unban by @NgoQuocViet2001 in #5483
- fix(audio): correct F5-TTS logging format by @David-Wu1119 in #5485
- BUG: Merge engine capabilities across workers by @oodadoudou in #5486
- FIX: Restore audio streaming with TorchCodec by @Minamiyama in #5488
- fix(model): make register_builtin_model idempotent across refreshes by @AmirF194 in #5481
- FIX: Improve model experience actions and latency feedback by @leslie2046 in #5492
- fix(ui): contain embedding result scrolling by @Minamiyama in #5495
- fix: handle runtime and device options in flexible Transformers launcher by @leslie2046 in #5497
- fix(rerank): launch GGUF rerank models without explicit engine/quantization by @7487 in #5503
- fix(chatglm): emit a usage-only final streaming chunk by @m199369309 in #5510
- fix(transformers): propagate batched generate request IDs by @m199369309 in #5513
- fix(transformers): handle batched logits when selecting next token by @m199369309 in #5511
- fix(transformers): enforce token limits across batched decode steps by @m199369309 in #5512
- fix(worker): decouple heartbeat and full status reporting by @m199369309 in #5518
- fix(webui): restore server-backed launch history by @m199369309 in #5516
- fix(core): preserve model GPU memory across transient failures by @m199369309 in #5519
- fix(worker): refresh dynamic model subpool PID ownership by @m199369309 in #5520
- FIX: avoid implicit event loop in async client cleanup by @qinxuye in #5521
Documentation
- doc: update README model highlights for v3.3.0 by @qinxuye in #5454
- DOC: preserve ACE-Step1.5 details outside generated docs by @qinxuye in #5455
- DOC: fix model type card layout by @qinxuye in #5489
Others
- test(pypiserver): align Jina flash-attn manifest pin by @m199369309 in #5456
- docs: add v3.3.0 to release notes index and locale catalogs by @martinma51 in #5462
- chore: bump actions/checkout to v7 by @qinrui777 in #5469
New Contributors
- @NgoQuocViet2001 made their first contribution in #5483
- @ericcurtin made their first contribution in #5457
- @David-Wu1119 made their first contribution in #5485
- @oodadoudou made their first contribution in #5486
- @7487 made their first contribution in #5503
- @leonoxo made their first contribution in #5509
Full Changelog: v3.3.0...v3.4.0