What's new in 4.1.0 (2026-10-25)

These are the changes in inference v4.1.0.

Highlights

TRT-LLM is now an inference backend. Run supported models through Xinference's unified serving API with TensorRT-LLM, including chat, generation and streaming.
Xavier now supports TRT-LLM. Extend prefix KV cache sharing to TRT-LLM and use native or Xavier prefill/decode separation. Compatible models can hand off between TRT-LLM, vLLM, SGLang and MLX, including NVIDIA–Apple silicon deployments.
LMCache disk storage for Xavier. Extend prefix KV cache storage beyond memory with an optional LMCache backend for vLLM, SGLang and TRT-LLM.
Live upgrades that preserve in-flight inference (experimental). Upgrade supported model engines and replace Worker, API and Supervisor processes on POSIX, with Web UI controls for local and distributed deployments. Existing requests and streams finish before the old serving processes retire.
Consistent context-length controls. Configure model context limits across inference engines and adjust them through the Web UI slider, with validation that follows the effective model configuration.
Expanded model support. Add DeepSeek-V4.1-Flash, GLM-5.3 and Qwen3.8-Flash-Next, alongside new multimodal, image, speech and OCR models such as MiniCPM-V-4.7, FLUX 3, Ming-3.0 TTS and Xiaomi-OCR-0.

Security

Xinference 4.1.0 is the first patched release for the following published advisories. Users of affected releases should upgrade to 4.1.0 or later.

Critical — CosyVoice model loading: require explicit operator trust before loading custom sources or explicit local paths, preventing untrusted HyperPyYAML configuration from reaching the loader without that opt-in. GHSA-2gq8-2q98-6w29 · #5765.
High — Initial administrator setup: require an operator setup token before creating the first administrator, preventing unauthenticated callers from claiming an uninitialized server. Affects versions >= 3.0.0, < 4.1.0. GHSA-v43j-w7xm-2vwh · #5703.
Medium — Auto-registration paths and Flexible launchers: confine model-configuration reads to approved roots and require an operator allowlist for additional launchers. The reported impacts are partial configuration disclosure/filesystem probing and arbitrary module import. GHSA-3jw7-r8gq-5ggj · #5766.
Medium — Chat-completion error disclosure: return client-safe HTTP/SSE errors without exposing raw worker addresses, process identifiers or other unexpected backend diagnostics. GHSA-h97g-6mc6-66hm · #5767.

Thanks to Paul White (Su1ph3r), Cairn Security for reporting the initial-administrator issue and verifying the API-level remediation.

Upgrade notes: For fresh instances, use the setup token in the API server startup log to create the initial administrator. Custom CosyVoice sources require worker-side XINFERENCE_TRUST_REMOTE_CODE=1 and must still be trusted. Additional registration roots and Flexible launchers require XINFERENCE_MODEL_REGISTER_ROOTS and XINFERENCE_FLEXIBLE_LAUNCHERS, respectively; see the advisories for scope and configuration.

Additional security improvements in this release:

Validate shared-store API keys against current state. #5770.
Integrate Xoscar actor transport authentication. This is opt-in: configure the same nonempty XOSCAR_AUTH_KEY on every actor peer; upgrading alone does not enable it. #5696.
Apply the remote-code trust policy to MOSS pipelines and update Next.js/frontend dependencies, PyArrow and follow-redirects for security advisories. #5769, #5768, #5803.

New features

feat(model): support DeepSeek-V4.1-Flash model by @llyycchhee in #5661
feat(image): add Qwen-Image-2.1-Turbo support by @Minamiyama in #5665
feat(api): add optional fastapi-guard security middleware by @rennf93 in #5664
feat(model): support glm-5.3 model by @llyycchhee in #5666
feat: estimate LLM launch memory across inference engines by @qinxuye in #5669
feat: support qwen3.8-Flash-Next model by @llyycchhee in #5673
feat: unify LLM context length across engines and add UI slider by @qinxuye in #5672
feat(llm): add MiniCPM-V-4.7 support by @Minamiyama in #5676
feat: support Anthropic context management by @m199369309 in #5679
feat(image): add Iris-3B and Turbo support by @Minamiyama in #5686
feat: enforce Anthropic multimodal request limits by @m199369309 in #5680
feat: negotiate Anthropic backend capabilities by @m199369309 in #5681
feat: update reasoning parsing and reload vLLM image limits by @qinxuye in #5698
feat: reload SGLang image limits with retained weights by @qinxuye in #5704
FEAT: Add request-scoped model catalog credentials by @qinxuye in #5720
feat: extend Anthropic engine controls and cache lifetimes by @qinxuye in #5724
FEAT: Support single-model catalog refresh by @qinxuye in #5728
FEAT: Add single-model catalog refresh to Web UI by @qinxuye in #5736
feat: support DeepSeek-V4.1-Flash-1017 model by @llyycchhee in #5753
feat(image): add MOSS-Image-SFT support by @Minamiyama in #5755
FEAT: Support pluggable advanced-auth storage by @qinxuye in #5750
FEAT: Isolate Xavier caches with per-request salts by @qinxuye in #5749
feat(ui): show generation elapsed time and ETA by @Minamiyama in #5763
feat(ocr): add Xiaomi-OCR-0 support by @Minamiyama in #5764
feat: add TRT-LLM backend with Xavier KV cache sharing by @qinxuye in #5771
feat: add pluggable Xavier cache backends with LMCache disk storage by @qinxuye in #5784
feat: enable Xavier LMCache storage for SGLang and TRT-LLM by @qinxuye in #5787
feat(tts): support Ming-3.0 TTS Tiny and 0.5B by @Minamiyama in #5788
FEAT: Support native and Xavier TRT-LLM P/D separation by @qinxuye in #5780
feat(image): add FIBO2 Dev and Edit Turbo support by @Minamiyama in #5790
feat: relaunch Workers without restarting model processes by @qinxuye in #5789
FEAT: Support TRT-LLM cross-engine Xavier P/D by @qinxuye in #5794
feat(image): add FLUX 3 Image Klein support by @Minamiyama in #5792
FEAT: Persist model favorites with pluggable storage by @qinxuye in #5802
feat(trt-llm): expand model support and speculative decoding by @qinxuye in #5804
FEAT: Support custom launch history storage by @qinxuye in #5808

Enhancements

ENH: Add EmbeddingGemma 2 GGUF multimodal support by @Minamiyama in #5658
ENH: Keep inference serving through model and Worker upgrades by @qinxuye in #5800
ENH: Replace API processes without interrupting inference by @qinxuye in #5811
ENH: Include model type and engines in cached model listings by @qinxuye in #5809
ENH: Replace Supervisor processes without interrupting inference by @qinxuye in #5823
ENH: Add API and Supervisor hot replacement controls to Web UI by @qinxuye in #5835
ENH: Support local component upgrades with on-demand process separation by @qinxuye in #5836
ENH: Improve upgrade controls and add workflow screenshots by @qinxuye in #5849
perf(llm): stop rescanning DeepSeek, GLM-5, Gemma, MiniMax and Kimi-K3 tool call streams on every delta (32 KB call: 12.5 s to 11 ms) by @deadstrobe5 in #5711
perf(llm): find Qwen tool call tags with str.find while streaming by @deadstrobe5 in #5710
perf: reduce embedding GC overhead and mixed-length padding by @qinxuye in #5739
perf: improve rerank auto batching and cache cleanup by @qinxuye in #5748
perf: batch native audio inference requests by @qinxuye in #5747

Bug fixes

fix(llm): keep every MiniMax tool call when streaming by @hpdkhoa in #5660
fix(llm): stop Gemma streaming from sending a second tool call as text by @hpdkhoa in #5659
fix: complete MLX virtualenv dependencies for built-in LLMs by @qinxuye in #5668
fix(llm): keep DeepSeek V3.2 and V4 streaming tool parsing per request by @Jinzhengxu in #5670
fix(cli): exit with an error when xinference launch fails by @MohammadHijjawi97 in #5675
fix: improve Claude Code compatibility for local models by @qinxuye in #5671
fix: preserve model indexes in the offline mirror by @qinxuye in #5677
fix(router): treat chat_template_kwargs.thinking as enable_thinking by @Jackmeson1 in #5684
fix: support bare filenames in safe log handlers by @original4422 in #5683
fix(supervisor): roll back earlier shards when a sharded launch fails by @Jackmeson1 in #5685
fix(llm): keep text around DeepSeek V3.1, V3.2 and V4 tool calls without streaming by @Jackmeson1 in #5687
fix(core): record token metrics for non-streaming completions by @Jackmeson1 in #5690
fix(llm): keep every GLM-5 tool call and text when streaming by @jipeng6036-del in #5689
fix(vllm): keep the caller's stop and stop_token_ids for reasoning models by @Jackmeson1 in #5691
fix(llm): keep string tool arguments as text in the XML-style tool parsers by @Jinzhengxu in #5682
fix(sglang): keep the final usage and finish reason when the last stream event adds no text by @Jackmeson1 in #5692
fix(api): abort non-streaming vLLM requests when the client disconnects by @Jackmeson1 in #5694
fix(llm): render Qwen3-VL tool-only assistant history by @ace-xc in #5051
FIX: Honor live account permissions and support self-service password changes by @qinxuye in #5695
fix(transformers): build batching attention masks for left-padded inputs by @Jackmeson1 in #5699
fix(transformers): keep a batch running when one request is aborted by @Jackmeson1 in #5701
fix(auth): require an operator token for initial admin setup by @qinxuye in #5703
fix: complete Anthropic vLLM cancellation and stop metadata by @qinxuye in #5697
fix(transformers): use greedy decoding for Qwen2-VL at temperature 0 by @Jackmeson1 in #5708
fix(model): reject backslash, "." and ".." in custom model names by @kartsan03 in #5707
BUG: Return HTTP 400 for malformed login requests by @qinxuye in #5705
fix(audio): stop capping Whisper transcriptions at 128 new tokens by @Jackmeson1 in #5709
fix(transformers): honour max_tokens for Qwen Omni and skip audio when streaming by @Jackmeson1 in #5714
fix: release unconsumed P/D decode streams by @qinxuye in #5712
fix(transformers): honour request thinking settings in direct-chat reasoning output by @Jackmeson1 in #5716
fix: handle empty assistant content in Qwen3 tool history by @qinxuye in #5718
fix: enforce SGLang reasoning boundaries and register Qwen3-VL by @qinxuye in #5719
fix(sglang): render tool-call history whose assistant content is null or omitted by @kartsan03 in #5722
fix(rerank): convert bfloat16 cross-encoder scores to float32 by @MohammadHijjawi97 in #5723
fix(transformers): cut batch output at the earliest stop string by @kartsan03 in #5725
fix(llm): keep leading chunks without text in async reasoning streams by @spicysharkco in #5726
fix(transformers): use greedy decoding at temperature 0 for Gemma 3, Gemma 4, GLM-4V and MiniCPM-V 4.6 by @kartsan03 in #5727
fix(llm): drop the start tag from non-stream reasoning content without an end tag by @spicysharkco in #5731
fix(transformers): enforce the launch context_length on multimodal direct chat by @Jackmeson1 in #5729
fix(model): use diffusers 0.35.2 for the specs pinned to 0.35.1 by @Jackmeson1 in #5732
fix(llm): cut non-stream content at the end tag when text comes before the start tag by @spicysharkco in #5733
fix(transformers): enforce the launch context_length on Gemma 3 and MiniCPM-V 4.6 by @Jackmeson1 in #5734
fix(transformers): enforce the launch context_length on Ovis 2 and DeepSeek-VL2 by @Jackmeson1 in #5735
fix(transformers): render tool-call history whose assistant content is null or omitted by @kartsan03 in #5737
fix(transformers): enforce the launch context_length on Qwen Omni by @Jackmeson1 in #5738
fix(transformers): return text-only replies for Qwen3-Omni-Thinking non-streaming chat by @Jackmeson1 in #5740
fix(vllm): pass images to vLLM for audio + vision models such as Gemma 4 by @Jackmeson1 in #5743
fix(llm): hold leading whitespace before inserting the stream reasoning start tag by @spicysharkco in #5746
fix(llm): keep a start tag after the end tag as content in non-stream reasoning by @spicysharkco in #5751
fix(transformers): pass the audio to the Qwen2-Audio processor by @Jackmeson1 in #5752
fix(transformers): enforce the launch context_length on Qwen2-Audio by @Jackmeson1 in #5754
fix(llm): keep an end tag after the reasoning ended as stream content by @spicysharkco in #5756
fix(embedding): honor flat llama.cpp launch options by @kartsan03 in #5761
fix(ci): recognize explicitly scheduled Linux tests by @qinxuye in #5762
fix(vllm): support Xavier KV transfer configuration on vLLM 0.32 by @Jackmeson1 in #5758
fix(vllm): respect checkpoint quantization for fp8 models by @Jackmeson1 in #5760
fix(ui): upgrade Next.js and frontend deps for security advisories by @martinma51 in #5768
fix(image): honor remote-code trust policy for MOSS pipelines by @qinxuye in #5769
fix(audio): gate custom CosyVoice loading on operator trust by @qinxuye in #5765
fix(models): restrict auto-registration paths and flexible launchers by @qinxuye in #5766
fix(auth): support live API-key validation for shared stores by @qinxuye in #5770
fix(api): sanitize chat worker error responses by @qinxuye in #5767
fix(mlx): render tool-call history whose assistant content is null or omitted by @kartsan03 in #5775
fix(rerank): return document objects from llama.cpp rerank by @kartsan03 in #5776
fix(vllm): honor tool_choice none in chat requests by @Jackmeson1 in #5783
fix(rerank): vllm path compat by @llyycchhee in #5774
fix(ocr): resolve the load device through the accelerator registry by @li-lizhe in #5778
fix(worker): select distinct GPUs for multi-GPU allocations by @Jackmeson1 in #5786
fix(sglang): parse streamed events split across HTTP reads by @kartsan03 in #5791
fix(transformers): apply stop strings in direct chat models by @kartsan03 in #5793
fix(model): decode local file model URIs by @Jackmeson1 in #5796
fix(llm): normalize tool-only history before counting tokens by @Jackmeson1 in #5799
fix(api): forward parallel_tool_calls from /v1/responses to the chat request by @MohammadHijjawi97 in #5801
fix(deps): require patched pyarrow and bump follow-redirects for security advisories by @martinma51 in #5803
fix(llm): preserve tool-call history in null-content templates by @Jackmeson1 in #5806
fix(llm): treat a missing stream_options.include_usage as false by @kartsan03 in #5810
fix(client): send multipart forms from async image_to_image and inpainting by @David-Wu1119 in #5812
BUG: Fix gpt-oss Harmony channel parsing in vLLM by @mikemikimike in #5807
fix(client): keep the async speech stream open until it is read by @David-Wu1119 in #5813
fix(client): send each timestamp granularity as its own form field in the async client by @David-Wu1119 in #5814
fix(client): keep the server's error detail in async client errors by @David-Wu1119 in #5815
fix(client): send a multipart form from async flf_to_video by @David-Wu1119 in #5817
fix(client): apply the model handle's timeout to its async session by @David-Wu1119 in #5816
fix(client): let async launch_model return without waiting for the model by @David-Wu1119 in #5818
fix(client): send positional args and return the result from async flexible infer by @David-Wu1119 in #5819
fix(cli): show running flexible models in xinference list by @David-Wu1119 in #5820
fix(audio): pass speed to CosyVoice speech synthesis by @David-Wu1119 in #5822
fix(audio): report the translate task in Whisper verbose_json by @David-Wu1119 in #5824
fix(core): keep clearing finished progress after the first cleanup by @David-Wu1119 in #5825
fix(api): return 404 for an unknown model in the Anthropic models endpoint by @David-Wu1119 in #5826
fix(llm): keep whitespace inside Qwen XML tool-call string arguments by @David-Wu1119 in #5828
fix(llm): restore the official GLM-4.7 chat template by @David-Wu1119 in #5830
fix(llm): parse GLM-4.5/4.6/4.7 tool calls with the XML tool parser by @David-Wu1119 in #5829
fix(llm): keep whitespace in GLM-5 and MiniMax string tool arguments by @David-Wu1119 in #5832
fix(api): accept tool calls for MiniCPM5 models by @David-Wu1119 in #5833
fix(llm): give Llama 3.3 Instruct the Llama 3 tool parser by @David-Wu1119 in #5834
fix(llm): forward Llama tools to vLLM, SGLang and MLX chat templates by @Jackmeson1 in #5838
fix(llm): keep all DeepSeek-V3.1 tool calls completed in one engine text delta by @Jackmeson1 in #5840
fix(mlx): honor stop strings in serial and batched generation by @Jackmeson1 in #5844
fix(llm): add DeepSeek-V3.2 reasoning tags by @Jackmeson1 in #5842
fix(llm): keep completion IDs stable across tool stream chunks by @Jackmeson1 in #5846
fix(image): accept controlnet given as a list by @Jackmeson1 in #5848
fix(docker): resolve TRT-LLM mirror dependencies on both architectures by @qinxuye in #5854

Documentation

DOC: update new models for v4.0.0 by @qinxuye in #5657
doc: fix starter guide anchors in translated READMEs by @pratikgx in #5662

Others

docs: add the v4.0.0 row to the release notes index and locale catalogs by @martinma51 in #5667
TEST: Add native Anthropic regressions and stabilize MLX mocks by @qinxuye in #5678
test: cover Anthropic compatibility controls and stop sequences by @qinxuye in #5693
SEC: integrate Xoscar actor transport authentication by @qinxuye in #5696
ci: fix SGLang setup and MLX native tool-loop probe by @qinxuye in #5713
ci: provide matching CUDA toolkit for SGLang native smoke by @qinxuye in #5717
ci: complete MLX native regression coverage by @qinxuye in #5721
CI: focus Windows and macOS on platform compatibility by @qinxuye in #5745
ci: run installer checks after lint succeeds by @qinxuye in #5772
CI: skip unrelated GPU runs and install dependencies by test group by @qinxuye in #5773
TEST: Handle SSE heartbeats in API cutover acceptance by @qinxuye in #5827
chore: drop the redundant pynvml dependency by @kartsan03 in #5741
chore: remove LMDeploy and validate test platform coverage locally by @qinxuye in #5781
chore(image): use released Diffusers for FLUX.3 and FIBO2 by @Minamiyama in #5831

New Contributors

@pratikgx made their first contribution in #5662
@rennf93 made their first contribution in #5664
@Jinzhengxu made their first contribution in #5670
@original4422 made their first contribution in #5683
@jipeng6036-del made their first contribution in #5689
@kartsan03 made their first contribution in #5707
@deadstrobe5 made their first contribution in #5711
@spicysharkco made their first contribution in #5726
@li-lizhe made their first contribution in #5778
@mikemikimike made their first contribution in #5807

Full Changelog: v4.0.0...v4.1.0

xorbitsai/inference v4.1.0 on GitHub