Highlights

950 PRs from 245 contributors.

New models

What's new

• DeepSeek-V4.1 gets up to 2.3x more throughput per GPU on H200, and now runs on AMD MI350X / MI355X with up to 1.9x throughput from DSpark (#41251, #41308).
• Qwen3.8-Flash-Next decodes 12% faster on B200, and GLM-5.3-Flash with DFlash 10% faster on B300 (#42175, #42489).
• MiMo-V2.6-Pro gets 1.85x decode throughput with DFlash on B300 (#41886).
• Models load 10x faster with the new InstantTensor loader (#40453).
• PD disaggregation gets up to 27% more throughput and half the p95 TTFT on long-prompt workloads (#40703).
• HiSparse now supports EAGLE and MTP for long-context serving with the KV cache in CPU memory (#37771).
• GLM-5.2 and GLM-5.3 can now combine decode context parallelism with MTP, with up to 2.1x faster decoding on GLM-5.3 (#43735).
• SGLang now runs on Apple Silicon Macs with the standard PyTorch backend, alongside MLX (#36780).
• MiniMax-H3 and Qwen-Image 2.1 now run in ComfyUI with SGLang Diffusion, which also loads community GGUF checkpoints and LoRAs (#43504, #43581, #42556, #39523, #42174).
• Important fix: Hybrid linear models could resume from wrong mamba states when speculative decoding and structured output were used together. Thanks to @Wirick and Baseten for finding it (#43029).

To upgrade

SGLang now requires Python 3.11 or newer.

uv pip install --prerelease=allow sglang==0.5.22

Full Release Notes

Speculative Decoding

• [RL][Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl: #34201
• [HiSparse] Add speculative decoding support with multi-step KV cache swapping: #37771
• [Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens: #38991
• [Spec][PP] Fix hybrid recurrent-state commit and micro-batch pairing under PP x speculative decoding: #40001
• [Spec] Reuse K3 auxiliary outputs across decode CUDA graph sizes: #40159
• [NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe: #40772
• fix(spec): enable Qwen3.5 EAGLE3 capture and streaming overlap: #41154
• [Qwen3.8-Flash-Next] Fuse QSA graph replay metadata across draft steps: #41173
• [Qwen3.8-Flash-Next] Fuse NEXTN verify and draft graph input preparation: #41175
• [HiSparse] fix: reject speculative decoding and mixed chunk at startup: #41347
• The sgalng api specs: #41585
• [Spec] Dspark support: #41623
• Fix XQA draft extend with FP8 KV cache in trtllm_mha: #41656
• [QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding): #41729
• [DFlash] Keep grouped-conv taps inside each request's block so NaN/Inf cannot leak across requests: #41736
• [DFlash2] Keep the greedy selector walk in range for all-NaN score rows: #41737
• [PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution: #41783
• [Fix] Pass the draft's attention ownership to DFLASH's eager LiLiCorr scope: #41808
• [Spec] Fix Qwen3.5 text model EAGLE3/DFLASH aux-layer capture: #42002
• [Spec][MegaMoE] Let the speculative draft choose its own W4A4 MXFP4 MegaMoE MMA type: #42022
• [Spec] LiLiCorr: named head MLPs and quantized head linears: #42057
• [Spec] Add opt-in block verification: #42297
• [Fix] Load a draft's tensor weight update from its deployment TP rank: #42346
• [Fix] Skip a draft's decode recapture when it owns no decode graph: #42347
• [Perf] Improve GLM-5.3-Flash DFlash decoding on B300 (decode 1091.9 to 1199.7 tok/s on 8x B300 at concurrency 1): #42489
• [DSpark] Mask uncommitted slots before commit-inject cache lookups: #42624
• Back the DFlash-family draft KV pool to the post-capture token count: #42651
• perf(kda): rebalance MTP verify precompute and state synchronization (verify kernel 13.2 to 9.7 µs on Vera Rubin): #42683
• [Spec] Fix wrong tokens from compact-verify CUDA graphs when a token tier is captured more than once: #42709
• [KDA] Support Helion MTP verification and harden prefill, decode: #42857
• [KDA] Default GLM-5-Next spec verify to the CuTe DSL KDA kernel: #42905
• [DSpark] Fold draft sampling for DeepSeek-V4.1 heads under AUTO: #42987
• [Spec] Support block verification for the DFlash family: #43012
• [Scheduler] Use committed length in spec mamba track boundary check (found by @Wirick at Baseten): #43029
• [Spec] Allow Qwen3.5 backends to opt into packed target verification: #43037
• [DFlash] Build the attention QKV projection through an overridable hook: #43179
• [DSpark] Run every markov head through one step interface: #43620
• [DSpark] Add MarkovSampleStep, one op for a markov draft step's add and token pick: #43621
• [DSpark] Draft the eager markov block through MarkovSampleStep: #43622
• [DSpark] Run the same markov block in the draft graph tail; drop SGLANG_DSPARK_FOLDED_SAMPLING: #43668
• [DSpark] Merge the draft TP sync sites into dspark-draft: #43669
• [DSpark] Add SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot: #43670
• [DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD: #43734
• [DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager: #43790

Breakable CUDA Graph

• [Fix] Llama4 local attention: read page ids from the graph tables in CUDA-graph capture/replay (page_size > 1): #38187
• [BCG][DeepSeek-V4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs: #38603
• fix(cuda-graph): remove unnecessary CP batch alignment: #40543
• [GLM-5.3-Flash] Capture the KDA extend in breakable prefill CUDA graphs: #41105
• Drop the GLM-5.3-Flash breakable prefill chunk-size pin: #41837
• [Fix] MiniMax-M3: default to the breakable prefill CUDA graph: #41845
• Size the VMM graph-input exchange by the widest input across ranks: #42125
• [DSA] Fix pooled page table refresh for target verify CUDA graph replay: #42750
• [GLM-5.3-Flash] Enable breakable prefill CUDA graph by default: #42845
• Clamp default decode CUDA graph max_bs to max_running_requests: #43172
• [Fix] Unblock CUDA graph startup for hybrid parallelism: #43211
• Allow mixed chunk prefill on the Full prefill CUDA graph backend: #43461
• [DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph: #43736
• Fix autograd retention in CUDA graph capture tail hooks: #43854

Attention Backends

• [SM120&90] Add CUDA fused Triton sparse-MLA prefill backend for DSA: #32779
• GLM5.2 DSA Indexer Fallback: #32947
• fix(fa4): isolate sheared-bias workspaces by CUDA stream: #33000
• fix: default Inkling to fa4 on Hopper: #35292
• Fix Kimi-K3 MLA output gate dispatch on non-CUDA devices: #36406
• Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init: #37525
• Skip fully-masked KV tiles in sliding-window extend attention (up to 39% faster kernel at 8K tokens): #39769
• [Linear Attention] Expose GDN/KDA prefill hooks and auxiliary cache accounting: #40227
• Support kv cache sharding for DSA indexer: #40925
• [Qwen3.8-Flash-Next] Fuse QSA KV preparation and sparse block expansion: #40972
• [Kernel] Expose FlashMLA kv_format in sgl-kernel sparse decode: #41336
• [DeepSeek-V4][DSA] Name the FlashMLA KV format and drop the V4.1 support probe: #41337
• feat(kda): integrate FlashInfer prefill checkpoints: #41400
• [KDA] Enable the ptx_kda prefill backend on SM100 (B200 / GB200): #41445
• [DeepSeek-V4.1] Add opt-in TRT-LLM sparse attention support: #41603
• [DeepSeek-V4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs: #41659
• [Perf] Use FA4 by default for MiMo on SM100 (1.85x DFlash decode throughput on 8x B300): #41886
• [Qwen3.8-Flash-Next] Fuse QSA prefill KV gathering and dtype conversion: #41933
• [DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer: #41987
• [Gemma4][Attention] Enable Gemma4 FA4 on SM90 with head-dim 512 support: #42019
• [Perf] Optimize MiMo local audio attention on Hopper: #42101
• [DeepSeek-V4] Guard V4.1 indexer SM100 check on non-CUDA devices: #42114
• [DeepSeek-V4.1] Bounded replay with sparse mla path: #42273
• fix(dsa): align FA3 metadata with unpadded queries: #42793
• [Fix] Run attention on a padded extend batch's real rows: #42896
• [FA4] Keep stream last in the SM100 MLA kernel so disk-cached objects load: #42926
• [Perf] Route BF16 mixed-batch decode tails through decode attention: #43238
• [KDA] Default safe-gate Blackwell prefill to FlashInfer: #43246
• [QSA] Bound the decode indexer's work by the context length (bs64 decode step 28.75 to 26.54 ms for Qwen3.8-Flash-Next on 4x B200): #43463
• [Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs: #43918

MoE & Expert Parallelism

• [Model] Add LLaDA2.2 Block Routing MoE support: #31768
• DeepEP v2: expanded (do_expand=True) prefill dispatch: #37261
• Add per-token NVFP4 MoE support for ReLU2 activation: #39199
• fuse shared experts with routed experts in MegaMoE's DeepGEMM: #39313
• [MegaMoE] Preserve W13 layout for ModelOpt NVFP4 experts: #39388
• [MoE] Use FlashInfer A2A for prefill instead of AG+RS: #39818
• [Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP: #40671
• [Fix] MoE: require TopK layer_id to ensure routed expert captures: #40980
• [Fix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path: #41072
• [Fix] Select the MXFP4 MoE runner for MiMo-V2 packed experts on SM100: #41668
• [MoE] Keep the moe_align_block_size pad fill inside sorted_token_ids: #41759
• [Fix] Shard MoE WNA16 and Quark INT4-FP8 weights by the MoE placement: #41807
• [EPLB] Warm default-group NCCL P2P transports before KV-cache sizing: #42026
• [Fix] Select the Marlin MoE runner for MiMo-V2 packed MXFP4 experts on SM90: #42172
• [MoE] Skip block-padding rows in the TMA down activation and quant: #42212
• [Fix] Offer the fused MoE finalize all-reduce only to a block that owes the sum: #42479
• [GLM-5.3-Flash] Autotune MoE kernels at prefill sizes: #42693
• [MoE] Fill num_token_non_padded for the customized A2A backend at EP 1: #43245
• support DeepEP v2 on GLM-5.3: #43432
• [FlashInfer] Add startup autotuning for MegaMoE: #43789

Quantization

• [Qwen3.8-Flash-Next] Optional fp8 (e4m3) storage for the compressed QSA indexer cache: #39614
• Fix/compressed tensors nvfp4 hopper: #40022
• [JIT] Fuse FP8 KV-cache quantization into the prefix-valid commit kernel (2.99x faster commit kernel on B200, kernel level): #40785
• fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends: #41765
• [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache (sparse prefill 1.53x to 1.72x faster per DSA layer than tilelang on H200): #41790
• [DeepSeek-V4.1] mxfp8 dispatch cache: #42156
• [Fix] Guard DeepSeek NVFP4 shared-expert fusion for LoRA and FP4 backends: #42203
• [Fix] Quantize reloaded FP8 weights after native sharding: #42585
• perf(mla): use native SATFINITE FP8 conversion in MLA KV/Q preparation: #42659
• [Perf] Cache compiled ModelOpt exclusion patterns: #42927
• [DSA] Support TileLang FP8 KV cache on CUDA: #42957
• [Fix] Account for captured inputs in DeepGEMM masked workspace budget: #43563

Parallelism & Disaggregation

• [Fix][DP] Prevent health checks from perturbing user request routing: #35721
• [PD] Centralize drain-aware abort acknowledgements: #37077
• NIXL: Use stride desc API: #37827
• [PP+PD] Fix #38206 Reduce bootstrap consensus latency: #38959
• [Metrics] Fix PD latency histogram accounting: #39706
• [PD] Preserve bootstrap metadata in native Messages requests: #39711
• [Qwen3.8 CP] Context parallelism for QSA attention and the sparse indexer: #39721
• [KV Shard] Enable Control Plane C: #40102
• [DeepSeek-V4.1][PD] Decode node support DP attention: #40177
• [DeepSeek-V4.1][PD] Support prefill node use cp parallel strategy: #40323
• [PD] Add opt-in prefill-complete decode KV allocation (26.8% higher throughput and 48.9% lower p95 TTFT on GLM-5.2-NVFP4 under decode KV pressure): #40703
• [PD] Support heterogeneous TP for hybrid SWA KV transfer: #40909
• [Fix] Stop PD-decode queue_time from counting decode time before a retraction: #41380
• [PD] Wait for prefill completion before Mooncake early KV transfer: #41395
• [PD][Mamba] fix: free the COW mamba slot of decode requests dropped before preallocation: #41451
• [PD] Validate state strides before Mooncake transfers: #41607
• [PD] Reload NIXL peer metadata after the peer is invalidated: #41691
• [PD] Publish prefill DP rank whenever forced lookup is enabled: #41786
• [Fix] State the configured MoE-DP width in the weight-cache fingerprint: #41806
• [Fix] Size the prefill delayer's gather buffer to its TP group: #41809
• Add --attn-dp-size and deprecate --enable-dp-attention: #41818
• [PD] Fix block scale registration for mixed full/SWA KV dtypes: #41924
• [PD] Fix crash for allocation when mamba + pd + decode radix cache: #41927
• [PD][NIXL] Fix prepared dlists for KV entries of different lengths (DeepSeek-V4): #42005
• [PD] Keep ingesting requests while a prefill forward result is pending: #42035
• [CP] Support DP x TP x CP for attention with interleave CP strategy: #42038
• [CP] Support scattered interleave CP inputs with expert parallelism: #42041
• [CP] GLM-5.3-Flash Prefill CP implemented on layer boundaries (P50 TTFT 2% to 9.5% lower than TP4 on 4x B300): #42044
• [PD] Validate decode state layout once at registration: #42051
• [PD] Make the prebuilt last-token H2D copy non-blocking under overlap: #42113
• [PD] Stop reserving unallocated mamba spec-verify scratch in prefill KV sizing: #42649
• [Metrics] Add per-rank DP attention imbalance metrics: #42755
• [Fix] Return an FFN's output to the attention rows at a pipeline handoff: #42892
• [Fix] Shard Kimi K3's shared experts over the TP group they sum over: #42895
• [PD] Allow state-only KV checksum inputs: #42904
• [GLM-V] Shard the vision tower over the attention-TP group under DP attention: #42931
• [PD] Charge decode admission the page-rounded allocation when the radix cache is disabled: #43173
• [Metrics] Count per-rank DP attention pairs and their imbalance: #43180
• [Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio: #43187
• [DeepSeek-V4.1] Enable DP attention with MegaMoE: #43228
• [Scheduler] Keep a DP-converted decode batch as the running batch: #43283
• [CUDA Graph] Size dp-local decode graphs by the runner's own gather requirement: #43303
• [PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages: #43329
• [PD] Use 1 to disable decode host receive and 0 for always-on: #43407
• [PD] Receive decode KV into separate paged SWA host pools; let host-capable receivers own abort drain tracking: #43410
• Skip the eager activation reserve in KV sizing on a capped Full prefill role: #43435
• [PD] Reclaim parked optimistic prefills when request slots are exhausted: #43442
• [Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks: #43479
• Preserve serialized page dtype in NIXL copy fallback: #43593
• Keep NIXL Mamba checkpoint keys separate by TP rank: #43594
• [PD] Poison the deferred abort ack when a Mooncake send fails: #43719
• [PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids: #43723
• [DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP (on GLM-5.3, 8x B300, adding MTP speeds DCP2 up 1.8x and DCP8 up 2.1x): #43735
• [PD] Retract transferred decode requests on a retract pause: #43757
• [DeepSeek-V4][CP] Write low-ratio page indices when CP prefill runs paged attention: #43935
• [PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode: #43936
• [DeepSeek-V4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches: #43993
• [PD] Return prompt routed experts from the PD prefill server: #44002

Scheduler & Runtime

• [Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather: #34076
• [Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities: #35594
• [Metrics] Fix non-streaming TTFT by flushing the first output through detokenization: #38600
• [Weight Cache] Coordinate daemon readiness across ranks: #38828
• feat(grpc): expose follower metadata and node-local KV sources: #39659
• tokenizer perf optimization: #39771
• [Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds): #40275
• fix(nccl): disable graph buffer registration for pausable graph pools: #40648
• feat(metrics): expose deferred decode KV release metrics: #41128
• [Perf] Tune SM90 GDN recurrent verify launch for small batches: #41486
• [KDA] Fix ptx_kda prefill NaN without a gate lower bound and workspace growth: #41572
• [Fix] Release consumed residual contributions: #41749
• [Fix] scheduler: stop latching batch_is_full when nothing can run: #41872
• [Fix] Make tokenizer request-state initialization atomic: #41911
• [Metrics] Label scheduler stage wall time by sampled forward overlap: #42025
• [Perf] Reduce multi-detokenizer router IPC sends: #42238
• Pick the default nccl_port below the kernel ephemeral range: #42405
• [Fix] Make the fused KDA beta sigmoid bit-identical to torch.sigmoid: #42611
• [Scheduler] Gather prefill-delayer queue timeout across ranks: #42625
• Bind the MessageQueue remote socket atomically: #42695
• [Scheduler] Skip the decode for a request whose last token is already in flight: #42720
• [Metrics] Read scheduled prefill KV tokens from batch prefix_lens: #42797
• [dLLM] Keep the request row across FDFO blocks: #42846
• [Perf] Fuse short-convolution checkpoint tracking metadata: #43013
• [Memory] Share one full prefill graph output buffer across shapes for tuple and list outputs: #43176
• Overlap scheduler startup with the data parallel controller: #43177
• [Perf] Avoid gate copies in gated RMSNorm: #43227
• [Perf] Fuse BF16 writes to native HND KV cache storage: #43237
• [Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool: #43290
• [Scheduler] Requeue preempted requests when admission admits nothing: #43298
• Fix torch.compile crash in fused gate-sigmoid-mul launcher: #43384
• [KV Events] fix missing topics in replayed kv events: #43386
• [Fix] Keep request-pool row generations monotonic across clear(): #43439
• [Scheduler] Fix the deterministic-inference hang when a prefill chunk is shorter than its alignment: #43444
• [Fix] Reserve remaining tokens for ignore_eos requests during prefill admission: #43564
• [Perf] Compact GDN causal-conv grids for skewed prefill batches: #43570
• [Fix] Adapt to transformers 5.18 / 5.19 changes: #43688
• [RL] Return a failed weight update when the cache flush is refused: #43758
• [Core] Add request-level customized_info keys and mark Full prefill capture batches: #43772
• [Observability] Report per-request storage prefetch and staging waits in meta_info: #43946
• [Fix] Work around a CPython fork deadlock in the CPU preprocess pool: #44143

Sampling

• Add Aaronson-Gumbel text watermarking: #37577
• [Kernel] Use Cake softmax for qualified SM103 sampling workloads: #42004
• [Debug] Add opt-in NaN detection before sampling: #42690
• Return output top logprobs as flat base64 arrays on request: #42890
• Inject "req" into custom_params only for custom logit processors: #43181
• [Sampler] Sort only the max(top_k) leading columns in the torch sampler: #43346

HiCache & Radix Cache

• [HiCache] Route --file-storage-path to the file storage backend: #33883
• [MemCache] Bound FlexKV checkpoint stores and restore DSPARK cache checks: #35694
• [Mamba] Warm cache COW kernel before serving (first-wave cold excess 924 to 28 ms): #36266
• [HiCache] Support Mamba in buffer-only host memory mode: #36345
• Allow platforms to declare Mamba extra-buffer support: #36410
• [Unified Memory] Speculative decoding on the unified memory pool: translate every read and write path: #37627
• [Unified Memory] Fuse the draft model's KV into the target's page envelope (mechanism): #37628
• fix: wait for HiCache load-back before hybrid MLA KV reads: #38217
• fix(inkling): translate unified-memory checkpoint destinations to physical slots: #38229
• [HiCache]: Fix host lock ownership across radix splits: #38480
• [Unified Memory] Token-major dense views for the unified memory pool (3/7): #38592
• Fix unified HiCache physical transfers: #39479
• [HiCache] Add staged write-back for the page-unified KV cache layout: #39606
• [Perf] Batch Mooncake hybrid-pool metadata and puts: #39696
• [Fix] HiCache: carry registered Mamba slot side states (Qwen4-Exp PLE) through the host tier: #39862
• fix: preserve QSA indexer state through HiCache: #39893
• [Fix] Fix unified-memory compaction gates and pending page reuse: #39982
• [HiCache] Preserve all MTP KV and recurrent state: #40223
• [Unified Memory] Derive KV row addresses from strides, not shapes (1/7): #40326
• [Unified Memory] Build paged KV views through one helper (2/7): #40327
• [Unified Memory] Mark the write loc physical and check it at every write door (4/7): #40328
• [Unified Memory] Remove the kernel-page multiplier plumbing (5/7): #40329
• [Unified Memory] Stride the KV translate kernel and route every translate through it (6/7): #40330
• [Unified Memory] Name the fused KV translate for what it computes (7/7): #40331
• fix(dsv4): ship the fp8 unified_kv rope pool through the direct external linker: #40471
• [Unified Memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts: #40491
• [Unified Memory] Admit DFLASH and DSPARK with fused draft KV: #40492
• [Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler: #40896
• Declare HiCache host pools for DSA indexer: #40913
• Declare hybrid Mamba indexer host pool: #40915
• [Fix] Allocate SWA KV pools inside the memory-saver region: #40979
• [DeepSeek-V4] Reserve the FULL logical page in the c4 indexer pool like its KV pool: #41163
• [HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free: #41346
• [HiCache][PD] fix: keep decode load-back polling out of rank-local collectives: #41393
• [HiSparse] Avoid host synchronization in eager backup: #41446
• [HiCache][PD] fix: clamp the decode restore to the prefix promised to prefill: #41450
• [MemCache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes: #41520
• [Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction: #41593
• [HiCache] Attribute buffer-mode storage hits against the joint device match: #41758
• [HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation (KV capacity +11.6% on prefill and +13.7% on decode for GLM-5.2-NVFP4 PD): #41769
• [Fix] Read TensorCast's WORLD placement under its current names: #41805
• Support post-capture KV sizing for the unified hybrid-SWA pool: #41961
• fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots: #42072
• [HiCache] Avoid full-store scans for uncached file prefix lookups: #42077
• [HiCache] Fall back to host memory when no cgroup fs is mounted: #42120
• [Fix][DeepSeek-V4.1] SWA page size with bounded replay: #42128
• [HiCache] Let the MiniMax-M3 K-only host pool join a host pool group: #42166
• [DSA] k-pool: 256-token logical page so index page id = logical page id: #42178
• [MemCache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public: #42194
• [HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag: #42215
• [Unified Memory] Copy page envelopes in place during compaction: #42252
• [HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert: #42264
• [Session] Run streaming sessions only on UnifiedRadixCache; reject unverified tree caches: #42295
• [Session] Release aborted streaming turns through the normal release path: #42330
• [MemCache] Run mamba models on UnifiedRadixCache when the radix cache is disabled: #42354
• [HiCache] Add SeaweedFS L3 storage backend (zero-copy path cuts L3 TTFT 3.5x to 4.7x for Qwen3-4B on one A10): #42399
• [HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails: #42420
• [Fix] Make Hf3fsMockClient reads and writes thread-safe with pread/pwrite: #42425
• [MemCache] Keep prefix_indices current for chunked requests kept out of the tree: #42467
• [Session] Let a streaming session own its KV record and tree lock from the first turn: #42537
• [Session] Count streaming session KV by owner in pool accounting: #42539
• [HiCache] Manage buffer-mode backups per storage pool: #42633
• Rename SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA: #42657
• [HiCache] Resolve write_back to write_through under buffer_only host memory: #42754
• [HiCache] Keep protected cgroup page cache charged in host budgets: #42788
• [MemCache] Return only the matched length from match_prefix; read KV indices off the node path: #42923
• [MemCache] Return only the loaded length from init_load_back; drop empty_device_indices: #43023
• Fix DSA index host buffer cleanup: #43100
• Fix K-only host buffer cleanup: #43101
• [DeepSeek-V4.1] Initialize the full-to-SWA mapping for request-window pools: #43165
• [HiCache] Re-admit a split node's queued buffer-mode write instead of dropping it: #43174
• [Session] Skip the KV free when page alignment empties a streaming-session rewind: #43183
• [MemCache] Count reclaimable tokens in the unified tree ledgers: #43185
• [MemCache] Serve disabled pure-SWA models with PureSWARadixCache: #43198
• [MemCache] Serve disabled full-attention models with UnifiedRadixCache: #43199
• [MemCache] Serve disabled hybrid-SWA models with UnifiedRadixCache: #43200
• [MemCache] Delete ChunkCache: #43201
• [MemCache] Skip SWA request-cap pool sizing when chunked prefill is off: #43203
• [Unified Memory] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts: #43206
• [MemCache] Skip KV events and eviction config on a disabled radix cache: #43253
• [HiCache] Keep storage workers alive on backend exceptions: #43550
• [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll: #43562
• [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking: #43567
• Share MLA host capacity initialization: #43595
• Build Mamba host pools from device buffer infos: #43596
• Capture KV host transfer buffers: #43711
• [HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction: #43868
• [Unified Memory] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode: #43919
• [Core] Keep request priority within the signed 64-bit range in radix caches: #43920

LoRA

• Fix LoRA usage-counter accounting across request lifecycle paths: #31808
• [LoRA] Support LoRA for the NemotronH_Nano_VL_V2 model family: #39798
• [Perf][LoRA] remove empty CTAs from fused LoRA expand and add split-K to shrink: #41951
• [KV Events] Publish LoRA adapter names and namespaced block hashes: #42455

Multimodal

• fix(mimo-vl): keep multimodal features when capturing aux hidden states, and correct vision preprocessing: #40888
• fix(multimodal): convert grayscale tensors to RGB in PIL backend: #42127
• Fix EVS for NemotronH_Omni_Reasoning_V3: #42933
• [Multimodal] Chunk and cache audio items whose offsets span several placeholder runs: #43301
• [Fix] Repair multimodal VMM lease cleanup and source-stream ordering: #43475

Model Support & Optimizations

• [dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding: #31773
• [Fix] Fix DeepSeek V4 Pro TP24 vocabulary-padding failure on Hopper: #31801
• [Kimi] Enable GB300 TP4 and GB200/GB300 TP16 SP collectives: #35330
• [Fix] Pass token-major Q/K tensors from Gemma-3 to RadixAttention: #37984
• [MiniMax-M3] Instantiate the decode block top-k in register buckets: #38615
• [MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection: #38841
• [Qwen3.8 CP] Collocated prefill CP integration: #39723
• [Kimi-K3] Guard optimized paths by platform: #40269
• [Model Loader] Add InstantTensor loader support (DeepSeek-R1 on 8x H200 loads in 15.3 s instead of 160 s from 50 GB/s storage): #40453
• [Fix] MiMo-V2: accept checkpoints with the split attention layout: #40978
• [Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection (2.1x to 2.3x output throughput per GPU on 8x H200): #41251
• [KDA][Kimi-K3] Speed up LTX-2.3 QK norm and split RoPE on H200: #41459
• [DeepSeek-V4.1] Fold q_rope_store into fused_q_norm_rope: #41657
• [DeepSeek-V4.1] Faster fp4 index-K gather and combine_topk_swa_indices: #41658
• [DeepSeek-V4.1] Fused c1/c2 compress for eager extend, faster c2 decode: #41660
• [Fix] Keep MiMo-V2 processor available without TorchCodec: #41667
• [DeepSeek-V4.1] Trim prefill model-entry overhead: #41704
• [Fix] Reduce the Sarvam dense MLP through its row-parallel projection: #41752
• [Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends: #41754
• [Refactor] Build seven more decoders from stage boundaries: #41779
• [Fix] Make the Solar model constructible and runnable: #41810
• Add blackwell dual gemm fusion support for MLP: #41887
• [DeepSeek-V4.1] Reduce eager Engram extend preparation: #42159
• [Perf] Optimize Qwen3.8-Flash-Next BF16 decode on Blackwell (242 to 271 tok/s on 4x B200 at concurrency 1): #42175
• Serve pplx-decider decision checkpoints on /v1/systemone: #42183
• [Fix] Validate per-layer rope_parameters on transformers 5.17 (Laguna-S/XS-2.1 nightly): #42214
• [Perf] Accelerate DiffusionGemma with FA4 and CUDA graphs on Blackwell (7.6% higher generation throughput): #42239
• [DeepSeek-V4.1] Unify mHC into one state machine and drop medium-batch fusion variants: #42245
• [Fix] Step-3.5 DeepEP routed scaling and Sarvam shared expert under dense TP1: #42302
• [Fix] EXAONE and ERNIE 4.5 VL MoE architecture, backend and PP issues: #42303
• [Fix] EXAONE MoE under DP attention and DeepEP: #42305
• [Fix] Jet-Nemotron build and Granite MoE hybrid final norm: #42306
• [Refactor] Build the Qwen2 decoders from stage boundaries: #42307
• [Refactor] Build the Llama and Nemotron-NAS decoders from stage boundaries: #42308
• [Fix] GigaChat 3.5: apply the sandwich norms to the complete sums: #42310
• [Refactor] Build the ZAYA1, IQuest-Q1 and Gemma 4 decoders from stage boundaries: #42311
• [Refactor] Build the Hunyuan V4 decoder from stage boundaries: #42477
• [Refactor] Build the Qwen4 experimental decoders from stage boundaries: #42478
• [Fix] GigaChat 3.5: build the stage boundaries once: #42480
• Serve pplx-decider-v1.1 and update the pplx-decider cookbooks: #42645
• [Model] Support Clef and Clef-Flash decision models on /v1/systemone: #42721
• [Model] Add support for EmbeddingGemma 2: #42782
• [DeepSeek-V4.1] Mega-mhc integration: #42808
• [Fix] Kimi K3 KDA value head size and SiTU input alignment: #42893
• [Refactor] Run Kimi K3 through stage boundaries: #42894
• Fix Mistral-native format detection when the Hub is unreachable: #43446
• [Fix] Keep residual rows and sums consistent across appends and pipeline handoffs: #43454
• [Model] Keep gpt-oss router logits batch-invariant under deterministic inference: #43624
• [DeepSeek-V4.1] Support decoder SWA bounded replay with CUDA DP attention: #43812
• [DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows: #43833
• [DeepSeek-V4.1] Reduce sparse-prefill workspace padding: #43921
• Reuse Hugging Face LFM2-MoE configuration: #43939

Kernel Library

• Add triton autotune on the Mamba2 SSD kernels: #39130
• [KDA] Add an opt-in decode-parity mode to the Triton multi-token recurrence: #42251
• Update AOT kernels for Torch 2.14: #42366
• [Kernel] Use explicit reciprocal scaling for recurrent Q/K normalization: #42486
• [SM90] Use native block32 DeepGEMM for DeepSeek-V4.1: #42604
• [Fix] Guard merge_state sgl_kernel import on non-CUDA platforms: #42682
• [Kernel] SwiGLU-OAI MXFP8 producer; move swizzled MXFP8 producers to ops/quantization: #42810
• [DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel: #43070
• [Mamba] Describe the track snapshot dtype by its kernel instead of fp32: #43192
• [Perf] Enable fast-math for JIT activations on Blackwell: #43236
• [GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout: #43332
• [Fix] Link cuda_ipc JIT against CUDA driver stubs dir: #43764

OpenAI-Compatible API

• [Responses] Fix max_output_tokens overestimate on multimodal requests: #35524
• [Score] Send stacked GPU embedding overrides through a CUDA IPC pool: #39730
• [Fix] Preserve inline instructions for Responses and Kimi K3 Messages: #40077
• [Frontend] Parallelize long chat prompt encoding: #41259
• [Fix] Emit Responses reasoning content-part events: #41789
• Advertise the KV-event replay endpoint in /server_info: #42274
• [Fix] Preserve empty K2 Horizon reasoning in chat responses: #42623
• Stream a chunk once stream_interval tokens are unsent: #42643
• [gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request: #42673
• [Perf] Serialize OpenAI chat/completion responses in one pass: #42889
• fix(openai): preserve zero cached tokens when cache reporting is enabled: #43005
• fix(openai): honor tool_choice=none in DeepSeek encoders: #43006
• fix(openai): keep chat stream created timestamp stable: #43026
• fix(openai): omit absent tool_calls from chat responses: #43027
• [HTTP Server] Let custom /generate routes skip the second contract check: #43114
• [Fix] Preserve HTTP errors for non-streaming /v1/messages: #43796

Rust Server

The experimental Rust router and frontend (experimental/sgl-router, sglang-processor) keep moving toward parity with the Python server.

• [SGL Router] Watch EndpointSlices for sibling router replicas (4/13): #40690
• [SGL Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13): #40691
• [SGL Router] Vet a peer's snapshot before it may touch the tree (6/13): #40692
• [SGL Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13): #40693
• [SGL Router] Sweep the fleet for a peer snapshot and hand it to the pump (8/13): #40694
• [SGL Router] Share one fleet-wide fetch across a discovery burst (9/13): #40695
• [SGL Router] Ask the fleet when a graft's splice goes unwitnessed (10/13): #40696
• [SGL Router] Gate readiness on peer bootstrap, and make it observable (13/13): #40699
• [SGL Router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb: #41268
• [SGL Router] Exclude portless prefills and retry bootstrap discovery: #41610
• [SGL Router] Allow load-only routing without a tokenizer: #41611
• [SGL Router] Fail fast and cancel decode when prefill fails: #41612
• [SGL Router] Enforce compatible PD version groups: #41613
• [Rust] Define semantic frontend contract: #41709
• [Rust] Extract shared runtime.v1 protobuf bindings: #41726
• [Rust] Implement the Rust gRPC adapter for native generation: #41766
• [Rust Renderer] sglang-processor lib: #41896
• [Rust] Wire the gRPC server into the frontend lifecycle: #41964
• [SGL Router] Route to a specific DP rank inside multi-rank workers (--dp-aware): #41974
• [SGL Router] Unify reorg session and cache affinity modes: #41990
• [SGL Router] Support engines started with --api-key (--worker-api-key): #41996
• [SGL Router] Harden peer-bootstrap edges: refuse redirects and metric docs: #41999
• [SGL Router] Drop the snapshot client's redirect-following fallback: #42010
• [SGL Router] Add native /generate endpoint: #42147
• [SGL Router] Add OpenAI /v1/embeddings: #42191
• [SGL Router] Add SGLang's /v1/rerank: #42205
• [SGL Router] Add SGLang's /v1/classify: #42258
• [SGL Router] Repair KV-event sequence gaps from the engine's replay socket: #42275
• [SGL Router] Credit routed prompts before their KV events arrive: #42276
• [SGL Router] Serve the Anthropic Messages API on /v1/messages: #42316
• [SGL Router] Move pressure and prefix signals out of legacy policies: #42428
• [SGL Router] Add reorg bucket config with per-group admission: #42429
• [SGL Router] Improved policy balanced mode: #42545
• [SGL Router] Let selection skip workers a request already failed on (retry 1/5): #42628
• [SGL Router] Make one request dispatchable more than once (retry 2/5): #42629
• [SGL Router] Retry failed dispatches on another worker (retry 3/5): #42630
• [SGL Router] Back off between retry attempts (retry 4/5): #42631
• [SGL Router] Bound streaming time-to-headers by --request-timeout-secs (retry 5/5): #42632
• [Rust Processor] Split sglang-processor into tokenizer, render and parser: #42662
• [Rust Processor] DeepSeek-V4 parity with SGLang, plus a reusable parity harness and skill: #42664
• [SGL Router] Render DeepSeek-V4 through sglang-processor: #42665
• [Rust] Update Dynamo frontend crates to latest compatible releases: #42666
• [SGL Router] Document and reject a zero request timeout: #42751
• [Rust] Upgrade sglang-radix-tree to PyO3 0.29: #42807
• [Rust Server] preserve PD routing in chat and completions: #42989
• [Rust Processor] Share the reasoning splitter and tool constraint across hosts: #42996
• [SGL Router] Serve /v1/completions through /generate, with the shared OpenAI layer: #42997
• [SGL Router] Serve /v1/chat/completions through /generate, with reasoning parsing: #42998
• [SGL Router] Parse tool calls when serving chat through /generate: #42999
• [SGL Router] Serve DeepSeek-V4.1 Flash through sglang-processor: #43000
• [Rust Server] warm up each rust listener in-process: #43004
• [Rust] Pass the media item fps hint to multimodal processors: #43143
• [SGL Router] Keep charset=utf-8 on streamed responses: #43232
• [SGL Router] Read finished /generate streams to the end; expose reorg admission caps as flags: #43233
• [Rust] Serve any model name on the OpenAI chat and completions routes: #43923
• [Rust] Let an embedding package wrap the frontend HTTP app: #43924
• [Rust] Refuse an unsupported multimodal model before the Rust listener binds: #43925

Simulator

• [Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks: #43188
• [DSpark] Apply SGLANG_SIMULATE_ACC_LEN inside the accept instead of switching paths: #43623
• [DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept: #43791

SGLang-Diffusion

• [Diffusion] fix: honor the lazy transformer loader contract: #15215
• fix(diffusion): validate output_quality and fix return type annotation: #19530
• [Diffusion] Cache LTX-2 RoPE coords to avoid per-step recompute: #22441
• Fix scheduler host not working when ipv6 in multi modal gen: #22813
• fix(zimage): align SGLD CFG combine formula with diffusers Z-Image: #23772
• [Diffusion] Enable channels_last_3d for Cosmos3 VAE: #28021
• [Diffusion] Support Ideogram TurboTime LoRA inference: #30487
• [Diffusion] attention: Allow disabling sequence masking in SP: #33855
• [Diffusion] support MiniMax H3 RL: #34365
• [Diffusion] Forward multimodal token type IDs in image encoding: #34494
• [Diffusion] cli: restore gated Diffusers model detection: #34903
• [Diffusion] Fail MiniMax-H3 quality="high" closed on substituted DiT weights: #34978
• [Diffusion] Fix width/height silently dropped on multipart /v1/videos requests: #35108
• [Diffusion] Allow per-role attention backend override: #35310
• [Diffusion] Z-Image: autotune the shared transformer block at both token scales (H100 1024^2 denoise 13.58 -> 9.58 s, 1.42x): #35755
• [Diffusion] Add configurable reference image resizing for MiniMax H3 Ref2VA: #35839
• [Diffusion][MiniMax-H3] Add community LoRA recipes and Kohya mapping: #35857
• [Diffusion] Add support for HunyuanImage-3.0-Instruct: #36311
• [Diffusion] feat: enable breakable CUDA graphs for FLUX.2 Klein: #36470
• [Diffusion] feat: add --async-output-save to overlap output saving with the next request: #37549
• [Diffusion] perf: FastH3 8-Step V2 native support, native Blackwell VSA, batched H3 VAE decode: #37662
• [Diffusion] Support LLaDA-Image BF16 generation and editing: #37907
• [Diffusion] fix: do not interleave comfy-kitchen MXFP8 scales twice: #38028
• [Diffusion] quantization: ConvRot INT8 online W8A8 for DiTs (sgl-kernel + convrot_int8_customkernel): #38040
• [Fix] MiniMax H3: honor requested inference step count: #38671
• [Diffusion] model: restore folded H3 GGUF patch embedding: #39523
• [Diffusion][MiniMax-H3] Avoid full-video clone in VAE postprocessing (drops a 4.2 GiB full-video copy, postprocessing about 21% faster): #40095
• fix: resolve QKV layout inference for MiniMax-H3 checkpoints: #40158
• [Diffusion] USP: accept strided row views in the packed QKV all-to-all: #40588
• [Diffusion] Skip warmup preferred preload when it would OOM: #40761
• fix: wait for multimodal process teardown: #40765
• [Diffusion][Model] Add native SANA-Video 2.0 support: #41492
• [Diffusion] fix: validate SageAttention 2 head sizes without overriding component policy: #41529
• [Diffusion] Deduplicate SM120 FP8 fallback warnings across layers: #41542
• [Diffusion] Respect explicit residency over pipeline preload hints: #41622
• [Diffusion] Keep generic warmup valid for step-floored models and stop reporting a failed warmup as warm: #41641
• [Diffusion] Fuse Flux3 rowwise FP8 quantization with Triton: #41671
• Fix CFG normalization to use per-sample norm instead of batch-flattened norm: #41706
• [Diffusion] kernels: stop specializing on per-request sequence lengths: #41710
• [Diffusion] perf dump: one writer per replica, no first-request NCCL setup: #41711
• [Diffusion] release the allocator cache when the server idles, not per request: #41718
• [Diffusion] Stream an oversized DiT in auto mode instead of OOMing: #41721
• [Diffusion] Keep the DiT off component offload under explicit multi-GPU FSDP: #41722
• [Diffusion] Fix the OOM hint's placement advice and add a 24 GB Wan2.1 recipe: #41723
• [Diffusion] video: let requests choose the libx264 preset: #41797
• [Diffusion] Encode the MP4 while the VAE decodes (MiniMax-H3, Wan) (Wan2.2 TI2V 5B request 6.27 to 5.83 s on one B200): #41819
• [Diffusion] H3: check the final MP4 in-process instead of spawning ffprobe: #41825
• [Diffusion] kernels: take the Wan VAE decoder's remaining first-decode work off the request path: #41830
• [Diffusion] conditioning cache: recycle warmup-only entries at the first served store: #41832
• [Diffusion] warm the image writer and the HTTP route models at startup: #41833
• [Diffusion] Action API: decode JSON pixel lists to numpy at the entrypoint: #41895
• [Diffusion] MiniMax-H3 fuse the video VAE decoder's RMSNorm and QK RoPE (10 s clip VAE decode 9.19 to 8.25 s on one H200): #41906
• [Fix][Diffusion] Fix hybrid SP+TP for I2V models: #41917
• [Diffusion][MiniMax-H3] Fix int32 offset overflow in SubBlock router kernels: #41965
• [Diffusion][MiniMax-H3] Route SubBlock sparse attention in head chunks: #41985
• fix(diffusion): load serialized H3 INT8 in ComfyUI integrated mode: #42121
• [Diffusion] FLUX 3 Action: CUDA graphs for observation encoding and denoising steps: #42171
• [Diffusion] Qwen-Image 2.1: load ComfyUI/ai-toolkit fused gate_up LoRAs and diffusers metadata alpha: #42174
• [Diffusion] fix: keep residual_gate_add on the JIT CUDA path for cont…: #42240
• [Diffusion] Fix native FP8 format handling for FLUX 3 rowwise linears: #42255
• [Diffusion] Name the quality tiers after what they guarantee: #42370
• [Diffusion] Fix TeaCache CFG state lifecycle and skip boundaries (skips 30% of DiT forwards, 1.35x on one GPU): #42422
• [Diffusion] Reset DiT cache state at the start of each request: #42437
• [Diffusion] fix: pin the JoyAI-Echo overlay's source revision: #42471
• [Diffusion] fix: load community ComfyUI-GGUF Qwen-Image-2.1 transformers: #42556
• [Diffusion] Fuse Anima AdaLN modulation and cache the FFI callable: #42608
• [Diffusion] Fuse Anima FP32 split-half Q/K RoPE (Anima denoise 2.84 to 2.25 s on one H200): #42609
• [Diffusion] model: support Kandinsky 6: #42743
• [Diffusion] Skip zero blocks when hashing large conditioning inputs: #42748
• [Diffusion] Hunyuan3D: implement paint_use_remesh (decimate to 40k faces before UV unwrap): #42770
• [Diffusion] model: support Wan-Animate-2: #42941
• [Diffusion] avoid redundant LoRA scaling while preserving exact numerics: #43028
• [Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5: #43062
• [Diffusion] Don't report video jobs completed after their outputs are deleted: #43084
• [Diffusion] perf: decode plain base64 MiniMax-H3 materials without a per-character loop: #43117
• [Diffusion] Skip LTX-2 SP masks only for globally unpadded sequences: #43134
• [Diffusion] Fix int32 offset overflow in the Ulysses packed-QKV pack kernel: #43140
• [Diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against attention over the copy engine: #43164
• [Diffusion] comfyui: report the real runtime import error: #43178
• [Diffusion] perf: tile MiniMax-H3's indexed modulation kernels by 2048 columns: #43267
• [Diffusion] fix: share the IPC all-to-all transport between AllToAll4D and USP: #43269
• [Diffusion] perf: fuse MiniMax-H3's RMSNorm and indexed AdaLN under quality lossless: #43327
• [Diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per warp, bit-identical: #43337
• [Diffusion] perf: move MiniMax-H3's pipelined Ulysses copies off the SMs: #43361
• [Diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell: #43373
• [Diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away: #43385
• [Diffusion] Support Qwen-Image 2.1 Turbo and checkpoint sampling grids: #43391
• [Diffusion] comfyui: consolidate FluxAdapter correctness fixes: #43490
• [Diffusion] comfyui: consolidate executor conditioning-cache fixes: #43492
• [Diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode: #43495
• [Diffusion] Add the dynamic LoRA delta inside the second GEMM: #43501
• [Diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode: #43504
• [Diffusion] Run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout: #43508
• [Diffusion] Pipeline every USPAttention Ulysses path's exchange over the copy engine: #43521
• [Diffusion] Bind each GPU worker to its GPU's NUMA node: #43565
• [Diffusion] BCG: warn instead of disabling on unvalidated models: #43568
• [Diffusion] Accept request-level profiling on the image API: #43573
• [Diffusion] FLUX: declare regional compile regions: #43575
• [Diffusion] Qwen-Image-2.1: skip QKV packing while layers are offloaded: #43576
• [Diffusion] torch.compile: keep Cosmos3's fused kernels in the graph: #43577
• [Diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode: #43581
• [Diffusion] IPC all-to-all: keep exported buffers out of expandable segments: #43584
• [Diffusion] torch.compile: keep Qwen-Image-2.1 and Ulysses graphs whole: #43586
• [Diffusion] Keep the overlay probe from stalling cached startup: #43590
• [Diffusion] torch.compile: skip inductor's saved multi-kernel pick (Z-Image warm starts crash or hang): #43601
• [Diffusion] torch.compile: keep the TP all-reduce inside the graph (Qwen-Image-2.1 TP2 compile 2.2 to 1.5 s on GB300 with --offload-during-compile false): #43602
• [Diffusion] torch.compile: serve whole graphs after offload-during-compile: #43631
• [Diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad: #43632
• [Diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths: #43633
• [Diffusion] MiniMax-H3: run the attention section eager under torch.compile: #43634
• [Diffusion] Encode LTX video/audio through the shared bounded MP4 path: #43656
• [Diffusion] Speed up LTX-2 guidance and video output: #43667
• [Diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder: #43683
• [Diffusion] split and batch LTX-2.3 one-stage guidance passes under CFG parallel: #43684
• [Diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP: #43691
• [Diffusion] bind worker CPUs to the GPU's NUMA node when memory policy is not permitted: #43756

Local & Desktop AI

• [Apple Silicon] Add an Apple Silicon (MPS) platform to the standard Torch model runner: #36780
• [MLX] Fix startup after logical token capacity change: #41314
• [MLX] Drain in-flight jobs before a retract-mode pause_generation: #43282
• [MLX] Drop a retracted request's runner state at its re-prefill: #43284
• [MLX] Sync decode KV to the pool once at release, clamped to the owned prefix: #43292

AMD / ROCm

• dsv4.1-amd: serve DeepSeek-V4.1 on gfx950 (DSpark up to 1.9x the throughput of plain decode on 4x MI350X; 128K-input TTFT 860 to 576 ms): #41308
• [AMD][DeepSeek-V4.1] Route low-ratio indexer and candidate-block top-k through top-k v2 (top-k 1.4x to 2.6x faster at decode shapes): #41947
• [AMD][DeepSeek-V4.1] Switch the fp8 dense GEMMs on gfx950 to aiter's MXFP8 GEMM: #41970
• [AMD][DeepSeek-V4.1] Pick the gfx950 wo_a batched gemm tile by row count: #41971
• [AMD][DeepSeek-V4.1] Fix shared-expert fusion accuracy and speed up MoE routing on ROCm (GSM8K 0.891 to 0.906 with fusion): #42011
• [AMD][DeepSeek-V4.1] Build DSpark draft metadata inside the CUDA graph on ROCm: #42014
• [AMD][DeepSeek-V4.1] OPUS sparse prefill on gfx950 through layout conversion: #42017
• [AMD][DeepSeek-V4.1] Fuse MXFP8 activation quant into producer kernels on gfx950 (per-step GPU time 9.46 to 8.88 ms on MI355X TP2 with DSpark): #42055
• [AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1: #42911
• [AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill: #43803
• [AMD][DeepSeek-V4.1] Fix DP attention + TP MoE decode NaN and support bounded replay (102.6K tok/s/GPU on AgentX at concurrency 32, 2x MI355X): #43077
• [AMD] Set maxseq length limit for DSV4.1 BCG metadata buffers: #42570
• [AMD][DeepSeek-V4.1] Fix wo_b input flatten for packed fp8 activations on gfx950: #42796
• [AMD] Add engram host/device address mapping: #42552
• [AMD] Update aiter version for v41: #42023
• [AMD] Support Quark DeepSeek-V4-MXFP4 checkpoints on the DSV4 stack, enabling shared-expert fusion and MTP (also fixes a loading bug that dropped about 25% of expert FFN channels): #31054
• [AMD] Port CP V2 to the DeepSeek-V4 HIP backend (mean TTFT 16% to 31% lower at C16 to C48, TPOT 2% to 12% higher): #34200
• [AMD][DeepSeek-V4] feat: enable PD-disagg with fp8 unified_kv on gfx950: #39166
• [AMD][DeepSeek-V4] feat: enable PCP with fp8 unified_kv on gfx950 (1.55x KV tokens, up to 15.8% higher throughput): #39923
• [AMD][MORI] Add MORI EP V2 support: #40666
• [AMD] Keep aiter's tuned GEMM for the DeepSeek-V4 compressors: #41931
• [AMD] Fix AITER weight slicing for CP decode TP: #39950
• [AMD][GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 (+14.3% geomean output throughput on AgentX, 4x MI355X): #39273
• [AMD] GLM-5.3-Flash: fuse shared expert and KDA projections on Quark MXFP4 (median ITL up to 13.6% lower): #41870
• [AMD] Let the GLM DSA NextN draft declare its own shared-expert fusion architecture: #41258
• [AMD] Eliminate remaining gfx95 FP8 scale relayout copies: #41030
• [ROCm] topk v2: split one long row across blocks, the CDNA cluster-path equivalent (median ITL 18.7% lower at 600K context): #39931
• [ROCm] GLM-5.2: gfx950 four-kernel fused DSA indexer decode path (ITL 5.3% lower): #38583
• [ROCm] GLM-5.2 decode path: decode-shaped MoE/MLA tiles, split speculative softmax, and bf16 GEMM routing (+5.7% end to end): #41725
• [ROCm] Use the fused MLA absorb + RoPE + KV-write kernel for decode-sized forward modes only: #41533
• [ROCm] Eliminate HiCache migration stalls and use exact-size host allocations (P90 interactivity +3.7% for GLM-5.2 MXFP4 on MI355X): #42553
• [AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm (+7.8% output throughput on GLM-5.2-Quark-MXFP4-AttnFP8): #28734
• [AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size: #39840
• [AMD] MiniMax-M3: fuse sparse QK norm, RoPE and cache writes with AITER (up to 6.8% higher throughput): #35357
• [AMD] Use AITER ASM prefill for MiniMax-M3 HD128 attention (+16.6% P90 interactivity, +8.4% throughput): #41707
• [AMD] Add opt-in MiniMax-M3 TP4 indexer context partitioning (2.64x faster indexer chain at 128K context): #41488
• [AMD] MiniMax-M3 indexer CP: packed scoring for EAGLE verify rows: #42614
• [AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm: #41497
• [AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path: #34540
• [AMD] gfx950 small-batch MoE: expert-count gate for the small sort and FP8 block-scale small-M kernel (about 20% better interactivity at concurrency 1 and 4): #41982
• [AMD] One-launch small-M MoE router for Qwen3.5 on gfx950: #41133
• [AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950: #41134
• [AMD] Qwen add Sparse attention to capturing hip graphs (+13% P90 interactivity on MI355X): #43383
• [AMD][Quark] Serve the Kimi-K3 MXFP4 checkpoint on ROCm (GSM8K 0.942): #40811
• [AMD][DCP] Enable aiter asm ps for kimi k3 prefill: #34438
• [AMD][DCP] add aiter asm for kimi k3 target verify: #34537
• [AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels: #43968
• [AMD][PD] enable mori transfer for PD decode DCP: #37358
• [AMD][Docker] Update MoRI to v1.2.4: #43834
• [AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears (+14% for Qwen3-8B-FP8 at concurrency 4): #41071
• [Diffusion][ROCm][Perf]: Set gfx942 AITER FMHA rounding mode to rtz instead of rtna (E2E latency down 5.0% on gfx942): #28650
• [ROCm] Select DSA indexer top-k wave size by arch (wave32 on gfx1250): #41687
• [ROCm] Skip the tilelang act_quant in the DSA indexer on gfx1250: #42747
• [AMD][Diffusion] VAE attention to the math SDPA backend on gfx1250: #42858
• [AMD][Diffusion] Route bf16 fuse_scale_shift to the native fallback on gfx1250: #42859
• [Diffusion] Default Qwen-Image 2.1 to VAE tiling on gfx1151: #41655
• [Diffusion] docs: Add gfx1151 Strix Halo serving guidance: #41670
• [AMD] fix kda decode flydsl import: #40546
• [AMD] AITER MLA DCP decode ASM path (opt-in): #40750
• [AMD] Gate flashinfer and TRT-LLM DSA paths on CUDA: #40911
• [AMD] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel: #41282
• [AMD] Resolve QSA packed-varlen decode to aiter on HIP: #41513
• [AMD][DeepSeek-V4.1] Greedy dspark draft/accept under SGLANG_SIMULATE_ACC_LEN: #41981
• [AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token): #42152
• [AMD] Skip the AITER #6042/#5967 patches on gfx1250: #42219
• [AMD] Fix RoPE cache dtype and diffusion CI failures: #42494
• [AMD] Let AITER unified verify run without host seq_lens: #42635
• [AMD] Keep MiniMax decode-score tile at 128 tokens off gfx950: #42772
• [AMD] Patch aiter's group32 MXFP8 GEMM masked-scale fix: #43319
• [ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP: #43328
• [AMD] Fix Flux2 kv-gather abort on non-contiguous K/V: #43793

NPU / Ascend

• [NPU] Add decode context parallel support for dsa models: #37787
• [NPU] Quote the arm64 Triton-ascend wheel URL in npu.Dockerfile: #43035
• fix: use dp_size instead of attn_dp_size for msprobe rank id: #43041
• [Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change: #43309
• [NPU] Bump SGLANG_KERNEL_NPU_TAG to 2026.9.0.post9: #43340

CPU / Intel / XPU

• [XPU] Support decode context parallelism (DCP) on Intel XPU: #34355
• [XPU] Add dynamic_expert_bias and track_state support: #39618
• [XPU] Misc changes to support XPU: #39817
• [XPU] Enable fused QK-norm + RoPE for Qwen3-MoE (60% less device time in the q/k-norm + RoPE stage on 4x Arc Pro B60): #40190
• [XPU] Enable ring attention on XPUAttentionBackend: #41406
• [XPU][Diffusion] Plain-torch scale-shift for inductor: #41885
• [Diffusion][XPU] FLUX.2: enable the bit-exact Triton SwiGLU on XPU: #41940
• [XPU] Upgrade XPU backend to PyTorch 2.14: #42003

MUSA / Moore Threads

• Fix MUSA detection under torch.compile fullgraph: #41065
• [MUSA] Fix fused MoE GEMV registration and torchada pin: #41444
• [Fix] Preserve available host memory for integrated MUSA devices: #43398

Security

• Harden OpenAI upload filename handling: #17270
• fix(security): pin multimodal-gen ZMQ scheduler ingress to loopback on wildcard --host: #36854
• [Fix] Require admin auth for /load_lora_adapter_from_tensors and /set_trace_level: #41322

Cookbook Updates

• DeepSeek-V4 and DeepSeek-V4.1: MI355X DeepSeek-V4 Pro PD recipes (#42757), refreshed AMD recipes and images, and fixed MoRI IO and UMBP playground options: #42847, #43033, #43962, #42512
• SemiAnalysis InferenceX AgentX: interactive launch commands for GLM-5.2, DeepSeek-V4, DeepSeek-V4.1, Qwen3.5 and Qwen3.8-Flash-Next, refreshed for B300, GB200 and GB300: #42726, #43083, #43799, #43983
• GLM-5.2: GB300 NVFP4 environment variables from the AgentX recipe, now in the agentic HiCache section; MI355X daily image and Top-K v2 for MTP: #42109, #42167, #42278, #42613
• GLM-5.3-Flash: the NVIDIA NVFP4 checkpoint, a measured MI355X MXFP4 agentic recipe, and a playground with B300 as default plus CP8, EP and PD: #41986, #43015, #43802
• Qwen3.5, Qwen3.8 and Qwen3: an MI355X FP8 agentic recipe for Qwen3.5, a GB300 long-context recipe for Qwen3.8, H200 mixed-chunk latency tradeoffs and the default attention backend on B300 for Qwen3: #41849, #43159, #43551, #43554
• Kimi-K3: Ascend NPU PD disaggregation recipes and an updated NPU feature list: #41717, #44052
• K2 Horizon: refreshed checkpoint revisions: #42130
• Clef: a new page with a selectable Clef Flash variant and accuracy results: #43235, #43349
• Wan-Animate-2: a command picker: #43380
• MiniCPM-V 4.7: a new page (runs on the dev-minicpm-v-4.7 image): #43829

Dependencies

• PyTorch 2.13.0 to 2.14.1 on CUDA, with sglang-kernel 0.4.9, sgl-deep-gemm 0.2.1.post1, sgl-deep-ep 0.1.2.post1 and torchcodec 0.17.0 (#38641); sglang-kernel 0.4.8 (#41337) and sgl-deep-gemm 0.2.2 (#42604) along the way
• Transformers 5.12.1 to 5.17.0 with tokenizers 0.22.2 to 0.23.2 (#39012), then 5.19.0: #42784
• FlashInfer 0.6.18 to 0.7.0.post1 with nvidia-cutlass-dsl 4.8.0 and quack-kernels 0.6.5 (#40709), then 0.7.1: #43494
• fastokens >= 0.3.2, now part of the all extra: #43104
• New optional extra sglang[instanttensor] (instanttensor >= 0.1.9): #40453
• Python 3.11+ is required (#43021); the CUDA image uses uv-managed Python 3.12: #42612
• Mooncake 0.3.13 to 0.3.13.post1: #41678
• Fix pip install by leaving the multimodal_gen/.claude symlink out of package data: #41703
• XPU: PyTorch 2.14.0+xpu, torchvision 0.29.0, sglang-kernel-xpu 0.3.0 to 0.4.0: #42003
• ROCm: AITER pin for gfx942 / gfx950 moves to e7d2453f (#42023) and MoRI to v1.2.4: #43834
• NPU: sgl-kernel-npu 2026.9.0.post9: #43340
• MUSA: torchada >= 0.1.90: #41444

Breaking Changes & Upgrade Notes

• SGLang requires Python 3.11 or newer. On Python 3.10, pip install sglang does not fail; it resolves to the last release that supported 3.10. Move to Python 3.11+ (for example off the Ubuntu 22.04 system Python) to get this and later releases: #43021
• --enable-dp-attention is deprecated; use --attn-dp-size N. --dp-size now counts model replicas only. The old --dp-size N --enable-dp-attention still resolves to --attn-dp-size N with a warning; combining replicas with attention DP, or passing both spellings, is rejected at startup: #41818. Under DeepSeek context parallelism, --dp-size N still means attention DP but now warns and will be rejected after 2026-12-31: #42348
• --disaggregation-decode-host-receive-threshold flips its encoding. 1 (the new default) disables host receive and 0 now means always receive on host. Configurations that passed 0 to disable it must pass 1: #43407
• ChunkCache is removed. With --disable-radix-cache, full-attention, hybrid-SWA and Mamba models run UnifiedRadixCache in disabled mode and pure-SWA models run PureSWARadixCache (#43198, #43199, #43200, #42354, #43201). Streaming sessions now start only on UnifiedRadixCache; pure-SWA, FlexKV and plugin tree caches fail at startup with a "not verified" error (#42295), and beam search rejects Mamba models with --disable-radix-cache too (#42354)
• The tc_piecewise prefill CUDA graph backend is removed. Passing --cuda-graph-backend-prefill tc_piecewise or a tc_compiler setting now raises an error; CUDA keeps breakable as the default prefill graph backend, and other platforms default to no prefill graph: #42287
• Renamed and retired environment variables. SGLANG_AITER_MLA_DCP_DECODE_BACKEND becomes SGLANG_AITER_MLA_DECODE_BACKEND and its default changes from gluon to asm (#34537); SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW becomes SGLANG_OPT_RELEASE_PREFILL_SWA (#42657); the old names keep working with a deprecation warning. SGLANG_DSPARK_FAST_SAMPLING, SGLANG_DSPARK_FOLDED_SAMPLING and SGLANG_DSPARK_OPT_FUSED_GREEDY_MARKOV are deprecated and no longer do anything (#43622, #43668), and SGLANG_TRITON_PREFILL_TRUNCATION_ALIGN_SIZE is deprecated: #43444
• OpenAI chat responses changed in three small ways. tool_choice: "none" no longer shows tool schemas to DeepSeek models (#43006), an absent tool_calls field is omitted instead of returned as null (#43027), and a stream keeps one created timestamp for its whole lifetime: #43026
• --enable-layernorm-sp is rejected with pipeline parallelism. It failed on the first prefill before: #43458
• For model and plugin authors: ModelRunner.tp_group is removed; read get_parallel().tp_group in the scope the code runs in (#41817). Unread placement attributes such as model.tp_size are gone; read get_parallel() (#41812). A model that declares stage boundaries and runs under pipeline parallelism must pass its shared stage declaration (stage_facts) to make_layers / make_pp_layers: #43460

New Contributors

• @TwinklerG made their first contribution in https://github.com/sgl-project/sglang/pull/40515
• @googs1025 made their first contribution in https://github.com/sgl-project/sglang/pull/41444
• @ajit283 made their first contribution in #35330
• @anthonyhe-bot made their first contribution in #41486
• @Manfredss made their first contribution in https://github.com/sgl-project/sglang/pull/41314
• @sohom-cs made their first contribution in #41380
• @shafeeqibraheem made their first contribution in #31808
• @stanleys12 made their first contribution in https://github.com/sgl-project/sglang/pull/41322
• @yunzshi made their first contribution in https://github.com/sgl-project/sglang/pull/39788
• @rodamani made their first contribution in #41736
• @shyeh25 made their first contribution in #39388
• @Jin-Chuan made their first contribution in #42002
• @ThomasNing made their first contribution in https://github.com/sgl-project/sglang/pull/41488
• @Tokha233 made their first contribution in https://github.com/sgl-project/sglang/pull/42255
• @livingshade made their first contribution in #37077
• @ekrim made their first contribution in #40703
• @trebladev made their first contribution in https://github.com/sgl-project/sglang/pull/41671
• @JarJarBeatyourattitude made their first contribution in #39862
• @JiaFengYu made their first contribution in #40785
• @cyhdmjzzy made their first contribution in #37261
• @aiyueqi made their first contribution in https://github.com/sgl-project/sglang/pull/40095
• @gpulost made their first contribution in #39523
• @chengshiwen made their first contribution in #22813
• @debanshd made their first contribution in https://github.com/sgl-project/sglang/pull/40158
• @reger-men made their first contribution in #33883
• @yunyang1999 made their first contribution in #32779
• @wildoranges made their first contribution in https://github.com/sgl-project/sglang/pull/36854
• @MaxMiksa made their first contribution in https://github.com/sgl-project/sglang/pull/17270
• @tiankongdeguiji made their first contribution in #42467
• @LiRunGuo made their first contribution in #40671
• @jetd1 made their first contribution in #40888
• @vvvdwbvvv made their first contribution in https://github.com/sgl-project/sglang/pull/41670
• @JayZenith made their first contribution in #19530
• @Seas0 made their first contribution in https://github.com/sgl-project/sglang/pull/41529
• @Zerohertz made their first contribution in #15215
• @TyGu888 made their first contribution in https://github.com/sgl-project/sglang/pull/38028
• @fusheng-ji made their first contribution in https://github.com/sgl-project/sglang/pull/42437
• @pingqiu made their first contribution in #42399
• @adenzhou1350 made their first contribution in https://github.com/sgl-project/sglang/pull/40765
• @Chillee made their first contribution in #42927
• @kunchengit made their first contribution in #31773
• @xuantengh made their first contribution in #39199
• @areneau-cf made their first contribution in #42721
• @murthy-l-habana made their first contribution in #36470
• @cr-gao made their first contribution in https://github.com/sgl-project/sglang/pull/41492
• @R00Kie-Liu made their first contribution in #31768
• @bookinstock made their first contribution in https://github.com/sgl-project/sglang/pull/38671
• @Raymondlol made their first contribution in #34978
• @frgossen made their first contribution in #35755
• @YitaoYuan made their first contribution in #40453
• @colleen-tml made their first contribution in #42807
• @LoCinder made their first contribution in #38217
• @SunskyXH made their first contribution in #43332
• @amd-nprotaso made their first contribution in https://github.com/sgl-project/sglang/pull/43383
• @Phoenix3334 made their first contribution in #35721
• @wanghemeng made their first contribution in #42212
• @ChengYM5685 made their first contribution in https://github.com/sgl-project/sglang/pull/40666
• @C1ves made their first contribution in #41911
• @ix-48 made their first contribution in #43854
• @jyizheng made their first contribution in #43734
• @apex-mochen made their first contribution in #42077
• @Vegetog made their first contribution in #43796
• @BingooYang made their first contribution in #39771
• @michal-shalev made their first contribution in #43723
• @yyh-sjtu made their first contribution in #42127
• @LittleNyima made their first contribution in #35839
• @kailash109 made their first contribution in #41951
• @andrewbriand made their first contribution in https://github.com/sgl-project/sglang/pull/44137
• @chuanrui made their first contribution in #39730

Full Changelog: v0.5.21...v0.5.22