github sgl-project/sglang v0.5.23
on GitHub

3 hours ago

Highlights

1007 PRs from 257 contributors.

New models

Model Type Cookbook
Muse Spark Open LLM / VLM link
Beam-1.0 LLM / VLM link
MiniMax-M3.1-Flash LLM / VLM link
MiniCPM-V 4.7 LLM / VLM link
Kev LLM / VLM link
Cosmos3-Nano-Sim (Bimanual, Transfer, Auto) Diffusion link
FastH3 Omni Diffusion link
FLUX.3 Image Klein Diffusion link
HY-Nomas Diffusion link
MOSS-Image Diffusion link

What's new

  • DeepSeek-V4.1 on GB300 gets up to 1.5x more prefill throughput under load and about 28% lower first-token latency on 32K to 64K prompts (#44403, #45059, #45107).
  • SGLang low-latency mode: the opt-in NVIDIA megakernel and AMD monokernel plugins run DeepSeek-V4.1-Flash decode as fused persistent kernels, for up to 44% faster single-user decode on GB300 and 75% on MI350X (#44945, #45450, #44353, #44621).
  • MoE models such as DeepSeek-V4.1-Flash, GLM-5.3-Flash and gpt-oss reach the first token 30% to 58% sooner on 8K prompts on H200 (#43363).
  • Paged Experts (opt-in) runs MoE models larger than GPU memory on a single consumer GPU, up to 30x faster than CPU offload (#42557, #45787).
  • SGLang now runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin for TPU.
  • You can now serve LoRA adapters on quantized MoE models (FP8 and NVFP4) with a new opt-in LoRA engine (#45892).
  • DeepSeek-style MLA models running decode context parallelism can now keep their prefix cache in HiCache file-based L3 storage (#41187).

To upgrade

uv pip install --prerelease=allow sglang==0.5.23
Platform Docker
NVIDIA (CUDA 13) docker pull lmsysorg/sglang:v0.5.23
AMD MI35x docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x
AMD MI30x docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x
Intel GPU docker pull lmsysorg/sglang:v0.5.23-xpu
Intel CPU docker pull lmsysorg/sglang:v0.5.23-xeon
Google TPU7x (TorchTPU) docker pull lmsysorg/sglang:v0.1.0-tpu

Full Release Notes

Speculative Decoding

  • [Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference: #33395
  • [Fix] observability: Correct speculative FPM timing and decode lengths: #38711
  • [Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph (about 1.5x decode throughput at batch size 1 with a single-layer MoE drafter): #40762
  • [Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation: #41169
  • [Fix] dsa: Fix ragged seed indices before MTP draft decode: #42565
  • [Spec] Fix DFLASH verify mask policy for wrapped attention backends: #43054
  • [Feat] spec: Capture aux hidden states on the target without a speculative draft: #43449
  • [Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails: #43502
  • [Fix] dsa-eagle-topk: Copy token keys and scales when relocating index cache: #44564
  • [Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree: #44673
  • [Perf] speculative: Fuse draft write-window gathering and KV translation: #44972
  • [Perf] attention: Capture translated FA4 verify metadata for DSPARK: #44973
  • [Fix] deepseek-v4.1: Start the PD decode draft SWA window at the prefill draft valid-from: #45037
  • [Fix] spec: Remove --speculative-skip-dp-mlp-sync: #45043
  • [Feat] spec: Let out-of-tree platforms provide their own speculative worker: #45144
  • [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners: #45145
  • [Fix] mamba: Skip zero-width conv pools in the MTP verify scatter: #45212
  • [Fix] deepseek-v4: Keep the draft MoE out of the target's expert placement: #45299
  • [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path: #45450
  • [Feat] HiCache: Support fused DFLASH KV in unified hybrid SWA pools: #45700
  • [Perf] gemma4: Capture single-step frozen-KV drafts (Gemma 4 E2B median TPOT 33% lower at concurrency 1 on H200): #45771
  • [Perf] dsv41: Trim per-step DSpark decode overhead: #46176
  • [Feat] spec: Support CUDA attention DP for DFlash and DFlash2: #46330
  • [Perf] [NVIDIA] kda: Narrower V tiles for the KDA spec-verify recurrence on SM107: #46465
  • [Perf] [NVIDIA] kimi-k3: Route draft/dense bf16 GEMMs to TGV and narrow the MLA output gate vector: #46468

Breakable CUDA Graph

  • [Perf] prefill-cuda-graph: Make prefill shared-read completion explicit and default-on: #37958
  • [Perf] deepseek-v4.1: Replay the breakable prefill CUDA graph at any context length: #43733
  • [CP] Automatically enable BCG for supported prefill CP configurations: #44022
  • [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage: #44131
  • [Fix] Qwen3.8next: Stop retaining per-size PLE outputs in breakable prefill CUDA graphs: #44150
  • [Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay: #44220
  • [DSv4.1] Support TRT-LLM decoder bounded replay with BCG: #44326
  • [Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph: #44930
  • [Perf] deepseek-v4.1: Run decoder replay tail layers from CUDA graphs on eager prefill steps (saturated 8K prefill throughput +33% on GB300 at TP4): #45059
  • [Perf] deepseek-v4.1: Run decoder replay tail graphs in prefill-graph steps and under DP attention (saturated prefill +37% at TP4 and +50% with DP4 attention): #45107
  • [Fix] glm-5.2: Record side-stream work right before its join to keep CUDA-graph replay on one stream: #45349
  • [Fix] [NVIDIA] deepseek-v4.1: Fix the breakable prefill CUDA graph in the late-layer tail on SM90: #45407
  • [Perf] cuda-graph: Capture FlashInfer KDA prefill with context parallelism: #45473
  • [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192: #45565
  • [Perf] dsv4: Let the decoder tail leave the prefill graph under DP attention with MegaMoE: #45581
  • [Fix] prefill-graph: Keep every DP rank eager when the replay tail captures behind rank barriers: #45881
  • [Perf] dsv4: Replay the V4.1 prefill graph past 16K contexts on SM90: #45941
  • [Fix] dp-attention: Align prefill graph buckets and decode padding across DP ranks: #46138
  • [Perf] deepep_v2: Reduce masked GEMM buffer memory during CUDA Graph capture: #46309

Attention Backends

  • [Perf] fmha_v2: use non-interleaved paged KV input for trtllm sm90/120 GQA prefill: #32272
  • [Fix] attention: Honor selected FlashAttention implementation: #33007
  • fix: FA4 paged-KV zero-size page-table registers on SM90: #35757
  • Fix overflow in legacy and v1 DSA top-k selectors: #37625
  • [Fix] dsa: Make the fused top-k exact on an overflowing threshold bin: #37941
  • [Fix] DSA: skip-topk pipeline handoff for owed outputs: #43832
  • [Fix] attention: Opt FlashMLA scheduler into dynamic shared memory: #44580
  • [Chore] attention: Remove the tokenspeed_mla backend and dependency: #44803
  • [Feat] attention: Add the MSA-v1 sparse backend and NVFP4 KV pool: #46141
  • [Fix] dsa: Preserve token-first sparse attention outputs: #46424

MoE & Expert Parallelism

  • [Fix] moe: Fix flashinfer TRT-LLM BF16 expert weight reload on refit: #33743
  • [Fix] eplb: Fix Qwen3 MoE output corruption after EPLB rebalance: #39580
  • [Fix] moe: Preserve negative routing sentinel in EP mapping: #40498
  • [Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16: #42837
  • [Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks (TTFT 30% to 58% lower for 8K prompts on seven MoE models on H200): #43363
  • [Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE: #44036
  • [Fix] moe: gated NVFP4 MoE intermediate alignment for FlashInfer TRTLLM: #44138
  • [Fix] moe: Map expert ids for layers whose runner needs local ids under EP: #44480
  • [Fix] megamoe: Respect request-pool limits in decode validation: #44816
  • [Fix] megamoe: Support EPLB rebalance on SM100: #44847
  • [Fix] moe: Count routed_scaling_factor once when top-k folds it: #44901
  • [Fix] moe: Renormalize per-rank shared-slot routing over routed experts only: #44904
  • [Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order: #45289
  • [Perf] moe: Preserve Marlin workspace and route-count budgets: #45462
  • [Perf] moe: Add H200 block-FP8 Triton configurations: #45591
  • [Fix] gguf: Preserve MoE routing, weighting and allocation budgets: #45916
  • [Fix] moe: Preserve CUTLASS routing metadata and execution budgets: #45917

Quantization

  • [Fix] quantization: Preserve BF16 NaNs in per-token FP8 quantization: #42689
  • [Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100: #44213
  • [Fix] nvfp4: Own streamed tensors while waiting for paired weights: #44500
  • [Fix] [NVIDIA] modelopt-nvfp4: Read w13 global scales in the TRT-LLM [up, gate] order: #44858
  • [Perf] mlp: Fuse dense SwiGLU with block-FP8 quantization on decode (single-request decode TPOT 1.9% lower on Qwen3-8B-FP8 on B200): #45219
  • [Fix] modelopt: Correct NVFP4 TRT-LLM clamps for untied gate/up scales: #45386
  • [Fix] [SM100] quantization: Honor deep_gemm for 32x32 block-FP8 linears: #45585
  • [Feat] quantization: Add the MiniMax-M3.1 NVFP4 MoE method: #46140

Parallelism & Disaggregation

  • [4/N] elastic-ep: Scale at DP-attention replica granularity: #33728
  • [Fix] hybrid-SSM: DP attention failures at low concurrency: #34535
  • [Feat] KVShard: PD transfer and e2e test: #38473
  • [Perf] pd: Stage decode-side KV preallocation lengths through pinned memory: #38955
  • [Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8: #39724
  • [Fix] disaggregation: Clean-up and optimize the KV checksum implementation: #39729
  • [Feat] KVShard: Support kv cache sharding for MTP: #40929
  • [Fix] epd: Accept tokens-in requests on scheduler-side transfer backends: #41376
  • [Fix] PD: retraction restores racing the in-flight decode forward: #42702
  • [Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8: #42811
  • [Feat] KVShard: Integrate GLM DSA/MTP with PD and CPU HiCache: #42973
  • [Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2: #43182
  • [Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective: #43216
  • [Feat] NIXL: Support per-GPU UCX device selection: #43482
  • [Fix] disaggregation: Reuse startup decode graphs across PD role switches: #43760
  • [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2: #43774
  • [Perf] CP: QSA Context Parallel Optimization (26.5% lower QSA prefill latency at 8K with CP4): #43792
  • [Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention: #43813
  • [Perf] KVShard: 0SM KV Cache Allgather for KV Cache Sharding: #44029
  • [PD] Send DSA indexer state only past decode_prefix_len: #44140
  • [Fix] pd: Allow runtime role switching with DP and DP attention: #44215
  • [Feat] KVShard: Add attention and indexer support for page-sharded DSA caches: #44280
  • [Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held: #44386
  • [Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques: #44387
  • [Fix] pp: Order output send/recv by rank parity to fix cross-node hang: #44453
  • [Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs: #44598
  • [Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer: #44605
  • [Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo: #44622
  • [Perf] distributed: Cache the eager launch of custom all-reduce v2 per size: #44642
  • [Perf] NIXL: prepared dlists for hybrid-model state transfer: #44740
  • [Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers: #44952
  • [Perf] deepseek-v4.1: Default decoder SWA bounded replay on for both PD roles on CUDA: #45035
  • [Perf] deepseek-v4.1: Default PD to MegaAttention, with the KV layout in the PD handshake: #45075
  • [Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning: #45338
  • [Fix] disaggregation: Keep a requeued optimistic prefill request's multimodal inputs: #45440
  • [Fix] disaggregation: Synchronize deferred decode KV release across TP ranks: #45453
  • [Perf] pd: Send prefill KV chunks without a host sync on the scheduler thread: #45492
  • [Fix] NIXL: Preserve explicit UCX network device selection: #45722
  • [Fix] disaggregation: Restart a lost prefill's heartbeat miss count: #45922
  • [Fix] disaggregation: Poison the deferred abort ack when a NIXL or Mori write fails: #45973
  • [Feat] layer-boundary: Fuse TP all-reduce, residual add and RMSNorm on the push plane: #46137
  • [Perf] dsa: Overlap CP KV gather with indexer computation: #46179
  • [Perf] pd: Apply the prefill WAR barrier after the previous result is processed: #46220
  • [Feat] dcp: Replicate a block-FP8 q_b_proj under --dcp-replicate-q-proj: #46359
  • [Fix] [NVIDIA] dist: Detect MNNVL fabric devices by NVML fabric state, not only by GB200/GB300 name (Kimi-K3 on two-node TP8/DCP8 decodes at 485 instead of 379 tok/s): #46463

Scheduler & Runtime

  • [Fix] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields: #31892
  • [Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config: #33715
  • perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell: #36810
  • [Fix] tracing: Fix OTel tracing context lost in multi-tokenizer router processes: #38211
  • [Perf] weight-sync: Allocate packed receive buffers from metadata: #39264
  • [Feat] bench: support --tokenize-prompt for generated-shared-prefix: #40900
  • [Fix] Keep rope_theta when --json-model-override-args overrides rope_scaling: #43067
  • [Feat] metrics: Expose multi-detokenizer router CPU usage: #43323
  • [Perf] Cache msgspec struct fields in the batch splitter: #43520
  • [Feat] runtime: Include FlashInfer’s default tactic in startup tuning.: #43927
  • [Feat] bench: report goodput and SLO attainment in bench_serving: #43967
  • [Feat] scheduler: Gate --prefill-decode-interval on running batch size: #44001
  • [Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted: #44023
  • [Fix] offloader: Preserve cached weights during CPU offload: #44139
  • [Perf] detokenizer: Group return IPC sends by tokenizer worker (+2.1% output throughput on DeepSeek-V4-Pro): #44288
  • [Fix] scheduler: Drop finished requests from the running batch before every admission: #44351
  • [Fix] scheduler: Count a request's held req_to_token row once at admission: #44401
  • [Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it: #44430
  • [Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing: #44568
  • [Perf] request_logger: skip per-token watermark redaction walk: #44692
  • [Feat] profiler: Add the forward pass id to detailed step span labels: #44775
  • [Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker: #44785
  • [Fix] scheduler: Exclude accepted storage-tier hits from the waiting prefill load: #45158
  • [Perf] detokenizer: Add SGLANG_BATCH_TOKENIZER_OUTPUTS to opt out of grouped return sends: #45280
  • [Feat] platforms: Let out-of-tree platforms bind group devices and custom ops: #45897
  • [Fix] weight_checker: Make the checksum device-agnostic: #45918
  • [Feat] metrics: Export prealloc_{full,swa}_token_usage for decode requests with a pending KV transfer: #45970
  • [Feat] benchmark: Add --input-len-list to bench_one_batch_server for per-request input lengths: #45982
  • [Fix] scheduler: Keep long prefill steps in the busy counter and exclude engine pauses: #46127
  • [Fix] request_logger: Skip watermark redaction when watermarking is disabled: #46207

Sampling

  • [Perf] logits: Write lm_head logits straight into the fp32 logits buffer: #45631
  • [Fix] grammar: Give grammar accept failures an error status: #46049
  • [Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel: #46167

HiCache & Radix Cache

  • [Fix] unified-memory: Bound joint reclaim and preserve cached prefixes: #39294
  • [HiCache] Add the page_unified host KV layout and the unified L3 key: #39727
  • [Feat] hicache: Expose the Mooncake L3 object-key layout for external KV routers: #40467
  • [Fix] hicache: Fail fast on a null pointer from the Mooncake host allocator: #40565
  • [Feat] unified-cache: Support out-of-tree external linkers: #40595
  • [mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache: #40857
  • [Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake): #41187
  • [Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker: #41343
  • [Fix] hicache: Key MLA Mamba state per TP rank in the file backend: #41907
  • [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward: #42701
  • [Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding: #42930
  • [Fix] VLM: harden Kimi image cache identity across preprocessing and radix hashing: #43043
  • [Fix] swa: Retain a whole window behind insertion pages: #43168
  • [Perf] unified-memory: Free SWA segments without a host sync: #43170
  • [Feat] unified-memory: Support asymmetric KV rows: #43325
  • [Feat] unified-memory: Fuse drafts with sliding-window and recurrent-state layers: #43326
  • [Fix] hicache: Keep unified memory off the direct page_first_direct copies: #43333
  • [Perf] hicache: Unify PP prefetch hit queries and buffer allocation: #43344
  • [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation: #43387
  • [Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes: #43548
  • [Feat] hicache: Export host Mamba pool occupancy next to host KV: #43727
  • [Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend: #43816
  • [Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend: #43817
  • [HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows: #43932
  • [Perf] hicache: Parallel THP-backed host pool allocation (opt-in) (host pool allocation about 4x faster, about 140 s saved per start): #43978
  • [Fix] PD: Skip the cached-prefix early send in batches that start a HiCache load-back: #44081
  • [Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction: #44097
  • [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode: #44173
  • [Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id: #44592
  • [Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll: #44614
  • [Fix] cache: Support DeepSeek-V4.1 external linker pools: #44623
  • [Fix] hicache: Order write-back host copies after the forward that produced the KV: #44800
  • [Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache: #44849
  • [Fix] cache: Accept packed draft layer mappings in the UMBP linker: #44929
  • [Perf] unified-memory: Reduce allocation and SWA retirement overhead: #44970
  • [Feat] unified_cache: Support FULL + MAMBA MLA models (Kimi-K3) in the direct external linker: #44977
  • [Fix] hicache: Count the EAGLE bigram boundary token in PD decode restores: #45076
  • [Fix] hisparse: Copy DeepSeek-V4 eager backups token by token: #45104
  • [Fix] hisparse: Keep PD destination host pages until the decode KV release: #45108
  • [Perf] unified-memory: Allocate free pages from each sub-pool's own end first: #45222
  • [Fix] mem_cache: Skip linker L3 lookup on PD decode: #45538
  • [Feat] disaggregation: Verify the KV checksum under a decode-side radix cache: #46423

LoRA

  • [KV events] Publish opaque LoRA block IDs and adapter names: #42817
  • [Fix] openai: distinguish s3 served-model-name from LoRA adapter syntax: #44450
  • [Fix] lora: Preserve MoE projection identity and routing lifetimes: #45681
  • [Feat] lora: Add plan-driven dense and MoE LoRA engines: #45892

Multimodal

  • [Fix] inkling: Return 400 instead of 500 for unfetchable/undecodable multimodal input: #40899
  • [Feat] runtime: Support tokenized multimodal generation: #42833
  • [Fix] glm4v: Clamp request video budgets to the deployment limit: #43871
  • [Perf] session: Cache multimodal prompt token counts across turns: #44112
  • [Fix] multimodal: Preserve caller hashes and offsets for precomputed inputs: #44160
  • [Fix] nemotron_vl: Preserve tokenized prompts during media expansion: #44184
  • [Fix] nemotron_vl: Pad visual spans using processor offsets: #44581
  • [Fix] multimodal: Clip pad spans to a truncated prompt: #46638

Model Support & Optimizations

  • [Fix] deepseek-v4: Fix DeepSeek-V4 non-EP TBO for attention TP > 1: #33250
  • [Fix] dLLM: Preserve prompt tails and literal mask tokens across prefix reuse: #39592
  • [Fix] dsv4: Skip DP-attention pad rows in the engram hasher and low-ratio sources: #40424
  • [Fix] dLLM: Size decode graphs to the default max running requests: #40941
  • [Fix] deepseek-v4.1: Match KV capacity to physical layouts: #41275
  • [Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends: #41303
  • [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode.: #41561
  • [Fix] deepseek-v4: Refresh attn_sink after loading weights: #41630
  • [Feat] MiniCPM-V-4.7: add model support: #41993
  • [Fix] qwen3.8: Fix batch-invariant kernels for Qwen3.8 Flash-Next: #42087
  • [Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay: #42901
  • [Perf] dsv4: Vectorize paged KV cache dequantization: #42929
  • Fuse MiMo verification RoPE and KV cache writes (E2E throughput +4.6%): #43106
  • [DSv4.1] Fold encoder-SWA replay into the extend forward: #43166
  • [Perf] gdn: Select FlashInfer prefill for page-major pools: #43171
  • [Perf] dsv41: Integrate DeepGEMM MegaGate routing: #43262
  • [Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry: #43559
  • [Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling: #43604
  • [Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo (decode throughput +11.8% with DFlash): #43639
  • Support EPLB for Nemotron-H: #43685
  • [Perf] dsa: Route the k-pool indexer top-k through the top-k v2 kernel: #43775
  • [Perf] DeepSeek-V4.1: Run the small-batch decode fusions at TP2: #43838
  • [Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV: #43922
  • [DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits: #44051
  • [Fix] glm5_next: Correct num_experts_per_tok default to 8: #44093
  • [GLM-5.3-Flash] Drop the NextN layer from the KDA state pool: #44098
  • [Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools: #44114
  • [Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode: #44210
  • [Perf] DSv4.1: Enable decoder bounded replay by default (about 28% lower first-token latency at 32K and 64K input): #44403
  • [Fix] nemotron: Preserve nested Puzzle configs: #44487
  • [Feat] nemotron: Port Nemotron 3.5 Super serving changes: #44518
  • [Perf] [NVIDIA] muse-spark-open: Fuse pre-expert RMSNorm with static-scale NVFP4 quantization: #44554
  • [DSv4.1] Fuse request-window KV copies (median E2E latency about 61% lower with encoder SWA bounded replay on a high-load replay on 4x GB300): #44555
  • [Perf] deepseek_v4: size the masked silu-mul-quant grid to the GPU, not the padded capacity: #44618
  • [Perf] DeepSeek-V4.1: Raise the SM100 swap-AB decode cap to 40 rows: #44632
  • [Feat] kernels: Add muse spark operators and collective GEMM (2/5): #44665
  • [Feat] srt: Add muse spark runtime, multimodal support and tests (3/5): #44666
  • [Fix] bailing: MoE gate dtype follows config.router_dtype: #44715
  • [Perf] DSv4.1: Skip discarded KV rows during window commit (median E2E latency 7% lower on 4x GB300): #44783
  • [Fix] engram: Avoid blocking history copies during PD admission: #44787
  • [Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size: #44837
  • [Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints: #44838
  • [Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8: #44842
  • [Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay: #44850
  • [Perf] mamba: Fuse strided state updates and replay index translation: #44971
  • [Fix] DSv4.1: Support CUDA image requests with encoder bounded replay: #45048
  • [Feat] dsv4.1: Return prompt logprobs under encoder SWA bounded replay: #45050
  • [Fix] nemotron: Correct weight-export failures and GEMV dispatch: #45082
  • [Fix] kimi-k3: Avoid vision attention child-name collisions: #45110
  • [Perf] mamba: Copy tracked decode states at HBM bandwidth in one persistent launch: #45213
  • [Feat] kernels: Beam model support: #45254
  • [Fix] inkling: Keep BF16 LM-head operands with FP32 output: #45335
  • [Feat] Inkling: render and parse chat via tml-renderers: #45340
  • [Fix] muse_spark: support unified KV memory with TRT-LLM: #45356
  • [Feat] systemone: serve Kev pointer head decision models: #45413
  • [Fix] mamba: Keep prefill rows' checkpoint tracking through mixed chunked prefill: #45435
  • [Perf] scheduler: Plan hybrid Mamba extend batches without host-device syncs: #45472
  • [Perf] [KDA] muse_spark: Run the one-token router GEMV on CuTe DSL: #45508
  • [Perf] dsv4: Restore sparse prefill for request-window rows (+9.6% throughput per GPU at 32K tokens per rank on GB300): #45573
  • [Perf] dsv41: Use tuned MXFP8 tactics at decode sizes: #45630
  • [Perf] dsv41: Mega-mHC and the wo_a einsum write MXFP8 for wqkv_a and wo_b: #45632
  • [Perf] dsv41: Fuse the q_norm MXFP8 quantize for wq_b at every row count: #45633
  • [Perf] [KDA] muse_spark: use BLOCK_M 240 for BF16 MegaMoE prefill shards: #45637
  • [Perf] QSA: Optimize the prefill indexer gather: #45752
  • [Perf] QSA: Optimize prefill indexer in long cached prefixes: #45791
  • [Perf] [NVIDIA] deepseek-v4.1: Align BF16 prefill indexer score width: #45794
  • [Perf] dsv4: Score the V4.1 index logits in bf16 on SM100: #45880
  • [Perf] dsv41: Serve token-interleaved V4.1 KV caches with sgl-flash-mla: #45893
  • [Perf] dsv4: Use the fused MXFP8 producers in eager prefill: #45903
  • [Perf] dsv4: Optionally stage V4.1 engram rows from the shared host table on the scheduler: #45921
  • [Feat] minimax: Support MiniMax-M3.1-Flash: #45925
  • [Perf] moe: Run the Marlin GPT-OSS SwiGLU through the fused Triton kernel: #46134
  • [Perf] minimax-sparse: Speed up MiniMax-M3 sparse attention on Hopper: #46135
  • [Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate: #46163
  • [Perf] dsv41: Read Engram rows from peer device shards: #46180
  • [Perf] [NVIDIA] deepseek-v4.1: Default aggregated servers to MegaAttention only when each rank holds 64 query heads (P90 interactivity up to +27% at TP4): #46263
  • [Perf] [NVIDIA] kimi-k3: FP32 warp split-K GEMM for the MoE front projection on SM107: #46464

Kernel Library

  • [Fix] attn_residual: int64 program ids in the score/combine kernels (int32 overflow at 37,451 tokens): #43438
  • [Perf] async-probe: Fuse asynchronous NaN validation into one kernel: #43720
  • [Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins: #44945
  • [Perf] qknorm: run head_dim-128 rows two per warp with 16B lanes, bit-identical: #45214
  • [Fix] [NVIDIA] jit: Build sm_100f on CC 10.7 until the bundled CUTLASS supports SM107: #45911

OpenAI-Compatible API

  • [Feat] parser: Enrich chat_parsing streaming events: #40478
  • [Feat] parser: Add checkpoint-driven response-template adapters: #40479
  • [Fix] glm47: Disable tool grammar for tool_choice="none": #41233
  • [Fix] parser: Accept whole-number floats for integer tool arguments: #43941
  • [Fix] chat-template: Pass return_dict=False to apply_chat_template(tokenize=True) for transformers 5: #44020
  • [Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams: #44537
  • [Feat] http_server: Make the multi-worker ASGI app path configurable: #44583
  • [Fix] metrics: record HTTP request metrics in tokenizer workers: #44663
  • [Fix] muse: Support Responses namespace tools and defer_loading: #44685
  • [Fix] Inkling: frame constrained output in native message tokens: #44716
  • [Fix] dsv41: Preserve tool-result image order and current-turn thinking: #44751
  • [Perf] function_call: stop rescanning unclosed GLM tool calls per chunk: #45143
  • [Feat] glm-5.3: Render GLM-5.3 and GLM-5.3-Flash chat prompts with a Python chat template (prompt rendering 4.7x to 17x faster): #45154
  • [Feat] entrypoints: add a chat encoding adapter registry: #45300
  • [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe: #45315
  • [Fix] http_server: Make SageMaker /ping reflect server readiness: #45445
  • [Feat] srt: Balance uvicorn tokenizer workers with per-worker SO_REUSEPORT listeners: #45482
  • [Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections: #45760
  • [Feat] openai: parse DeepSeek output through deepseek-recipe: #45907
  • [Fix] minimax-m3: Type nested tool-call arguments and count self-opened reasoning: #46133
  • [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe: #46219

Rust Server

  • [Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1: #42118
  • [Fix] sglang-server: Keep the matched stop token in output_ids: #42637
  • [Feat] sgl-router: Serve native gRPC ChatComplete: #42674
  • [Feat] sgl-router: Serve the remaining native gRPC inference RPCs: #42681
  • [Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys: #42818
  • [Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware: #43511
  • [Feat] sgl-router: Render Kimi-K3 through sglang-processor: #43608
  • [sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor: #43625
  • [Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor: #43689
  • [Rust] Serve /weights_checker from the Rust frontend: #43926
  • [Rust] Serve scheduler metrics on the Rust frontend's /metrics: #43942
  • [Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware: #44072
  • [Fix] sgl-router: Resend on a closed pooled connection; add --disable-circuit-breaker: #44075
  • [Fix] sglang-server: Honour extra_key on the Rust frontend's /generate: #44207
  • [Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers: #44218
  • [Fix] sglang-processor: Follow Python's OpenAI layer through #43027, #37577, #43006, #43026, #43005: #44243
  • [Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models: #44252
  • [Fix] sgl-router: Receive kv-indexer-bridge event batches outside tokio's coop budget: #44305
  • [Feat] rust-server: Apply the x-override-* request headers: #44312
  • [Feat] rust-server: Attach configured metadata headers to a healthy /health_generate: #44315
  • [Feat] sgl-router: Add --decode-dispatch-limit to pick decode DP ranks near KV arrival: #44402
  • [Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info: #44531
  • [Feat] sglang-server: serve /v1/loads from the Rust frontend: #44532
  • [Fix] rust-server: Honour cache_salt and require_reasoning on /generate: #44533
  • [Feat] rust-server: Add HiCache storage management endpoints: #44726
  • [Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend: #44764
  • [Feat] sglang-server: Serve /flush_cache from the Rust frontend: #44766
  • [Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend: #44767
  • [Fix] sgl-router: Keep the kv-indexer connection through bursts of cancelled prefix queries: #45100
  • [Feat] sgl-router: Add --pd-stream-after-prefill: #45102
  • [Feat] sgl-router: Route prefill buckets by uncached tokens: #45103
  • [Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit: #45120
  • [Feat] rust-server: Let the launcher set the default max_new_tokens for /generate: #45431
  • [Fix] rust-server: Drop the fixed 64-item multimodal cap in favor of --limit-mm-data-per-request: #45432
  • [Feat] rust-server: Render, tokenize and serve chat through sglang-processor: #45449
  • [Fix] rust-server: Answer health probes a busy scheduler skipped on the response ring: #45475
  • [Fix] rust-server: Match Python's chat layer on templates, streaming and lowered-chat errors: #45659
  • [Fix] sgl-router: Do not abort forwarded requests on reqwest's default 30 s TCP_USER_TIMEOUT: #45696
  • [Feat] rust-server: Serve /abort_request from the Rust frontend: #45888
  • [Fix] rust-server: Deliver scheduler-side aborts to Rust frontend clients: #45899
  • [Feat] rust-server: Let a multimodal processor finalize item layout after caller hashes: #45914
  • [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill: #46066
  • [Fix] rust-server: Start a transfer backend's own KV bootstrap server on prefill: #46074
  • [Feat] rust-server: Report weight versions and media token counts in generate meta_info: #46205
  • [Feat] rust-server: Let a multimodal processor check and fetch media during admission: #46206
  • [Feat] rust: Render DeepSeek-V4/V4.1 prompts with deepseek-recipe in sglang-processor and sgl-router: #46243
  • [Feat] sgl-router: Confirmed-first pending prefixes for cold prompts, with a follower cap: #46505
  • [Feat] rust_server: Run --sidecar once per node: #46635
  • [Feat] grpc: Advertise media inputs in GetServerInfo: #46882

Simulator

  • [Feat] simulator: Add baseline collection and ML latency predictor training tools: #44299

SGLang-Diffusion

  • [Perf] diffusion: fuse Wan VAE DupUp3D shortcut add: #32860
  • [Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention: #34385
  • [Perf] diffusion: Accumulate chunked LoRA merges with addmm (1.7x faster chunked LoRA merges (geometric mean)): #37620
  • [Diffusion] Fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel: #38650
  • [Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview: #39475
  • [Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach: #40171
  • [Diffusion] Qwen-Image 2.1 VAE decoder fast paths: bit-exact norm/bias fusions by default, channels_last decode under quality=extra-high: #41340
  • [Perf] diffusion: fused QKV norm + RoPE, cached RoPE tables for FLUX 3 video VAE: #41894
  • [Perf] flux2: Build the single-block output projection without a cat: #41943
  • [Perf] diffusion: fuse FLUX3 SwiGLU with rowwise FP8 quantization: #42758
  • [Diffusion] Bound frame_interpolation_exp to RIFE's supported range: #43014
  • [Diffusion] comfyui: consolidate 6 unrelated plugin fixes: #43493
  • [Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin: #43536
  • [Feat] comfyui: video-to-video input and prompt enhancement for server-mode nodes: #43673
  • [Feat] comfyui: Apply ControlNet residuals in Flux integrated mode: #43698
  • [Diffusion] cache-dit: respect CFG parallel for dual-transformer models: #43704
  • [Fix] diffusion: keep ZMQ ingress sockets on their configured port: #43732
  • [Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions: #43768
  • [Feat] VSA-H3: support sm80 (A100) with a raw-pointer Triton load path (FastH3 on A100 in 59.6 s instead of 85.0 s): #43823
  • [Diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight: #44058
  • [Diffusion] Z-Image: replace padded rows without a host sync: #44069
  • [Feat] wan-animate-2: Support the distilled model: #44085
  • [Fix] wan-animate-2: Size the in-context slot from both input grids: #44090
  • [Perf] diffusion: convrot_int8 comfy_kitchen issues one GEMM per layer by default: #44179
  • [Perf] MiniMax-H3: fused indexed modulation for FP32 AdaLN tables: #44180
  • [Perf] diffusion: fused gate/up + SwiGLU MLP on comfy_kitchen INT8 for Qwen-Image 2.1: #44196
  • [Fix] diffusion: Invalidate merged LoRA cache after adapter rewrites: #44204
  • [Fix] flux: use true_cfg_scale as the FLUX CFG weight: #44260
  • [Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output: #44275
  • [Fix] diffusion: Fix VMoBA chunk selection bugs: #44286
  • [Feat] diffusion: Add support for the MOSS-Image family: #44325
  • [Feat] diffusion: support FLUX 3 Image Klein (peak memory 54.3 to 29.5 GiB with a 2048x2048 reference on GB300): #44418
  • [Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning: #44493
  • [Feat] diffusion: Add opt-in pi0.5 completion-boundary action batching: #44634
  • [Perf] diffusion: Cache the fused linear-GELU verdict per linear: #44643
  • [Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol: #44662
  • [Perf] [NVIDIA] diffusion/FLUX.3: Run dynamic LoRA through PrimsTS GEMMs: #44761
  • [Fix] diffusion: Unmerge LoRA layers the next adapter does not cover: #44770
  • [Feat] diffusion: Add worker-side ComfyUI compatibility patches: #44841
  • [Feat] ComfyUI: Add online INT8 quantization for BF16 Qwen-Image-2.1 single files (about 1.4x faster than BF16 on L40S with half the DiT weight memory): #44865
  • [Fix] diffusion: runtime utils bug fixes: #44887
  • [Fix] diffusion: Prevent int32 overflow in Qwen-Image-2.1-Turbo VAE RMSNorm: #45118
  • [Perf] diffusion/minimax-h3: Batch the video VAE decoder tiles and fuse the output chain: #45179
  • [Feat] diffusion/disaggregation: Split FLUX.3 prompt and image encoders: #45309
  • [Perf] cosmos3: optimize v2v preprocess performance: #45398
  • [Feat] diffusion: Support FastH3 Omni (Ref2VA 8-step PDD): #45530
  • [Perf] diffusion/preprocess: Extend exact GPU input processing across pipelines: #45733
  • [Feat] diffusion/openai: Report per-step progress for async video jobs: #45919
  • [Fix] diffusion/ltx: Keep tiled VAE decode off the sharded convs under parallel decode (E2E 4.7% lower and 1.7 GB less memory per rank at 768x512): #45942
  • [Fix] diffusion: Keep dynamic LoRA deltas on fused linear-GELU sites: #45943
  • [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses: #45965
  • [Perf] diffusion: Cut LTX-2 denoising-step host overhead: #45966
  • [Perf] diffusion: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1: #45967
  • [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving: #45977
  • [Perf] diffusion: Keep every component resident in auto mode when memory allows: #45978
  • [Perf] diffusion: Serve image responses from in-memory encoded bytes: #45987
  • [Fix] diffusion: Step MOVA latents outside the serial-CFG branch: #45993
  • [Fix] diffusion: Give every SP rank the same Wan TI2V timestep: #45994
  • [Fix] diffusion: Offset LTX-2.3 BCG video RoPE by the SP shard start: #45995
  • [Fix] diffusion: Fail closed after local worker loss or multi-rank OOM: #46025
  • [Fix] diffusion/zimage: Stop serving a previous request's RoPE tables: #46044
  • [Perf] diffusion: Keep shared denoising paths off the host and add host-sync test guards: #46045
  • [Perf] diffusion: Remove per-step host syncs and uploads in image models: #46046
  • [Perf] diffusion: Remove per-step host syncs and uploads in video models: #46047
  • [Fix] diffusion/server: Clean up rejected and cancelled video requests: #46057
  • [Fix] diffusion/api: Publish image inputs atomically: #46060
  • [Fix] diffusion/storage: Join uploads before request cleanup: #46068
  • [Fix] diffusion/mesh: Guard asynchronous job publication and defaults: #46081
  • [Fix] diffusion/loader: Preserve checkpoints on read failures: #46095
  • [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters: #46099
  • [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps: #46101
  • [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA: #46102
  • [Fix] diffusion/disaggregation: Keep decoded images in tensor transport: #46104
  • [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop: #46105
  • [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality: #46106
  • [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell: #46107
  • [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT: #46108
  • [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma: #46109
  • [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate): #46110
  • [Perf] diffusion: Vectorize the LTX-2 Q/K RMSNorm + split-RoPE kernel: #46111
  • [Fix] diffusion/disaggregation: Isolate malformed request and result messages: #46120
  • [Feat] diffusion: add Hy-Nomas offline and realtime support: #46124
  • [Fix] diffusion/server: Clean up workers on startup failure: #46142
  • [Fix] diffusion/disagg: Recompute automatic topology per role: #46144
  • [Perf] diffusion: Prefer FA4 over cuDNN SDPA for long self-attention on B300: #46146
  • [Perf] diffusion: Speed up Wan's spatial-parallel VAE decode: #46149
  • [Fix] diffusion/openai: Bound job lists and complete cursor pagination: #46156
  • [Fix] diffusion/scheduler: Isolate invalid IPC requests: #46157
  • [Fix] diffusion/api: Do not publish failed generation outputs: #46162
  • [Fix] diffusion/http: Allow bounded graceful shutdown: #46169
  • [Perf] diffusion: Encode Cosmos3 video while it decodes: #46172
  • [Perf] diffusion: Run Cosmos3's multi-GPU VAE decode on the lossless fast path: #46173
  • [Perf] diffusion: Build spatial-parallel causal conv inputs in one pass: #46174
  • [Perf] diffusion: Walk several row chunks per warp in the CuTe-DSL norm kernels: #46227
  • [Perf] diffusion: Fuse Wan's q/k RMSNorm with its interleaved RoPE: #46228
  • [Perf] diffusion: Split LTX tiled VAE decode across the decode-parallel ranks: #46237
  • [Perf] diffusion: Validate breakable CUDA graphs for LTX-2.5 two-stage serving: #46239
  • [Perf] diffusion/worker: Move cyclic GC out of request execution: #46493
  • [Fix] diffusion: Resolve cached named weights without the Hub API: #46679

Local & Desktop AI

  • [Feat] paged-experts: Serve MoE models whose weights do not fit VRAM (12x to 30x faster single-stream decode than --cpu-offload-gb on a 16 GB card): #42557
  • [Perf] [NVIDIA] fp8_fa_sm120: Use the block-scaled MMA encoding (1.47x faster at the kernel level): #43390
  • [Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM: #43556
  • [Feat] paged-experts: AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8 experts: #45787
  • [Fix] mlx: Accept draft_parallel_layout in the stub's alloc_memory_pool: #46441

AMD / ROCm

  • [Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation: #27702
  • [Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) (long-context KV transfer 68% to 72% smaller): #30929
  • [Fix] [AMD] custom-allreduce: Stop registering oversized inputs in the deterministic all-reduce: #34543
  • [Fix] [AMD] dsa: Fix AITER prefill with an FP8 KV cache: #39083
  • [Perf] [AMD] quark: Add opt-in AITER ASM FP4 GEMM for Quark dense linear: #39598
  • [Perf] [AMD] dsv4: fp8 target lm_head GEMM for DSPARK verify logits: #39622
  • [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader: #40776
  • [Perf] [AMD] attention: Spread uneven Triton decode batches across XCDs: #41229
  • [Fix] [AMD] mamba: Keep a chunked prefill cut on the mamba chunk grid so DCP can donate checkpoints: #41701
  • [Perf] [AMD] quark/Qwen3.8: Quantize the BF16 shared expert of Quark MXFP4 checkpoints to MXFP4 while loading for shared-expert fusion (+4.5% output throughput): #41899
  • [Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8): #41903
  • [AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 (opt-in with SGLANG_USE_ROCM_AITER_TGEMM=1; Wan2.2 denoising 3.55x faster on MI450): #41988
  • [Perf] [AMD] qwen3.5: Fuse activation quant into small-M FP8 block-scale GEMM on gfx950 (Qwen3.5 interactivity 5% to 10% higher at TP8): #42229
  • [Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA: #42716
  • [Perf] speculative: Parallelize EAGLE spec-decode KV index building over token blocks (flashinfer draft decode; verify/draft-extend for flashinfer, flashinfer_mla, aiter): #42803
  • [Fix] [AMD] aiter: Fix garbage decode output for non-sliding-window models on the vectorized_5d KV layout: #43335
  • [AMD] Use the preallocated EAGLE topk=1 draft chain on HIP: #43640
  • [Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP: #43642
  • [Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk (GLM-5.3 MXFP4 P50 interactivity +6.8%): #43643
  • [Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph: #43770
  • [Fix] [AMD] deterministic: Refuse RCCL all-reduce in deterministic mode above TP2: #43809
  • [Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup: #43952
  • [Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel: #43991
  • [Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8): #44003
  • [Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) (decode step about 4% lower): #44004
  • [Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8): #44006
  • [Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8): #44007
  • [Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) (TP8 decode step 3.1% to 3.9% lower): #44016
  • [Fix] [AMD] activation: Run ReLU2 natively on ROCm: #44198
  • [Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path (Qwen3-VL GSM8K 0.005 to 0.942): #44266
  • [Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode (TPOT up to 10.5% lower): #44267
  • [Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins (DeepSeek-V4.1-Flash single-request decode +75% with DSpark): #44353
  • [Perf] [AMD] dspark: No per-step host sync in decode (1/4) (DeepSeek-V4.1-Flash DSpark TPOT 6.8% lower at concurrency 1): #44405
  • [Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index: #44412
  • [Perf] [AMD] monokernels: Let a plugin skip per-step attention metadata: #44601
  • [Fix] monokernels: Honor resolved CUDA graph configuration: #44630
  • [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance: #44657
  • [Fix] [AMD] diffusion: Keep gfx1250 FlyDSL BF16 attention out of torch.compile: #44671
  • [Fix] [AMD] diffusion: Accept residual and skip_bias in the ROCm Conv3d-to-Conv2d VAE patch: #44672
  • [Perf] [AMD] deepseek-v4: Skip top-k refinement for proven positive-zero ties (P90 interactivity +8.2%): #44927
  • [Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm: #44934
  • [Perf] [AMD] deterministic: Faster extend attention launch on gfx950 (deterministic-mode prefill 21% to 130% faster on long prompts): #45098
  • [Perf] [AMD] attention: Restore the gfx950 extend tile for head_dim <= 128 (+28% throughput at 32K input with the Triton backend): #45099
  • [Perf] [AMD] dsv41: run gfx950 MXFP8 dense GEMMs on aiter's preshuffled FlyDSL kernel (DeepSeek-V4.1-Flash decode step up to 4.8% lower on MI355X): #45123
  • [Perf] [AMD] dsv41: faster mHC boundary Triton tile for 96+ rows, bitwise unchanged: #45124
  • [Fix] [AMD] grok: accept allow_patterns_overrides in presharded _prepare_weights patch: #45217
  • [Fix] [AMD] diffusion: Enable --quantization fp8 on gfx1250 (torch.compile lowering + per-channel weight scales): #45304
  • [Perf] [AMD] dsv41: Skip unneeded layer 20 computation under decoder-bounded-replay: #45339
  • [Fix] [AMD] gpt-oss: support EP for gfx1250 W4A8 MXFP4 MoE: #45365
  • [Fix] [AMD] dsv41: Set moe act to fp8 by default: #45480
  • [Fix] [AMD] attention: Pass the layer softcap to aiter unified-attention decode: #45531
  • [Perf] [AMD] kda: Use a HIP chunked-prefill kernel for KDA on gfx950 (Kimi-K3 single-request TTFT 11.9% lower at 1K input): #45710
  • [Perf] [AMD] kda: Use a HIP fused decode kernel by default for KDA (Kimi-K3) on gfx950: #45713
  • [Perf] [AMD] Qwen3-Next: fused QK Gemma RMSNorm + partial RoPE on gfx950: #45739
  • [Perf] [AMD] GPT-OSS: Pick the gfx1250 W4A8 MoE backend, in-kernel GEMM 1 gather (opt-in with SGLANG_GFX1250_W4A8_KERNEL_GATHER=1; gpt-oss-120b W4A8 output throughput +30%, TTFT 40% lower): #45758
  • [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention: #45838
  • [Fix] [AMD] rotary: Pass a query-dtype cos_sin_cache to the HIP fallback: #46161
  • [Fix] [AMD] MiniMax-M3: default SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ to 1: #46340
  • [Perf] [AMD] diffusion: Route large BF16 linears to the FlyDSL a16w16 GEMM on gfx1250 (FLUX.2-dev 1.21x to 1.28x faster end to end on MI450): #46366
  • [Fix] [AMD] aiter: Fix garbage output with an fp8 KV cache and calibrated KV scales: #46528

NPU / Ascend

  • [NPU] [Fix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism: #29753
  • [Fix] [NPU] qwen3.5: Fix dense serving on Ascend 950: #32745
  • [Feat] [NPU] llguidance: add bitmask path for Ascend: #36074
  • [Perf] [NPU] attn: add not needs_cpu_seq_lens for dsa modules: #38461
  • [Fix] [NPU] scope: fix dtype error in sampling backend ascend: #39609
  • [Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback: #40131
  • [Fix] [NPU] glm5.2: support pcp v2 with interleave & zigzag: #40165
  • [Fix] [NPU] dspark: Fix attn_tp > 1 draft wo_b all_reduce group: #40399
  • [Fix] [NPU] requantization: gate arch35 block-FP8 -> MXFP8 requantization: #40435
  • [Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP: #40516
  • [Perf] [NPU] deepep: enable MXFP8 low-latency dispatch for Fp8MoEMethod FP4 experts (MiMo-V2.5-Pro with DFlash mean TTFT 21.8% lower): #40519
  • [Perf] [NPU] mem_cache: Add optional ScatterPa KV cache writes: #40535
  • [Fix] [NPU] qwen: adapt pcp-v2: #41564
  • [Fix] [NPU] glm5.2-mtp: L2 cache adaptation: #41642
  • [Fix] [NPU] mla-pool: DCP allocator page and index-K bound, short-pool warning: #42976
  • [Fix] [NPU] pre_warm_nccl: init npu backend before the parallel runtime touches a device: #43039
  • [Perf] [NPU] glm5.2: Support dynamic INT8 MLAProlog for attention (decode latency 8.4% lower): #43469
  • [Fix] [NPU] disaggregation: Fix host-pool backup for decode retraction: #43814
  • [Fix] [NPU] kimi-k3: skip fused gmm1 for situ activation in Ascend runner: #43837
  • [NPU][Fix] use CANN libdevice for DSpark Markov sampling: #43849
  • [Perf] [NPU] mem_cache: Increase GLM-5.2-W4A8C8 KV cache capacity with compact DSA indexer allocation: #44436
  • [Perf] [NPU] scheduler: Remove blocking seq_lens D2H from graph replay: #45170
  • [Perf] [NPU] AscendAttnBackend: D2H-free block-table gather for mirrorless DSA graph replay: #45949

CPU / Intel / XPU

  • [Feat] [Intel] hicache: Enable L1<->L2 support on XPU: #32503
  • DSPARK enable for XPU: #33739
  • [CPU]: Add group-aware CPU SHM collective kernels: #35435
  • [Feat] [CPU] model: Support LTX and MOVA models on CPU: #36632
  • [Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all: #39989
  • [Perf] [XPU] fp8: Optimize FP8 scale layouts: #40052
  • [Fix] [CPU] memory: Honor cgroup memory.max when sizing CPU memory: #40066
  • [Feat] [Intel] disaggregation: Support XPU for staging-buffer KV transfer: #40861
  • [Fix] [XPU] runtime: Fix XPU rotary embeddings and tokenizer padding: #41213
  • [Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel: #41401
  • [Perf] [XPU] flux2: Fuse bit-exact LayerNorm and gated residual norm: #41942
  • [Perf] [XPU] diffusion: Avoid QKV copies in flash attention: #41944
  • [XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba…: #42122
  • [Feat] [CPU] Grok-model: enable grok-2 model on xeon cpu: #43341
  • [Perf] [XPU] server_args: Enable fused QK-norm + RoPE by default: #43345
  • [Feat] platform: Add initial T-HEAD SAIL platform support [2/N]: #43826
  • [Fix] [XPU] gemma3: Enable bidirectional image attention: #44225
  • [Feat] [CPU] model: Support Muse models on CPU: #44624
  • [Feat] [CPU] model: Support Cosmos3 series models on CPU: #45478
  • [Fix] [CPU] memory: budget overestimation caused by filtering empty NUMA nodes: #45516

Security

  • [Fix] model_loader: Avoid unsafe pickle fallback when loading PyTorch checkpoints: #42760
  • [Fix] diffusion: OpenAI API hardening: #43585
  • [Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler: #44440
  • [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349: #44509
  • [Fix] srt: Admit only allowlisted peers on TCP-bound ZMQ IPC sockets: #45250
  • [Fix] srt: Pin the source address of TCP ZMQ connects for the peer allowlist: #45437
  • [Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories: #45526
  • [Fix] sgl-router: Bump tokio for RUSTSEC-2026-0386: #46260

Cookbook Updates

  • DeepSeek-V4 and DeepSeek-V4.1: a DeepSeek-V4.1-Flash 1017 variant, PD with DSpark, refreshed AMD recipes for V4 and V4.1 (including the PD image and c1 capacity fix), and Ascend A3 / 950 single-node recipes: #44937, #44848, #44368, #44454, #44903, #45070, #45297, #45693, #45602
  • SemiAnalysis InferenceX AgentX: launches for Muse Spark Open, GLM-5.3 and MiniMax-M3.1-Flash, and refreshed points for GLM-5.2, Qwen3.5, Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on B300, GB200 and GB300: #44065, #44228, #44654, #44712, #45039, #45111, #45311, #45495, #45513, #45749, #46005, #46506, #46547
  • Muse Spark Open: a new page with GB300 Dynamo deployments, an MI355X AgentX recipe and tuned GB300 BF16 recipes: #44527, #45629, #45689, #45422, #45429
  • MiniMax-M3.1: a new page with GB200 recipes and a B200 low-latency monokernel recipe: #45924, #46546
  • Beam 1.0: a new page with Python and Docker commands: #44694, #45991
  • MiniCPM-V 4.7: DSpark benchmarks refreshed at the final model PR: #44310
  • FLUX.3 Image Klein: an expanded model description and a prompt upsampler section: #44515, #44512
  • Cosmos3 and diffusion guides: prompt enhancer tasks, Cosmos recipe links, reference resizing, LoRA auto mode and image profiling: #45705, #44377
  • GLM-5.2 and GLM-5.3: Ascend deployment recipes for GLM-5.2, GLM-5.3 and GLM-5.3-Flash, and a TPU7x recipe for GLM-5.3-Flash: #43372, #45597, #45642, #45326
  • Qwen3.5, Qwen3.6 and Qwen3.8: Qwen3.5 MI355X FP8 agentic tuning with HiCache write-through, an RTX Spark Qwen3.6 NVFP4 recipe, Ascend A3 scenarios for Qwen3.6, FP8 KV and the QSA indexer by default for Qwen3.8-Flash-Next, and the released Qwen3.8-Max W4A8 checkpoint on Ascend: #46017, #46032, #44707, #38678, #45245, #45574
  • Jais 3: a new page (runs on the K2 Horizon architecture): #42766
  • Ornith-1.5: a new page (runs on the Qwen3.5 architectures): #45455
  • New pages for models that run on preview images in this release: Ling-3.1-flash (#44301, #44307), Nemotron 3.5 Super VL (#44510, #44587), Step 5 Preview (#44357) and Recursive Flint: #45625
  • TPU: the README and TPU guide introduce SGLang-TorchTPU next to SGL-JAX: #45246
  • Ascend NPU docs move to CANN 9.1.0: #44177, #44432, #44695

Dependencies

  • FlashInfer 0.7.1 to 0.7.2: #44591
  • sglang-kernel 0.4.9 to 0.4.10 (#43922), sgl-deep-gemm 0.2.2 to 0.2.3 (#44666) and then 0.2.3.post2 (#45933), and the new sgl-flash-mla 1.0.0: #45893
  • New dependencies: deepseek-recipe 0.2.0 for DeepSeek-V4 / V4.1 chat (#45315) and tml-renderers 0.1.0 for Inkling chat: #45340
  • tokenspeed_mla is no longer installed (#44803) and cuda-tile is dropped: #44393
  • OpenAI SDK 2.6.1 to 2.8.0 (#45291), uvicorn >= 0.51 (#45482), timm 1.0.16 to 1.0.25 (#46620), python-dateutil < 3 to keep botocore resolvable (#45857), and sgl-eval 0.1.3: #44131
  • Docker: CUDA images ship the sgl-nvidia-mega megakernel plugin, inert unless --enable-nvidia-megakernel is set (#44945, #46030); AWS EFA image variants (#41006) and numactl in runtime images for SGLANG_NUMA_BIND_V2: #43883
  • ROCm: a ROCm 10.1 image for gfx1250 alongside ROCm 10.0 (#42699), the sgl-amd-mono monokernel plugin in the ROCm image (#44621, #45389), AITER pins 2bb549102 (#44379) and 285579b62 for gfx942 / gfx950 (#45498), and Mooncake v0.3.14-rc1 for cross-node KV transfer: #44404
  • XPU: oneAPI 2026.1.3 and sglang-kernel-xpu 0.4.0 to 0.5.0: #43989
  • MLX: mlx >= 0.32.4: #44367
  • T-HEAD SAIL: a new srt_sail extra (in pyproject_other.toml) for the initial platform support: #43826

Breaking Changes & Upgrade Notes

  • DeepSeek-V4 and V4.1 chat now follows deepseek-recipe. Prompts, outputs and the native /separate_reasoning and /parse_function_call endpoints go through DeepSeek's official encoder, and SGLang's own encoding_dsv4.py / encoding_dsv41.py are gone. Clients may notice: streamed tool-call arguments arrive in pieces (the first chunk carries arguments: ""), so concatenate them as the OpenAI SDK does; tool-call ids use the official call_00_... format; V4 reasoning effort follows DeepSeek's official profile and the dsv4_reasoning_effort_profile override is removed; a numeric reasoning_effort on V4 returns 400; developer messages render as system messages; /parse_function_call also returns calls to tools outside tools with tool_index -1; requests that combine tools with continue_final_message return 400. Output parsing goes through deepseek-recipe when the DeepSeek reasoning or tool-call parser is set; sgl-router and sglang-server keep their Rust parsers for now: #45315, #45907, #46219
  • DeepSeek-V4.1 decoder SWA bounded replay is on by default for supported CUDA configurations, including both PD roles. It is not numerically identical to full prefill. Pass --no-enable-decoder-swa-bounded-replay to keep full prefill, for example for RL rollouts whose logprobs must match the trainer: #44403, #45035
  • DeepSeek-V4.1 defaults to MegaAttention under PD. On SM100 / SM103 both PD roles now resolve flashmla_mega and the PD handshake checks the KV layout, so prefill and decode must run the same attention backend (#45075). Upgrade prefill and decode together, since an older decode cannot read the new handshake field; pass --dsv4-attn-backend flashmla on both roles to keep FlashMLA. Aggregated servers keep FlashMLA when attention is tensor-parallel across ranks: #46263
  • The tokenspeed_mla attention backend is removed. tokenspeed_mla still starts as a deprecated alias for cutedsl_mla with a warning. To switch, pass --attention-backend cutedsl_mla --kv-cache-dtype fp8_e4m3, plus --speculative-attention-mode decode with speculative decoding: #44803
  • --speculative-skip-dp-mlp-sync is removed. Launch commands that pass it now fail with an unrecognized-argument error; drop the flag: #45043
  • The kitchen_int8 diffusion quantization name is removed. Use convrot_int8, deprecated in favor of it since v0.5.22: #45706
  • SGLANG_ENABLE_UNIFIED_RADIX_TREE is now ignored without a warning. It already had no effect because the unified radix tree is the default; drop it from launch scripts: #45090
  • Legacy tar-format .pt checkpoints no longer load. The unsafe weights_only=False pickle fallback is removed; convert such checkpoints to safetensors or zip-format .pt offline: #42760
  • Diffusion HTTP endpoints validate more strictly. URL values for upscaling_model_path and frame_interpolation_model_path return 400, an unsupported response_format is rejected before generation, and a failed cloud upload returns 502 instead of 400: #43585
  • The first startup of a FlashInfer CUTLASS MoE server takes longer. Startup autotune now also covers prefill sizes, so a server with no saved tuning results takes longer to become ready (on DeepSeek-V4.1-Flash on 8x H200, autotune goes from about 41 s to 177 s). Later startups reuse the saved results, and --disable-flashinfer-autotune skips tuning: #43363
  • Diffusion PNG responses are larger. PNG output now defaults to zlib level 0: the same pixels, about 3x the bytes, several times faster to encode. Set output_compression to a value other than 75 to pick a level: #45987
  • sgl-router moved to the top-level sgl-router/ directory. Builds and scripts that point at experimental/sgl-router/ must update the path (#45414); --max-in-flight is now --max-inflight-requests, and the old name still works as an alias: #44950

New Contributors

Full Changelog: v0.5.22...v0.5.23

Check out latest releases or
releases around sgl-project/sglang v0.5.23

Don't miss a new sglang release

NewReleases is sending notifications on new releases.

Get notifications