github sgl-project/sglang v0.5.20
on GitHub

4 hours ago

Highlights

713 PRs from 237 contributors.

New models in this release (see the cookbook for all supported models):

Model Type PRs Cookbook
GLM-5.3-Flash Autoregressive #36507, #38621 link
Hy4-Preview Autoregressive #36805 link
Qwen3.8-Flash-Next Autoregressive #37500 link
K2 Horizon Autoregressive #37654, #38033 link
Nanbeige4.2 Autoregressive #32151
SenseNova-U1.5-8B-MoT Diffusion #36606 link
FastH3 (4-step MiniMax-H3 distill) Diffusion #37480 link
VDN-H3 (hybrid-attention MiniMax-H3 distill) Diffusion #37903 link

Sampling masks for RL rollouts. With return_sampling_mask, each decode step returns the exact token support the sampler drew from and the log-probability of the sampled token under it, so a trainer can replay the rollout without reconstructing top-k or top-p (#36630). Masks now run under overlap scheduling: on Qwen3-8B, decode throughput is 17% higher at batch 1 and 52% higher at batch 64 than the previous implementation. Capacity is set by --sampling-mask-max-tokens (default 4096) (#36631). DisallowedTokensLogitsProcessor is supported alongside masks (#38279).

Unified radix tree. Branching-point caching for the SWA component keeps the sliding-window state at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. On DeepSeek-V4-Flash with a shared system prompt, token hit rate rises from 43.8% to 60.8% and mean TTFT falls from 1.57 s to 1.07 s (#34565). An opt-in external linker lets the tree address a shared global memory pool through Mooncake or UMBP (#37381).

DSpark under PD with decode context parallelism. A DCP1 prefill can now transfer its DSpark draft KV to a DCP-N decode, so hybrid models such as Kimi-Linear run DSpark in disaggregated, context-parallel serving. Verified on 8x B300 over NIXL and Mooncake up to 256K input (#37709).

Responses API storage is opt-in. /v1/responses no longer retains results in memory unless the server starts with --enable-response-store. Without it, retrieval, previous_response_id chaining, and background requests return 400; PD deployments cannot enable it (#39122).

SGLang Simulator. A CPU-only simulator runs the real scheduler, radix cache, and hierarchical cache with a latency predictor in place of the model forward. Against measured serving traces it predicts TTFT within about 6% on most traces (up to 10% on the longest 32K to 128K ones) and prefix reuse within 0.05 percentage points, for cache and scheduling studies without GPUs (#33824).

Prefill context parallelism v1 removed. The strategy-based implementation is now the only prefill CP path; the v1 runtime and its CLI options are gone. Prefill CP on HIP, NPU, and MUSA is rejected until those platforms are ported (#36228).

Faster model loading on ROCm. Large pageable host-to-device copies are staged rather than pinned in place, which stops the driver from suspending GPU queues on every eviction. GLM-5.2 at TP4 on 4x MI355X loads in 40.4 s instead of 505.7 s (#37720).

Intel XPU joins the release images. Every tagged release now builds and publishes lmsysorg/sglang:vX.Y.Z-xpu from the XPU Dockerfile, so Intel GPU users get a versioned image instead of relying on nightly builds (#37340).

DeepSeek-V4 on Blackwell. TRT-LLM attention kernels now cover DeepSeek-V4's CSA and HCA layers on SM100 and SM103: about 1.2x faster prefill and 1.45x faster decode than FlashMLA at the kernel level on B200 (#30805). FlashInfer MegaMoE is available as --moe-runner-backend flashinfer_megamoe; on DeepSeek-V4-Flash NVFP4 at TP4/DP4 it adds up to 11.9% prefill throughput at 8192 tokens per rank, with decode within 2% of the trtllm runner at saturation (#31470).

DeepSeek-V4 on RTX PRO 6000. On SM120 the sparse-MLA indexer now runs on DeepGEMM's paged-MQA kernel and the DeepGEMM FP4 MoE backend is enabled, replacing the torch fallback that was the only working path. On 4x RTX PRO 6000, DeepSeek-V4-Flash decode TPOT drops from 36.1 to 10.5 ms at batch 1 and TTFT falls 20% from 8K to 128K input. Opt-in through the environment flags in the PR (#29927).

Dependencies and images. The CUDA 12 lane is retired; v0.5.19 was the last release with -cu12x wheels and images (#38404). sglang-kernel moves to 0.4.7 (#39346) and sgl-deep-gemm to 0.2.0 (#39371). New images: ROCm 10 for MI30x and MI35x with a matching kernel wheel (#38763), gfx1151 for Strix Halo / Ryzen AI MAX+ (#33939), and Moore Threads MUSA (#36709). ROCm 7.0 CI, images, and kernel wheel are retired (#38632, #38767).

Full release notes by category below; breaking changes are at the end.

Speculative Decoding

  • [Speculative Decoding] Add native UNO serving support: #37667
  • feat: add optimized Domino rollout to DFlash V2: #36899
  • feat: support TP>1 Domino rollout for DFlash V2: #37069
  • [Spec] Stage Inkling MTP draft metadata before verify: #38169
  • [Spec] Allow speculative workers to stage prefill shared reads: #38554
  • [Spec] Support large MTP batches in short-convolution metadata: #38558
  • [EAGLE] Prune draft-extend logits to selected rows (+2.5% output tokens/s/GPU at 102K context per rank): #35546
  • [KDA] Support ReplaySSM ring-write in the fused chain-verify kernel: #36821
  • [GDN] Amortize ReplaySSM checkpoint materialization: #35544
  • perf(gdn): select ReplaySSM verify loop unrolling by shape: #36970
  • Allow custom policy for adaptive speculative decoding: #37274
  • Support speculative decoding with unified SWA memory: #36403
  • Improve CUDA graph and speculative execution output handling: #37329
  • Fix DSpark CUDA graph replay with MegaMoE TP attention: #34919
  • [Fix][Mamba] Clear deferred init metadata before speculative decode: #37165
  • [Fix] Track DFlash Mamba state at checkpoint boundaries: #37818
  • [Fix] Load Qwen3.5 MTP embedding under PP: #37471
  • fix: stop shadowing the DSpark shared-experts fusion guard: #39366
  • [Profiler] Label draft-runner steps DRAFT and target verify VERIFY in step spans: #38630

Piecewise & Breakable CUDA Graph

  • [Memory] Size the CUDA graph pool from warmup measurements and fix graph-pool borrowing: #36911
  • [Memory] Retire graph borrow pool before updating static runs: #37966
  • [Memory] Reuse output storage across full prefill CUDA graphs: #38038
  • [Unified Memory] Enable prefill cuda-graph capture: #37418
  • Decouple ragged CUDA graph request and token capacities: #37300
  • Reuse live CUDA graph executables during dedup registration: #39176
  • Scope graph-pool borrowing to the runtime and reduce fragmentation: #39177
  • Expose a capacity check for graph-pool borrows: #39178
  • Keep graph-pool borrows on their allocation stream: #39180
  • Add a provider hook for prefill-buffer ceilings: #39182
  • [Refactor] Generalize attention graph variants in the decode runner: #38993
  • [Fix] Coordinate FullCG prefix variants across DP ranks: #37888
  • [Fix] Keep a shared MAX_LEN prefill CUDA graph bucket when the graph captures a DP gather (MegaMoE sparse-DP hang): #37933
  • [Fix] Key CUDA graph dedup signatures on kernel function identity: #37657
  • [Bug] Include DSA variant in exact-bucket graph admission: #38992
  • Fix KV-canary workspace accounting after graph capture: #38596

Attention Backends

  • [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (kernel-level prefill about 1.2x and decode about 1.45x faster than FlashMLA on B200): #30805
  • [DSV4] Support raw-index output in TopK v2: #33672
  • [DSV4] hc-prenorm: fuse the combine step into a Triton kernel: #35118
  • [FlashInfer] Avoid D2H sync for sliding-window lengths (TPOT 5.4 to 5.1 ms on 8x A800): #32218
  • [Attention] Size FlashInfer MLA indptr buffers to the padded max batch: #38590
  • [Perf] trtllm_mla: reuse the fused fp8 KV/Q prepare on target verify: #39232
  • [Performance] Optimize Qwen3.5 GDN prefill projection layouts: #36267
  • Fix Qwen3.5 GDN multi-item scoring (+18.8% throughput, P50 latency -16%): #33922
  • Support NoPE layers in the tokenspeed_mla FP8 prefill hook: #38152
  • fix(qsa): make the paged sparse-decode gather memory-safe (zero-fill scratch, int64 offsets, dequant FP8 on gather): #38851
  • fix(qsa): dequantize FP8 cached prefixes in the sparse prefill kernels: #38855
  • Reland fix(qsa): clamp the compress gather to the rows (#38346): #39446
  • [Qwen 3.8 Next] reuse old cuda stream instead of endlessly creating streams: #39474
  • [Fix][DSA] Bound prefill Triton specializations for page-table stride: #37093
  • [Fix] Wait for PDL before reading DeepSeek V4 K cache locations: #38409
  • [Fix] Fix Llama 4 FA3 local attention with paged KV cache: #32902

MoE & Expert Parallelism

  • [NVIDIA] Support flashinfer Mega Moe (prefill input throughput up to +11.9% vs the trtllm runner on DeepSeek-V4-Flash NVFP4 at 8192 tokens per rank): #31470
  • [MoE Refactor] Migrate SM100 trtllm-gen mxfp4 MoE onto MoeRunner: #32405
  • [Deepseek V4] Keep fp32 routing weights in the mxfp4 trtllm MoE: #33608
  • Keep fp32 routing weights in the fp8 block-scale and bf16 trtllm MoE: #33631
  • Drop the routing bias casts in flashinfer trtllm MoE: #33591
  • Fix DeepSeek-V4 routing: sqrtsoftplus underflow and unfloored renorm: #34459
  • keeping router GEMM in fp32 for deterministic inference (DeepSeek V3/V4): #38176
  • Fuse Nemotron latent MoE projection and shared add: #30430
  • [Kernel] Replace dsv3_router_gemm with the unified tiny GEMM: #34693
  • [Kernel] Avoid zero-bias allocation in fused softmax routing: #36811
  • [Kimi-K3] Accept fp32 routing weights in the fused MoE finalize: #38612
  • Cast fp32 routing weights to bf16 in the Kimi-K3 fused finalize: #38588
  • Use fp32 in TRTLLM all reduce buffers: #36143
  • [Comm] Drop the in-tree MNNVL CuTe DSL port in favor of FlashInfer 0.6.18: #37206
  • [Moe] Fix flashinfer_trtllm silently dropping swiglu_limit clamped SwiGLU activation (GLM-5.3-Flash FP8 GSM8K 0.928 to 0.936, matching the Triton runner): #39920
  • Fix MegaMoE buffer allocation and caching for effective SM budgets: #39223
  • Fix native MoE handling of noncontiguous top-k IDs: #36407
  • [Fix] Accept int64 top-k IDs in FlashInfer routed MoE packer: #32882
  • fix(moe): cast filtered-activation expert_ids to int32 for torch.compile: #38085
  • fix(gpt-oss): avoid duplicate MoE reduction with DP attention: #37199
  • Fix mooncake scale joiner groups: #38503

Quantization

  • [Quant][ue8m0 fix] group requant_weight_ue8m0 reduce reserved gpu memory: #31755
  • [1/N] Quantization Refactor: remove dead code and dedup the FP4 marlin helpers: #37552
  • Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention: #32114
  • fix(modelopt_fp4): skip NVFP4 swiglu-fusion interleave for shared experts with swiglu_limit: #37378
  • Disable Hopper GLM shared-expert fusion for modelopt_fp4 Marlin: #37325
  • [Fix] Preserve FP32 in SM107 MXFP8 fallback: #37489
  • Fix block-scale swizzling device placement: #37849
  • Keep NVFP4 blockscale swizzle padding on the input device: #39141
  • Fix dummy initialization of inverse weight scales: #35491
  • [Chore] harden checkpoint quantization metadata parsing: #36922

Parallelism & Disaggregation

  • [PD] Transfer the DCP-replicated DSPARK draft KV in DCP1->DCP-N relayouts: #37709
  • [Qwen3.8-Next] Add PD state transfer for Flash Next: #36651
  • [Intra-node PD][DSV4] Pack all layers into one batch for INTRA_NODE_NVLINK path (KV transfer latency 178 to 119 ms, E2E latency -20% on a 2P1D B200 node): #38984
  • [PD] Share the prefill->decode failure notification across backends: #36612
  • [PD] Gate deferred decode KV release on backend capability: #37454
  • [PD] Bound transfer engine init with SGLANG_DISAGGREGATION_ENGINE_INIT_TIMEOUT: #37874
  • [PD] Preserve the prefill rank during rebootstrap: #39357
  • [PD] Optimize paged allocator free-list release: #37146
  • [PD][LoRA] Gate decode admission on adapter slots: #39332
  • [Unified Memory] PD disaggregation for every unified pool shape: #37506
  • fix(disagg): poll receivers during decode preallocation: #37483
  • Fix DSA compression tail capacity for PD decode request slots: #38417
  • fix(glm-5.2-nvfp4): bound Mooncake synchronous transfer batches: #32758
  • [DCP] Allow fi_a2a on single-node systems Blackwell without MNNVL fabric ( ex B200 B300) (median TPOT 1.5 to 5.5% lower than the NCCL all-to-all on 8x B200): #37767
  • [DCP] Resolve --dcp-comm-backend to fi_a2a/a2a by default for every model: #39165
  • [Feature] Unified memory: support decode context parallelism for Kimi-Linear: #36890
  • [Feature] Unified memory: support decode context parallelism for the trtllm_mla family: #37693
  • [Perf] Unified memory: close the DCP decode gap on Blackwell: #37926
  • [Fix] Merge adjacent KV-row frees so a mid-page split under DCP cannot double-free: #38941
  • [Fix][DCP] Localize widened KV ids in MLA retraction CPU backup/restore: #39487
  • [CP V1 Deprecation 2/5] Make strategy prefill CP canonical: #36223
  • [CP V1 Deprecation 3/5] Remove generic prefill CP v1 runtime: #36228
  • [CP V1 Deprecation 3.5/5] Deprecate HIP/NPU/MUSA prefill CP and remove legacy implementation: #38293
  • [CP V1 Deprecation 4/5] Canonicalize prefill CP API names: #36229
  • [CP V1 Deprecation 5/5] Update prefill CP documentation: #36230
  • fix: gather CP-sharded tokens before TP-sharded dense MLP under prefill CP: #38078
  • Fix inflated row pitch when a CP round-robin shard has a single row: #34142
  • [SP] Make attention-TP sequence sharding a per-forward batch property: #37546
  • [Fix] Stamp sequence-parallel state on dummy forward batches: #38564
  • [DP Attn] Fix crash for no token all-gather case: #39899
  • [Fix] Disable NCCL graph buffer registration for the TP LM-head all-to-all (pure-DP decode hang under request bursts): #38936
  • [Misc] Extract PP dynamic chunk sizing into a DynamicChunkSizer scheduler component: #37674
  • [Fix] Lock PP dynamic-chunk profiling requests before releasing through the tree cache: #37509
  • [Fix] Apply the attention-CP broadcast result in PP dynamic-chunk profiling: #37669
  • [Fix] Broadcast PP dynamic-chunk profiling failures so every rank disables together: #37675

Scheduler & Runtime

  • [Scheduler] Add HRRN schedule policy to significantly reduce TTFT (--schedule-policy hrrn; mean TTFT -69% and p99 -8% vs FCFS on a GLM-5.2 production trace): #32911
  • [Scheduler] Make request-timeout aborts rank-consistent to fix TP collective hangs: #37143
  • [Scheduler] Count the parked chunked-prefill request in the busy mem check: #37502
  • [Scheduler] Unify per-iteration request intake into ingest_requests(): #38389
  • Fail fast on undersized swa pool: #37610
  • Reduce tokenizer overhead and offload CUDA VMM publication: #37330
  • Keep VMM capability votes on CPU: #38722
  • state_capturer: pin the exact host-cache size via mmap + cudaHostRegister: #37285
  • Generalize auxiliary outputs: #39164
  • [Model Loader] Split weight loading from postprocessing: #34981
  • Fix gpt-oss RunAI streamer weight ownership: #38908
  • Fix RunAI object-storage checkpoint index filtering: #38988
  • Clean logging under --weight-loader-prefetch-checkpoints: #33930
  • [Fix] Release NCCL on scheduler exit and let the ASGI server own shutdown: #39560
  • [Fix] Aggregate all IPC weight update responses: #39534
  • [Fix] Avoid scanning crash-dump token buffers during GC: #37297
  • add reindex_device_id to device OOT plugin: #36220
  • [Metrics] Add rolling scheduler utilization counters: #37461
  • [Metrics] Export scheduler stage wall time: #37636
  • [Metrics] Report logical prefill token counts: #38566
  • [Observability] Fix missing e2e/decode/inference latency span attributes: #37789
  • [Ray] Support Ray metric backend for engine metrics: #31415
  • [Profiler] Add SGLANG_PROFILE_BY_STAGE_DECODE_MIN_BS to defer the decode-stage capture: #38067
  • Add num_prealloc_ready_tokens to decode load snapshot: #38065
  • [Bench] Support real-traffic replay with early-stop-aware steady-state metrics in bench_one_batch_server: #37469
  • [Benchmark] Add an opt-out for the token-capacity check: #39284
  • Fix reasoning metrics and add TPOT to bench_multiturn: #35443

Sampling

  • [Sampling] Capture masks from sampler support: #36630
  • [Sampling] Support sampling masks with overlap scheduling (Qwen3-8B decode throughput +17% at batch 1 and +52% at batch 64 vs the previous non-overlap path): #36631
  • [Sampling] Allow sampling-mask replay with DisallowedTokensLogitsProcessor: #38279
  • perf: use Gumbel-max trick in the main sampler to cut decode CPU dispatch (Qwen3.5-2B decode +75% at batch 1 and +23% at batch 32): #38117
  • Fix custom logit processor params when num_tokens_in_batch is used: #38730

HiCache & Radix Cache

  • [Unified Tree] Support Branching-Point Caching for the SWA Component (shared-prefix token hit rate 43.8% to 60.8%, mean TTFT 1.57 to 1.07 s on DeepSeek-V4-Flash): #34565
  • [Unified Tree] Port SWA Branching-Point Caching to the Rust TreeCore: #37584
  • [Unified Tree] fix: make dfs weight ordering iterative: #38313
  • [Unified Tree] Preserve aux LRU recency when splitting nodes: #38482
  • [Unified Cache][5/N]: Integrate external linker mode end to end: #37381
  • [Unified Cache][6/N]: Add UMBP external linker: #37578
  • [Unified Cache][7/N] Support MTP, EAGLE, and DSpark draft KV caches in the external linker: #37914
  • [Unified Cache][AMD] Support DeepSeek-V4 unified KV in direct external linkers: #38269
  • [Rust TreeCore] Harden runtime and CI parity: #37303
  • [Rust TreeCore] Support external cache linker: #37306
  • [Rust] Rename mem-cache to sglang-radix-tree: #37290
  • [KV Shard 1/4] Logical-page placement with UnifiedRadixCache: #38356
  • [Cache] Forward fast prefix matching capability: #37844
  • [Refactor] Let eviction policies take construction parameters: #37795
  • [Perf] Walk the radix tree by offset instead of re-slicing token storage (ported from #36507): #37324
  • [Perf] Vectorize alloc_extend_naive to remove the per-request Python loop: #37938
  • [Radix Cache] Fix PureSWA tail release without insertion: #38349
  • fix: preserve SWA host lock on node split: #38138
  • fix: collect prefix hash values iteratively: #38204
  • fix: align write-through pending across tree cores: #37278
  • [HiCache] Add MLA host-dedup primitives: #36800
  • [HiCache] Replace skip_lock_node_ids with a segment lock protocol: #36848
  • [HiCache] Buffer mode support sidecar pool: #37424
  • [HiCache] buffer mode: anchor-lock staged prefetches by default: #37464
  • [HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes: #37503
  • [HiCache] Count hit allocations and in-flight backups in the buffer pipeline idle check: #37883
  • [HiCache] Fix side pools to use resolved host allocator: #38350
  • [HiCache] Account for newly pinned ancestors in load-back quota: #38481
  • [HiCache] Release buffer prefetch anchor locks during storage cleanup: #38483
  • [HiCache] Publish a host store event for storage-prefetch refills: #38486
  • [HiCache][LoRA] Isolate storage pages by extra key: #38577
  • [HiCache][LoRA] Simplify decode offload hash inputs: #39162
  • [HiCache] fix: preserve SWA host lock boundaries across splits: #38835
  • [HiCache] Keep the file backend temp file name within NAME_MAX: #38925
  • [HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping: #39283
  • [HiCache] Optimize buffer-mode storage existence bookkeeping: #39480
  • HiCache: Add @rank_consensus to various functions: #37425
  • HiCache: Reduce the number of all_reduce in check_hicache_events for PP (TTFT -7% on DeepSeek-V4-Flash with HiCache L3, PP4 TP2): #37562
  • Fix buffer-mode idle tracking and VLM memory sizing: #37567
  • fix: keep queued Mooncake linker loads after abort: #38195
  • fix(nixl): make FILE path-mode devId globally unique: #34362
  • Scope prefetch cache state to the request attempt: #39318
  • [Fix] HiCache startup ImportError on the pinned kernel wheel: #39516
  • [Mem Cache] Make free_swa sync-free on page_size == 1: #36723
  • [Mem Cache] Drop the torch.unique sync from the SWA page expansion: #37463
  • [Mem Cache] Split duplicate insert frees at the SWA eviction floor: #37481
  • [Mem Cache] Require page-aligned starts in free_segment and drop the boundary trim: #37729
  • [Mem Cache] Route hybrid SWA full-side kv-row frees through free_segment: #37876
  • [Mem Cache] Move the unified-memory allocators into allocator/ and split the composites out: #38072
  • [Mem Cache] Free hybrid SWA pages by one representative per page on page_size > 1: #38159
  • [Mem Cache] skip duplicates host evict via environ: #38462
  • [Misc] Resolve SWA ownership at enqueue time for grouped free(): #36646
  • Fix unified SWA: size a non-owner's v2p by the id space it must address: #37560
  • Size the unified read-table grid from bs, and fuse the allocator's tombstone scatters: #37511
  • Build the unified read stream directly, without the page-table rectangle (decode CUDA graph memory 2.89 to 1.89 GB on Kimi-Linear-48B TP2): #37512
  • fix(unified-memory): evict Full KV for Mamba byte shortfalls: #36713
  • Store mamba prefix-cache checkpoints at the configured SSM state dtype: #34820
  • Fix mamba radix cache ssm state indexing: #37836
  • [Fix] Vacuous marker writes in the cache tests, and an undebited Mamba admission slot: #36415

LoRA

  • [LoRA] Support MoE in full and breakable prefill CUDA graphs (prefill throughput +39.5 to 40.9% with Triton LoRA on Qwen breakable graphs, GB300): #38578
  • [Intel][XPU][LoRA] Enable LoRA on Intel XPU: #30345

Multimodal

  • [Perf] Optimize Qwen3-VL unique-image serving on H100: #36411
  • fix(multimodal): handle tensor images in exact-token preprocessing: #30368
  • Fix multimodal embedding cache retaining full batches through views: #39120
  • [Fix][Qwen-VL] Normalize <image> sentinel on artifact fast path: #39278
  • fix(glm4v): disambiguate mixed image video offsets: #37971
  • [Fix] Alpha-channel images and tool-result media ordering (port of #36507): #37320
  • [MM] Add flag to force Kimi image preprocessing onto CPU: #39148
  • [VLM] Retire aborted disaggregated prefill results: #36988
  • fix(vlm): contain EPD request lifecycle failures: #36944
  • fix(vlm): harden EPD receiver validation and liveness: #36945
  • fix(vlm): make EPD cache publication transactional: #36949

Model Support & Optimizations

  • [Cosmos3] Add cosmos3 Reasoner to llm only inference: #33572
  • Support NemotronH_Omni_Reasoning_V3 in SGLang: #35599
  • Inference Support Mamba 2 and 1 (Mamba-Codestral-7B, Falcon-Mamba-7B, and the state-spaces Mamba checkpoints): #34556
  • [GLM-5.3 Flash] Restore and enable KPool metadata fusion: #38845
  • MiniMax-M3: share the sparse index top-k across layers and reuse the decode top-k buffer (output throughput +6.4 to +20.6% at 80K input on 8x MI350X): #36527
  • MiniMax-M3: Triton split-K router GEMV with in-kernel fixup: #36557
  • perf(lfm2): fuse gating and short convolution on SM90: #37622
  • [GraniteMoE] Load split per-expert quantized MoE weights: #37679
  • Fix Muse Glimmer ModelOpt mixed weight mapping: #37510
  • [Fix] Fix Spark2.5 hybrid SWA config: #37335
  • fix(mamba): unify causal_conv1d col* dtype to x (MiniCPM-V-4.6 GDN prefill bf16/fp16 mismatch): #38039
  • [Refactor] Clarify DeepSeek V4 metadata names for V4.1: #38947
  • [Refactor] Generalize DeepSeek V4 compressed pool management: #38954

Kernel Library

  • [Kernel] GLM 5.3 Flash related kernels (ported from #36507): #37477
  • [Kernel] Register merged diffusion agent kernels with KDA backend: #37385
  • [Kernel] Add fused MoE Triton configs for Qwen3.8-Flash-Next FP8 on NVIDIA H200 NVL (TP2+EP2) (1.14 to 1.23x at decode-sized batches): #38116
  • [Kernel] Add tuned LFM2.5 Triton MoE configs on B300: #37158
  • [Kernel] Drop the vendored dense BF16 GEMM port in favor of FlashInfer 0.6.18: #38124
  • [Kernel] Add OOT dispatch for clamp position: #38687
  • [JIT] Port the expert-pack MXFP4 kernels to load_jit and fix their launch limits: #38830
  • feat(kernels): generalize persistent CuTe JIT cache: #33911
  • Add SGLANG_CRASH_ON_JIT_COMPILE to forbid on-the-fly JIT compilation: #36615

Configuration System

  • [Config] Round 6.1: "unset" gets its own spelling, and the declaration says what it means: #38046
  • [Config] Round 6.2: the field declarations move to their namespaces, and the record is assembled from them: #38047
  • [Config] Round 6.3: the record remembers how it was asked for, and is sealed while resolution runs: #38048
  • [Config] Round 6.4: the runtime reads the bags, not the record: #38049
  • [Config] Round 6.5: a namespace declares what it derives, next to what it derives it from: #38113
  • [Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement: #38375
  • [Config] One writer for the declaration stash; no exception to the write seal: #38752
  • [Config] msgspec.Struct for the config tier: #38753
  • config: an out-of-tree replacement point for every resolution-pipeline step: #39134
  • Fix dataclasses.asdict on the msgspec ServerArgs: #38958

OpenAI-Compatible API

  • [Feat][Responses API] Support custom tools, encrypted reasoning replay, developer tier and model validation: #38690
  • [PD][OpenAI] Gate /v1/responses persistence behind --enable-response-store, default off: #39122
  • [OpenAI] Propagate PD routing metadata through /v1/responses: #35503
  • [Responses] Fix empty-prompt routing for token-first chat encoders (kimi_k3, inkling): #35486
  • [Feature] Add response-level input/output token ids to chat completions via SglExt: #34488
  • Support non-strict GLM47 tool calls with EBNF constraints: #38890
  • [Fix] Preserve GLM tool argument types across JSON Schema unions: #39136
  • Auto-detect GLM-5.3 chat templates as glm45/glm47 parsers: #38297
  • Force reasoning mode for GLM-5.3 chat templates: #39227
  • Add granite_thinking_parser reasoning parser for Granite 4.2: #38693
  • [Kimi K3] Rework skipped-think fix as opt-in force_nonempty_content with streaming coverage: #34187
  • [Kimi-K3] Recover the reply when the model skips the think channel: #37743
  • [Session] Fix session idle timeout after rejected requests: #39035
  • [Session] Work with PD and Fix empty continuations: #39038
  • [Session] Fix image append positions and parent metadata: #39145
  • [Session + MM] Fix text positions in session continuations: #39144
  • feat(agent sessions): attribute stored KV cache blocks to sessions: #37482
  • Fix /model_info serialization when a config value is a class: #39237
  • Fix: abort handling for dispatched requests after client disconnect: #35255

Rust Server

  • [Router] Add composable scoring and eligibility policies: #37731
  • [Router] Add load-aware prefill admission and bounded policy proposals: #37843
  • [Router] Add bucket-aware policy domains and native cache indexing: #38108
  • [Router] Publish cache-aware load state: #38139
  • [Router] Preserve global cache affinity with bucket routing (TTFT p50 -18.5% vs power-of-two bucket routing): #38814
  • [Router] Honor KV-event storage tiers in the cache-aware tree (1/4): #39108
  • [Router] Expose the KV storage-tier stream and tree occupancy on /metrics (2/4): #39109
  • [Router] Stream outcome observability for 2xx SSE streams: #38737
  • [PD] Add /v1/responses support to the HTTP PD router: #36141
  • [Rust] Gate health on startup warmup completion: #37994
  • [Rust] Use Dynamo native renderers when chat templates are missing: #38939
  • [Rust Server] Use node-local HTTP ports for DP attention: #34430
  • [Rust Server] fix p/d bootstrap across dp listeners: #36234
  • Rust server: align launcher and request validation behavior: #37327
  • Allow CUDA VMM feature transport with the Rust frontend: #39347
  • [gRPC] Stream engine state changes: #39915
  • [gRPC] Expose native pause status: #37488
  • [Fix] Seed raw tokenizer_path for smg-grpc-servicer in gRPC mode: #39105
  • [SGL Router] Prepare dynamo-render dependencies: #39457

Simulator

A CPU-only inference simulator under tools/sglang-simulator that reuses the real scheduler and cache stack with a latency predictor in place of the model forward.

  • [Simulator] Add high-fidelity CPU-based inference simulator: #33824
  • [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom: #38732

SGLang-Diffusion

  • [Diffusion] Add MiniMax H3 cube sparse attention (denoise 19.3% faster with the aggressive schedule on 2x H100): #34893
  • [Diffusion][MiniMax-H3] Add SM90 Sage compute for SubBlock sparse attention (generation time -6.3 to -20.9% vs BF16 SubBlock): #37982
  • [Diffusion] Add SpargeAttention backend: #37437
  • [Diffusion] Add request-scoped Skip Softmax attention: #37959
  • [Diffusion] Pick the attention backend by measuring it: #38689
  • [Diffusion] Admit explicit attention backends by capability: #37441
  • [Diffusion] Add cumulative extra-high quality tier: #37422
  • [Diffusion] Support MiniMax-H3 Singularity hybrid checkpoints: #38455
  • [Diffusion] MiniMax-H3: tiered AdaLN plan cache (pinned-host tier + per-plan LRU): #37266
  • [Diffusion] MiniMax-H3: warm up at the served clip shape: #37945
  • [Diffusion] MiniMax-H3: Only use fused qk_norm on NV: #33994
  • [Diffusion] Fix MiniMax H3 WebUI inference settings: #36320
  • [Diffusion] fix: stabilize H3 reference audio across repeated requests: #38225
  • [Diffusion] Return Qwen-Image-Layered outputs and preserve CFG2 rounding (engine E2E 34.2 s on one H200 to 17.3 s on two, byte-identical layers): #38549
  • [Diffusion] Optimize Qwen-Image-Edit attention on Hopper: #38584
  • [Diffusion] Port the Wan VAE decoder fast paths to the Qwen-Image VAE: #38020
  • [Diffusion] Keep the Wan VAE decoder channels_last and add a Triton NHWC nearest upsample: #38182
  • [Diffusion] Fuse FLUX.2 NVFP4 FC1, SwiGLU, and FC2 quantization: #37096
  • [Diffusion] Enable lossless BCG for FLUX.1-dev: #38591
  • [Diffusion] Enable breakable CUDA graph for JoyEcho: #38110
  • [Diffusion] Optimize LTX-2 QKNorm and split RoPE on Hopper: #38396
  • [Diffusion] Preserve BF16 rounding in Hopper LTX QKNorm and RoPE fusion: #38533
  • [Diffusion] Support diffusion decoder parallel tiling for LTX-2.5: #36026
  • [Diffusion] Fuse LingBot per-token gated residual and RMSNorm modulate: #37910
  • [Diffusion] fuse LingBot MoE group-limited top-k index selection: #38044
  • [Diffusion] add Helios per-token gated-residual fusion (quality-gated): #38042
  • [Diffusion] Fuse LongCat Image normalization and modulation: #38530
  • [Diffusion] Optimize SANA-WM convolution post-processing and streaming GDN: #38529
  • Cosmos3 fp8 mixed precision: #36380
  • [Diffusion] Support mixed INT8 embeddings and Comfy NVFP4 encoders: #38506
  • [Diffusion] auto-keep video DiT resident on high-memory GPUs (SANA-WM end to end -38% on one B300): #38001
  • [Diffusion] Add explicit snapshot-offload component residency: #38535
  • [Diffusion] Stream mapped weights on a shared host/device pool (layerwise-offload request 478 to 302 s): #38441
  • [Diffusion] Read mapped layers directly when the host cannot cache them: #39022
  • [Diffusion] Measure warmup memory and layer usage per phase for residency calibration (1/4): #37916
  • [Diffusion] feat: spill large tensors over shared memory like numpy arrays: #38656
  • [Diffusion] Set an explicit x264 preset for video output: #38657
  • [Diffusion] Compose third-party component bundles safely: #37816
  • [Diffusion] Consolidate plain state-dict component loaders: #38128
  • [Diffusion] loader: reuse plain state-dict loading without per-model classes: #38127
  • [Diffusion] loader: filter duplicate precision variants across custom loaders: #37616
  • [Diffusion] Remove component loader capability switches: #36824
  • [Diffusion] Key the VAE decode-dtype store by module layout: #38496
  • [Diffusion] Serve requests that turn CFG off on a CFG-parallel server: #39303
  • [Diffusion] Make the SP sequence gather pass contiguous shards: #39291
  • [Diffusion] Don't route an unreadable checkpoint into the native fallback: #39292
  • [Diffusion] Recover IPC JIT initialization after interrupted builds: #39034
  • [Diffusion][Refactor] Refactor and unify RoPE execution for DiT models using RotaryEmbedding based on CustomOp: #33555
  • [Diffusion] feat: add maybe_record_function profiler spans for request phases: #35922
  • [Diffusion] Improve BCG warmup frame-count diagnostics for video models: #37890
  • [Diffusion] Fix #38166: Restore non-layer placeholders before releasing host copies: #38171
  • [Diffusion] fix: preserve mapped courier tensor lifetime: #37965
  • [Diffusion] fix host-resident vocab tables loaded on GPU: #38012
  • [Multimodal Gen] feat: support key masks on USPAttention's replicated-prefix path: #36735

Local & Desktop AI

  • [SM120] DeepSeek-V4: DeepGEMM paged-MQA indexer +FP4 MoE+ page-split (decode TPOT up to 3.4x faster and TTFT -20% on 4x RTX PRO 6000): #29927
  • [SM120] Use exact query-head widths for DeepSeek-V4 sparse MLA decode: #36655
  • [Kernel] Add KDA NVFP4 GEMM for Qwen3.x on SM120 (1.32x geometric-mean kernel speedup at decode shapes on RTX PRO 6000): #36865
  • [Kernel] Add KDA FP8 skinny GEMM for SM120: #38082
  • [FP8] SM120: route FP8 linear to per-tensor (cudnn/nvjet) instead of channelwise cutlass (Ministral-3-14B decode 46.9 to 86.6 tokens/s at batch 1): #38006
  • sm120 32GB mem-tier: raise decode cuda-graph max_bs 24->48 + chunked_prefill 2k->4k: #37898
  • [Qwen3.8] Enable NVIDIA NVFP4 on DGX Spark with file-backed PLE and PDL router fix: #39126
  • [Diffusion][minimax-h3] Add SM120 support for SubBlock sparse attention (1.22 to 1.85x lower latency on 8x RTX PRO 5000): #37332
  • [Diffusion] MiniMax-H3 VAE decoder: unfused w2 bias on SM12.x (cuBLAS 16x16 kernel mis-dispatch): #37835
  • [Diffusion] Pin layerwise host stores in place at their exact size: #39021
  • [Fix] Register triton.runtime.cache.triton_key in the MPS stub so torch.compile keeps working: #37937

AMD / ROCm

  • [ROCm] Stage large pageable H2D copies instead of pinning them in place (GLM-5.2 TP4 load on 4x MI355X from 505.7 s to 40.4 s): #37720
  • [AMD] Optimize Kimi-K3 Triton MLA prefill on gfx950 (1.73 to 2.09x faster prefill kernels; +16.7% geometric-mean throughput at 68K/350 TP8): #35770
  • [AMD] Perf Kimi-K3 MoE optimization: #33838
  • [AMD] Perf Kimi-K3 fuse ROCm KDA decode boundary: #34198
  • [AMD] [Kimi-K3] Fuse the KDA input projection into a single GEMM on ROCm: #35176
  • [AMD] support qlen>1 for aiter gluon path for Kimi K3: #37601
  • [AMD] Support aiter fa mha chunked kv for Kimi-K3: #37691
  • [AMD] Enable topk v2 GLM ROCm (+10.2% throughput per GPU and -9.9% median TPOT geometric mean on MI355X): #36851
  • [AMD] Enable Fast Triton Sparse MLA backend: #30575
  • [AMD][gfx95] Fill the chunked-prefill compute budget exactly (mean TTFT -5.4 to -9.5% on GLM-5.2-MXFP4 MI355X TP4): #32888
  • [ROCm] Take the fused DSA metadata kernels and drop redundant work from the absorb path: #37124
  • [ROCm] Make DSA indexer top-k exact with cooperative selection (about 3x faster top-k kernel): #37591
  • [AMD] Skip unused TOPK v2 plan kernel on ROCm: #37580
  • [AMD][GLM-5.2] Keep GlmMoeDsa MoE e_score_correction_bias in fp32: #37133
  • [AMD] Enable FP4 indexer for Deepseek V4 (+5.8% throughput per GPU, -5.5% median TTFT on MI355X): #37353
  • [AMD] Fix FP4 indexer OOR: #37660
  • [AMD][DSv4] Fuse the DSv4 FP4 indexer prefill-schedule preamble into one kernel (output throughput +15.3% at concurrency 4): #37764
  • [AMD][DSv4] Switch output projection gemm (oproj_a) to fp8: #37423
  • [AMD][DSv4] Fuse inverse-RoPE into the fp8 wo_a quant (stacked on #37423) (decode TPOT -14.3% vs bf16 at 128K/1K, concurrency 64): #37658
  • [AMD][DSV4] feat: enable fp8 two-pool unified_kv on gfx950 (about 1.5x more resident KV tokens): #37413
  • [AMD][DSV4] Reland unified-KV pool sizing and SWA ring accounting, fully gated (+83.6% full-attention KV tokens): #38192
  • [AMD][DSV4] Skip the paged SWA page return under the per-request ring: #38571
  • [AMD][gfx95] DSV4 wo_b (dp-attention): route to tuned bpreshuffle GEMM instead of triton: #38227
  • [AMD] Fix DSV4 unified attention sink TP slice: #35092
  • [AMD] Fix Dspark accept length and reduce host bubble on DSV4: #39116
  • [AMD] Fix DeepSeek block-FP8 loading on gfx94x: #38446
  • [AMD] gfx950 assembly attention for EAGLE verify, draft extend and decode (median TPOT -25.6% on Qwen3.5-397B MXFP4 TP4): #37465
  • [AMD] gfx950 assembly attention: length-aware split-KV for dynamic workload: #39172
  • [AMD] Parallelize aiter spec-decode KV index building over token blocks: #37659
  • [AMD] Quantize the bf16 MTP draft experts online to MXFP4 for Qwen3.5 (+13.6% throughput per GPU, -10.3% median TPOT): #38748
  • [AMD][Fix] Fix aiter bpreshuffle GEMM for output sizes it cannot dispatch for qwen3.5 mxfp-attn-fp8-v2 TP4: #37564
  • [AMD][DCP 1/N] add dcp support for aiter backend: #34432
  • [AMD] aiter: honor per-layer softmax scale: #38754
  • [AMD] aiter: fail loudly on cross-layer KV sharing in target_verify: #38755
  • [AMD] aiter: resolve SWA KV pool for draft workers + guard paged decode: #38756
  • [AMD] aiter: route head_dim>256 prefill through Triton unified_attention: #38757
  • [AMD] Allow aiter attention backend for Gemma-4 (about 17% more output throughput than Triton at concurrency 128): #38758
  • [AMD] Skip AITER FP8 ASM prefill when GQA is unsupported: #38467
  • [AMD] fix: use the hardware fp8 e4m3 convert on gfx950: #37140
  • [AMD] Fix weight checking for AITER-shuffled block FP8 weights: #34330
  • [AMD] Fix Quark load of MiniMax-M3 MXFP4 index_qkv_proj: #37254
  • [AMD] Restore SWA reprefill-tail on UnifiedRadixCache when HiCache is off: #32759
  • [AMD] Fix registered HiCache host pointer aliases: #35233
  • [ROCm] Raise HiCache JIT block quota for mapped-host throughput: #39036
  • [AMD] Fix EAGLE crash when no kv_index_translator is bound on the DSA fp8 read door: #38318
  • [AMD] CI: fix Lean decode crash on the EAGLE path: #37119
  • [MoE][ROCm] Admit the unified Triton router on ROCm, including single-group routing: #38328
  • [AMD][Diffusion] Migrate FlyDSL fused norm kernels to the v0.3.0 stable API: #36349
  • [AMD] Fix ROCm VAE Conv2D fast path breaking spatial-parallel decode: #34424
  • [AMD][Fix] Fix regression in jit build error with FLUX.2-dev on gfx1250: #38329
  • [AMD] Add gfx1151 (Strix Halo / Ryzen AI MAX+) Docker image: #33939
  • [AMD][CI] Publish ROCm 10 release images and kernel wheel: #38763

NPU / Ascend

  • [NPU] Add NPU arch35 support and enhance DSV4 processing in DeepSeek-V4: #37373
  • [NPU] Support DSV4 host memory cache management: #37382
  • [NPU] Add sparsity-driven KV offload for DeepSeek DSA on Ascend (decode throughput 1.6x): #33089
  • [NPU] Support GLM5.2 and FP8 DSA&Indexer kvcache for 950: #38250
  • [NPU] glm5.2 fp8 memory opt: #38807
  • [NPU] Adapt DFlash2 speculative decoding to Ascend NPUs: #35629
  • [NPU] Support DFlash speculative decoding for MiMo-V2.5-Pro (mxfp4): #37565
  • [NPU] Enable non-greedy MTP sampling: #32495
  • [NPU] Support EAGLE when PP enabled in prefill nodes: #32207
  • [NPU][HiCache] Optimize HiCache L2 IO with Memfabric acc_offload: #38826
  • [NPU][HiCache] Add Ascend Memcache Hicache L3 storage backend: #38827
  • [NPU] support mf device urma and host rdma trans type: #38174
  • [NPU] Strip padding before FIA kernel for vision encoder padded sequences: #36329
  • [NPU] Fix ViT graph key layout handling: #37758
  • [NPU] fix pp 2 hang on npu: #38249
  • [NPU] [Fix] Fix NPU MLA HiCache backup accessing missing data_ptrs.: #36813
  • [NPU] fix extend_seq_lens_cpu shape in eager mode: #36843
  • [NPU] Fix error due to missing parameter quant_linear passing: #37384
  • [NPU] Fix device mismatch in SWA mask for DSpark verify graph capture: #39353
  • [NPU][Fix] Disable pinned memory to fix DeepSeek-V2 DP-attention hang: #39423
  • [Feat] Add base NpuSRTPlatform implementation: #36472
  • [Diffusion] [NPU] Optimize LTX-2/2.3 inference performance for NPU (latency -7.6 to -10.9%): #34722

CPU / Intel / XPU

  • [CPU] Support FP8 KV cache: #32733
  • [CPU] Implement fused QK Norm and RoPE kernels: #37748
  • [CPU] Add native CPU kernel for MurmurHash32: #35604
  • [CPU] Support Qwen3.8 text+video: adding torchcodec, ffmpeg and removing pin_memory: #35492
  • [Diffusion][CPU] Enable MiniMax-H3 on Xeon CPU: #35147
  • [CPU] Fix shm allreduce collision and sglang-router import: #37179
  • [CPU] Fix native KV hash compilation in Xeon image: #34767
  • [CPU] Update base image to Ubuntu 26.04: #35313
  • XPU: Enable GLM5.1 (GlmMoeDsaForCausalLM) DSA Attention: #24959
  • [Intel GPU] Add MLA support to Intel XPU Attention backend for Prefill: #35866
  • Speculative Decoding support for intel_xpu attention backend on XPU target: #30548
  • DFLASH support added for XPU: #32798
  • [XPU] Support GPT-OSS MXFP4 checkpoints on Intel XPU: #35751
  • [Feature][Intel XPU] Add memory saver support for Intel XPU via upstream torch_memory_saver: #29935
  • [Intel][XPU] Add NUMA node binding support for Intel XPU: #31113
  • [XPU][Diffusion] Enable MiniMax H3 on XPU platforms: #33366
  • [XPU] Re-add intel xpu on triton paths in diffusion platforms: #36654
  • [XPU] Adapt device agnostic API usage: #32093
  • [XPU] Make checkpoint_engine worker device-agnostic: #32382
  • [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow: #35051
  • [XPU] Add xpu forward in Gemma3RMSNorm & Add test and benchmark for Gemma3RMSNorm: #36278
  • Fix/whisper xpu varlen encoder decoder: #36298
  • Xpu/weekly simple model enablement 2026 08 30: #37193
  • [Intel GPU] Align XPU toml file for rust support: #31031

MUSA / Moore Threads

First installation path for Moore Threads GPUs: a MUSA Dockerfile, install guide, and runtime extras.

  • [MUSA] Add installation guide and Dockerfile: #36709

Security

  • [Rust] Bound multimodal media ingress: #37967
  • [Router] Raise chat body cap to 32 MB for multimodal payloads: #38735

Cookbook Updates

  • MiniCPM5-2B deployment guide: link
  • DeepSeek-V4: DGX Spark (2x GB10) Flash FP4 recipes, Flash Vision verified on B200 and GB300, Pro B200 FP4 agentic HiCache with DSpark: #37479, #37737, #37468, #37492, #38026
  • DeepSeek-V4 on MI355X: agentic-workload arguments, DSpark config, and PD disaggregation recipes for all three strategies: #38677, #39252, #39396
  • GLM-5.3-Flash: FP8 speed data on B300, H100, and B200, NVFP4 rows, MTP with the flashinfer_trtllm runner on Blackwell, and a breakable prefill CUDA graph opt-in: #37412, #37576, #39213, #38522
  • GLM-5.3: DFlash2 speculative option: #37392
  • GLM-5.2: NVFP4 on B200/B300 for AgentX with HiCache, and MI355X MXFP4 on the Triton DSA backend with TP8/EP1 and TP4/EP4 recipes: #35368, #39106, #39230, #39406
  • Kimi-K3: measured B300 1x8 numbers, DCP under HiCache L1+L2 with DSpark, and measured MI350X/MI355X cells: #37878, #39190, #39029, #37781
  • Qwen3.8-Flash-Next NVFP4 on DGX Spark (1x, 2x) and RTX PRO 6000, and the NVIDIA NVFP4 export for Qwen3.8-27B: #37995, #38611
  • Qwen3.5: FlashInfer GDN on B200, FP8 on B200/B300 with trtllm-gen MoE and symmetric memory, and MI355X MXFP4 with the AttnFP8-V2 checkpoint and HiCache: #37360, #38374, #39104, #39358
  • MiniMax-H3: GB300 and GB200 recipes, DGX Spark, a measured RTX 5090 desktop recipe, and VDN-H3 numbers on B200: #38296, #37456, #39373, #39244
  • JoyEcho: measured H200 residency and BCG recipe: #38534
  • Diffusion performance guide: per-model tuning decision table and single-GPU B300 notes: #38148, #37891

Dependencies

  • [Deps] Retire the CUDA 12 lane: #38404
  • chore: bump sglang-kernel version to 0.4.7: #39346
  • chore: bump sgl-kernel version to 0.4.7: #39324
  • bumping sgl-deep-gemm to 0.2.0: #39371
  • [Deps] Raise smg-grpc-servicer floor to >=0.9.0 to unbreak SMG E2E CI: #38801
  • [CI] Install sgl-eval from PyPI through the test extra: #37504
  • Add missing test dependencies to pyproject.toml variants: #36098
  • [DeepSeek-V4.1] Bump FlashMLA to the fork's rebase head (v4.1 kernels): #39171
  • [AMD] Update ROCm AITER pin to 4ad9983: #37784
  • [AMD] [Docker] Update MoRI to v1.2.3: #38411
  • [AMD][CI] Retire the ROCm 7.0 kernel wheel: #38767
  • [NPU] Update sgl-kernel-npu version to 2026.9.0 and move memfabric deps into pyproject: #37399
  • [NPU] Bump memfabric and sgl-kernel-npu versions in docs and pyproject_npu.toml: #38437
  • [XPU] install xpu-kernel by released wheel: #37394
  • [XPU] Add Regular Docker Image Release workflow for Intel XPU: #37340
  • docker(xpu): unblock nightly build (setvars.sh + sgl-kernel rename): #38617
  • pyproject(xpu): drop human-eval git dep to unblock image build: #38824

Breaking Changes & Upgrade Notes

  • The CUDA 12 lane is retired. No -cu129 / -cu12 wheels or images are published from this release on; already-published tags are unaffected, and v0.5.19 is the last release with a CUDA 12 lane: #38404
  • Prefill context parallelism v1 is removed. The strategy-based CP path is canonical; the SGLANG_ENABLE_CP_V2 gate, the v1 runtime, and the generic v1 CLI options are gone, and CP APIs are renamed to versionless names: #36223, #36228, #36229. --enable-prefill-cp is rejected on HIP, NPU, and MUSA until their CP refactor lands; decode CP and non-CP inference are unaffected: #38293
  • /v1/responses storage is off by default. Start with --enable-response-store to keep retrieval, previous_response_id chaining, and background requests; PD deployments cannot enable it and reject those features at admission. background=true with store=false is rejected: #39122
  • Sampling-mask capacity moves to --sampling-mask-max-tokens (default 4096). SGLANG_DISAGGREGATION_SAMPLING_MASK_MAX_TOKENS is rejected at startup, and masks under PD require SGLANG_ENABLE_DISAGG_SAMPLING_MASK=1 on both prefill and decode servers: #36631
  • The cutlass_mla attention backend, the non-Marlin gptq quantization choice, the AWQ AOT dequantize kernel, and Dual Chunk Flash Attention are deleted. GPTQ checkpoints continue through gptq_marlin; auto-round and moe_wna16 lose their non-Marlin GPU fallback: #32114
  • Seventeen deprecated CLI flags are removed and get_global_server_args now raises; read resolved values from the namespace bags or the raw input from get_server_args(): #38375. The config tier, including ServerArgs, is now msgspec.Struct, so dataclasses.asdict and dataclass introspection on it no longer work: #38753, #38958
  • --dcp-comm-backend now resolves per platform for every model: fi_a2a on Blackwell when the DCP group sits inside one node or MNNVL domain, otherwise a2a on CUDA and ROCm; pass --dcp-comm-backend ag_rs to restore the old default: #37767, #39165
  • SM120 FP8 linear layers with per-channel weight scales take the per-tensor route (cudnn/nvjet) instead of channelwise CUTLASS: #38006. On 32 GB GPUs the decode CUDA-graph cap rises from 24 to 48 (160 at TP >= 4) and chunked prefill from 2K to 4K: #37898
  • An undersized SWA pool now fails at startup with a ValueError instead of hanging at warmup: #37610
  • gRPC clients: GetIsReady is replaced by the WatchEngineState stream, which publishes health, pause state, instance ID, and discovery snapshots: #39915
  • Diffusion defaults change: video DiTs stay resident on GPUs with enough free memory unless --dit-cpu-offload is passed explicitly (#38001); the SDPA attention backend is chosen by measurement, retiring the no-op allow_cudnn_sdp (#38689); component loader capability switches are replaced by resolver contracts that fail closed (#36824)
  • ROCm 10 is the default AMD lane. ROCm 7.0 CI, nightly images, and the rocm700 kernel wheel are retired; ROCm 10 release images ship for MI30x and MI35x with a +rocm1000 kernel wheel, and ROCm 7.2.x images remain: #38659, #38632, #38767, #38763
  • The CPU image moves to Ubuntu 26.04 with GCC 15.2 as the default compiler: #35313
  • Contributors: Python formatting moves from black to ruff-format, with a .git-blame-ignore-revs entry for the reformat commit: #37210, #37695

New Contributors

Check out latest releases or
releases around sgl-project/sglang v0.5.20

Don't miss a new sglang release

NewReleases is sending notifications on new releases.

Get notifications