smg-project/smg v1.11.0 on GitHub

🚀 Shepherd Model Gateway v1.11.0 Released

OpenAI-compatible is no longer the only standard. Every major frontier vendor now publishes its own protocol contract and ships its own conformance checker to enforce it: Kimi's dynamic tools and fixed sampling parameters, MiniMax's root role and streaming rules, z.ai's thinking constraints, DeepSeek's tool-call syntax and usage placement. A gateway that flattens all of them onto the OpenAI baseline breaks each one in a different place. SMG v1.11.0 is the first gateway to honor every major frontier vendor's own spec, and the first to prove it by running the vendors' own verifiers against a released image.

This is the first pillar of the SMG 2026 H2 roadmap: per-model protocol dialects. Provider profiles for Kimi, MiniMax, z.ai GLM and DeepSeek apply each vendor's request rules, validation and sampling defaults once per request, backed by full-stack support for Kimi-K3, MiniMax-M3, GLM-5.3-Flash and DeepSeek-V4.1: prompt renderers matched to the reference encoders, tool and reasoning parsers matched to the engines, and vision preprocessors bit-identical to the checkpoints' own. Kimi's and MiniMax's official verifiers run green, and every report is public in smg-project/artifacts. This is a standing commitment, not a one-off: SMG will deliver maximum model compatibility, and "supported" will mean "conformance-green" for every vendor we list.

Alongside it: multimodal at production scale (MiniMax-M3 video, worker-side media processing so a vLLM worker can take plain URLs instead of pixel tensors, and three new vision families), prefill/decode disaggregation that pairs itself by KV transfer protocol, and nightly wheels and crates next to the nightly images. 231 commits from 27 contributors, 12 of them first-timers.

🤝 Provider Profiles and Vendor-Verified Model Support

One gateway, every vendor's dialect, without breaking OpenAI parity. The OpenAI wire format is now the lowest common denominator, not the contract. Each vendor's checker enforces rules the baseline cannot carry: Kimi requires a 400 when tools appears on a user message, MiniMax requires a root role and strict tool histories, z.ai forbids disabling thinking on the GLM-5.3 series, DeepSeek puts aggregate usage on the final finish chunk. Before this release those rules either did not exist in SMG or were being satisfied silently by the upstream vendor API, which meant a self-hosted deployment quietly failed the vendor's own tests.

How it works. SMG picks a profile from the model id and applies it at every entry point, so there is no per-model wiring in the routers. Vendor-only request fields are typed and travel with the request, but serialize to nothing for every other model, so OpenAI-only traffic is byte-for-byte unchanged. A typed thinking field gives the renderer, the reasoning parser and validation one shared answer to "is thinking on, and at what effort".

Kimi-K3, verified with Kimi-Vendor-Verifier.

Dynamic tools declared on system messages are honoured end to end: tool parsing, forced tool_choice, and the verifier's role, shape and duplicate-name rules
Sampling contract: Kimi's defaults filled in, immutable parameters rejected when changed, stream usage on by default
Prompt parity to the token: prompts are encoded piece by piece the way the checkpoint does, and every text and vision case in the vendor's prompt_tokens suite now matches Moonshot's count exactly
Structured outputs: JSON responses no longer end in a literal [EOS], and tool schemas that use $ref compile (42 schemas that failed before now pass)

MiniMax-M3, verified with MiniMax-Provider-Verifier.

Protocol: the root role, strict tool histories (unknown or unanswered tool calls and non-JSON arguments are a 400), and tool calls recognized even when the request declares no tools
Parsers: M3's namespaced tool-call markup with nested and composite schemas, empty containers, missing closing tags, false tool starts and stray thinking markers all handled
Stream QoS: delta events re-chunked into the vendor's 80–160 character band; the streaming suite goes 6/10 → 10/10 through SMG, which is better than the official API scores directly
Media contract: the 112 px short-side floor, max_long_side_pixel tiers, per-clip fps, and reference frame placement and timestamps (details in the multimodal section)

z.ai GLM and DeepSeek.

z.ai profile: thinking cannot be disabled on the GLM-5.3 series, efforts are validated per generation, clear_thinking and tool_stream are honoured, unsupported tool types and file_url parts are rejected by name, and max_tokens is checked against the worker's context window
GLM-4.7 and GLM-5 tool calls are constrained with a structural tag like the other native formats, only when the client forces a call, and reason first on a thinking prompt
DeepSeek profile: V4 and V4.1 model ids get the vendor's Chat streaming usage placement and one consistent thinking toggle across validation, rendering and parsing
DeepSeek-V4.1, the full stack: a native chat renderer with 29 reference-parity fixtures, reasoning and tool-call parsers for the new spaced DSML syntax matched to vLLM at every chunk size, the vision preprocessor, and gateway support for enable_thinking, native continuation and raw tool-call arguments
Also: Qwen XML streamed arguments survive literal braces and adjacent calls; reasoning parsers flush buffered text at stream end; unknown content[] parts pass through the HTTP path untouched; every OpenAI reasoning-effort tier is accepted on Responses

Vendor verifier results. Run on 2026-09-23 against the released image nightly-20260923-e381640 (ten commits before this tag, none touching these profiles), SMG in gRPC mode in front of vLLM nightly with smg-grpc-servicer 0.12.0. Full reports, stacks and failing-case lists: smg-project/artifacts/verification/2026-09-23-nightly-e381640.

Model	Verifier	Result
Kimi-K3 (NVFP4, TP8)	Kimi-Vendor-Verifier	`params` 18/18 · `prompt_tokens` 59/59 (exact token counts) · `k3_features` 108 pass / 18 upstream skips · `tool_call_json_schema` 408/408
MiniMax-M3 (MXFP8, prefill/decode over NIXL)	MiniMax-Provider-Verifier	pass@10: query success 1.000, tool-call match 0.981 (vendor baseline 0.988), schema accuracy 0.982 (0.989), error-only-reasoning 0 · format checks: text 156/156, image 110/110, stream 10/10, video 88/88
GLM-5.3-Flash (FP8, TP8, 1M context)	providers-verifier, z.ai golden set	prompt-token match 1.000 · query success 0.913; the rendered prompt is byte-identical to the reference template

🎬 Multimodal: Video, Worker-Side Processing, and Three New Vision Families

MiniMax-M3 vision and video, matched to the reference processor.

Images: M3's own pixel bounds and marker framing, up to 200 images per request, with max_long_side_pixel tiers that change the prompt token count as the spec says
Video: several clips per request, image and video together, per-clip fps, and frame sampling, placement, per-frame budget and timestamps that reproduce the reference processor, so the router path and a vLLM worker produce the same tokens
Long videos keep exact frame selection past ffmpeg's filter limits, and a container claiming an absurd frame count can no longer stall the decoder

Worker-side media processing: hand the worker a URL, not a tensor.

Until now the gateway fetched, decoded and preprocessed every image and video itself and shipped pixel tensors to the engine. A vLLM worker that runs vLLM's own multimodal processor can now take the media references instead.

Turn it on with --mm-processing worker on the gateway and --mm-processor inprocess or redis on the vLLM servicer. The default auto forwards references only when the model opts in and every worker of the model can take them; everything else keeps today's path unchanged, and a metric says which path each model took and why
A Redis-backed sidecar does the fetching and preprocessing next to the worker without a GPU, so the vLLM API-server process stays free; a reference sidecar ships with the servicer package
Models that opt in: Qwen3-VL (image and video), Llama 4 (image), MiniMax-M3 (image and video), GLM-5.3-Flash (image). Kimi-K3 stays on the router path for now
Hardened for real workloads: a 254 MB video that the sidecar could not return used to take 25 minutes to fail; now oversized results are a prompt 400, transport failures answer immediately, a spent sidecar budget is not retried on the same worker, transient fetch failures are retryable, and an overloaded worker sheds instead of queueing without bound
Configured with flags on both sides instead of environment variables, with the old variables kept as a deprecated fallback for one release
Covered in CI by dedicated worker-side chat and prefill/decode lanes and a parity check against vLLM's real processor

New vision families.

GLM-5.3-Flash: image and video preprocessing that follows the Hugging Face processor, including the per-frame video layout with timestamps
DeepSeek-V4.1: the model's unusual contain-fit and patch pipeline, bit-identical to the checkpoint's own image processor on twelve golden fingerprints
TokenSpeed over ZMQ: multimodal inputs now travel the direct ZMQ wire as well as gRPC

Faster and bounded.

Parallel preprocessing of the images and video clips in a request: four 9 MB images drop from 146 ms to 39 ms, and multi-clip batches gain up to 1.7×
Less repeated work: media named more than once in a request is fetched once, small multi-image requests on Qwen-VL reuse cached pixels per image, and base64 image URLs skip a full URL parse (8.5 % lower time-to-first-token summed over a 12-turn, 70-image conversation)
Half-precision media tensors between gateway and vLLM worker, roughly halving the largest part of a media request
Memory bounds: an opt-in cap on preprocessed media bytes held in flight (requests wait briefly, then get 429, or 413 when one request can never fit), ffmpeg threads bounded by the shared decode budget, and a JPEG header can no longer make the gateway allocate 12 GiB
--mm-per-request-image-limit overrides every model's built-in image limit from the gateway

🔀 Prefill/Decode Disaggregation: Pairing, Fan-Out, Fail-Fast

Prefill and decode legs now pair only when they can actually hand over the KV cache.

Pairing by KV transfer protocol: every PD worker reports its runtime, transport (NIXL or Mooncake), KV layout and engine version, and placement only pairs compatible legs. --pd-pairing-mode off|lenient|strict (default lenient) decides how missing facts are treated. A fleet with no compatible pair answers 503 no_compatible_pd_pair, and /workers shows each leg's pairing key. An engine can also declare its protocol outright through SMG_PAIRING_PROTOCOL
n>1 on SGLang and TokenSpeed fans out into one pair per sample, since those engines cannot share a bootstrap room across samples; the request used to fail or hang
Fail fast: the first failed leg returns immediately and the stranded room is released, instead of waiting out the other leg's five-minute deadline. Per-pair admission is bounded by what the decode engine can run, with a short wait (--pd-admission-wait-secs, default 30) before shedding
Streams start when decode answers, not when the prefill body ends; the prefill is drained in the background so the KV transfer is never cut short
Media processed once per pair: with worker-side media, the prefill worker processes the media and hands the decode leg a pixel-free identity
Language-model-only decode workers, the shape production fleets actually run, are detected automatically and served a stripped decode leg; the few request shapes such a worker cannot serve get a clear 400 instead of a crash
Qwen-VL over PD works again: the decode leg now receives the M-RoPE grid it needs (it was producing empty output), and per-image identity is folded into decode-side cache hashes so two images behind the same prefix can no longer alias
Restarts and updates: a prefill that restarts on the same address no longer hangs every request for eight minutes, and a PATCH /workers/{id} no longer drops the worker's KV connector and bootstrap port
Kubernetes: Pod annotations for the KV connector and engine id feed PD metadata, with configurable annotation names; vLLM's MultiConnector wrapping one PD connector is resolved correctly
Tested for real: a 4-GPU topology sweep (1p1d through 2p2d) on SGLang, vLLM and TokenSpeed checks that every leg takes traffic, KV reaches decode, concurrent requests keep their own context, abandoned streams free the engines, and a leg survives losing a worker. SGLang over NIXL and TokenSpeed PD join the CI lanes

🛣️ Routing and the HTTP Path

Cache-aware routing respects cache partitions: requests that differ in cache_salt, extra_key or LoRA adapter can never share cached blocks on the engine, so they no longer share router-side affinity either
Sticky routing keys are scoped per model, so one conversation touching several model pools keeps its pins; a body rid now wins over the header key under every policy, and an enabled routing-key override always reads the body
Round robin rotates per candidate set, which fixes two PD cohorts side by side leaving half their decodes idle
A worker without a fresh load report is scored like its peers, not as idle; sequential traffic no longer herds onto the one worker nobody has heard from
Clearer error answers: 503 worker_overload_protection_shed for a shed versus 503 no_available_workers for an outage (the gRPC path used to say 404 model_not_found), and 400 context_length_exceeded from the gateway for prompts no engine could accept, with the tighter PD leg governing
Anthropic clients: /v1/messages/count_tokens is routed on the HTTP and PD paths, and the anthropic-version and anthropic-beta headers are forwarded
Numbers arrive as sent: a client's top_p: 0.95 reaches the engine as 0.95, not 0.949999988079071, on every path including bring-your-own-key providers
Cheaper request rewriting: data-parallel and PD bodies are edited in place instead of being rebuilt through a JSON tree; a 50k-token generate request spends 469 µs instead of 947 µs, and a 1.6 MB intermediate allocation is gone
HTTP/2 to workers, per worker: --upstream-http2 now negotiates at registration, so a fleet mid-rollout mixes HTTP/1.1 and HTTP/2 workers safely and /workers shows which is which
HTTP/2 correctness: empty upstream chunks are no longer relayed as empty frames (current h2 clients dropped the connection after about 100 streamed responses), the configured 16/32 MiB windows now actually reach the wire, and plain HTTP streams disable Nagle buffering
Streaming errors keep their content type, so an SSE client sees the JSON error instead of losing it
IGW with disaggregation keeps PD and EPD semantics, role policies and readiness when service discovery enables the multi-router gateway

🧾 Responses API

Codex-grade regular path: custom tools, streamed tool calls and reasoning as paired item events, encrypted reasoning replay, merged assistant turns, tool_choice: none, image inputs, and incomplete status on length
Correct item statuses and terminal events: completed function calls are marked so, cut-off arguments are incomplete, and streams end on their JSON terminal event rather than the Chat [DONE] sentinel
MCP loops stop dispatching after truncation or failure, keep executed results, never expose server-side calls as client functions, and report usage across every model request
Namespaced tools survive declarations, calls, streaming and replay; tool results may be content arrays; web_search.external_web_access is forwarded
MCP calls are bounded: a per-call timeout (default 120 s), no replay of non-idempotent tools after a reconnect, and interactive approvals keyed per call

📊 Observability

GET /loads serves the gateway's cached fleet load in the engine's own schema, one entry per worker per DP rank plus an aggregate. It reads the snapshot the routing policies already use, so an upper gateway can poll a lower one with no fan-out to engines. /workers gains a matching engine_load field
Cache-aware routing counts its decisions by branch and records the prefix match ratio as a histogram, replacing fleet-wide debug logging
Tokenizer caches export lookups, evictions and reused bytes per layer for sizing and OOM investigations
Usage: speculative-decoding accepted and rejected token counts on TokenSpeed, SGLang and vLLM; opt-in running usage on gRPC chat streams; Kimi usage excludes the model's unbilled response-channel tokens as Moonshot does
/engine_metrics preserves colons, so vllm:* and sglang:* queries work against the gateway
PD admission waits and sheds are counted, every selected pair is logged at debug level, and /workers shows pairing keys

🧱 Workers, Discovery and Health

Service discovery split into worker reconciliation and mesh-router discovery with independent lifetimes, the first step toward file, Slurm and Consul providers (breaking for Rust API consumers; see below)
Failed workers keep being probed, so a worker restarted on the same address rejoins instead of staying dead until the gateway restarts; worker auto-recovery now defaults to on when service discovery is on
Registry races closed: hash-ring snapshots publish in order (a race could drop a healthy worker from routing), each data-parallel rank is guarded by its own revision on removal, URL reservations are released when registration fails, and removal targets resolve from registered URLs whatever the global DP setting
Load reports from disaggregated SGLang workers arrive again, and an e2e test now proves every worker delivers one

⚙️ Engines and Servicers

vLLM: POST /flush_cache works on gRPC workers across every data-parallel rank; engine-side validation errors surface as 4xx; a --language-model-only worker no longer claims vision support
SGLang: KV events are replayed from the engine's history before live traffic resumes, unsupported resume cursors are rejected cleanly, scheduler IPC is bound before startup with full cleanup on failure, and the gRPC server stops when every scheduler exits
KV events: stores whose hashes and tokens do not line up are skipped instead of publishing a wrong mapping, and an eviction race that could delete a live subtree of the token tree is closed
TokenSpeed: explicit sampling seeds are forwarded (including zero), the engine's running window is read for PD admission, and the engine bump came with a compatibility review of every imported surface
Go bindings drain buffered completion chunks before reporting EOF

🏗️ Architecture and Internals

Third-party routers in their own crate: the OpenAI-compatible, Anthropic and Gemini routers moved out of the gateway behind per-provider Cargo features (all on by default). A self-hosted build can carry no third-party proxy code at all, and refuses workers whose router it lacks
One dispatcher: the router manager became the gateway's single dispatch, choosing a router family from the routing snapshot instead of scanning every worker per request; non-routing endpoints moved out of the routers
Shared placement for single workers and PD pairs across the HTTP and gRPC families
Requests are dropped once the backend request is built, so a streaming response no longer keeps the parsed request in memory, and a retry redoes only dispatch instead of re-tokenizing
Transcription is a first-class pipeline endpoint with model knowledge in a registry
Tokenizer hot paths compiled with optimization on: Kimi-K3 encodes 60 % faster on Grace CPUs and 50 % on EPYC

⚠️ Breaking Changes and Upgrade Notes

GET /get_loads now returns the /loads body. The route stays on the admin tier and is deprecated; callers reading workers[].load should move to loads[] or aggregate
Rust API of the smg crate: the service-discovery configuration types changed shape when mesh-router discovery split out (router selector fields moved to a new mesh config, the discovery start function takes fewer arguments). CLI, Python, Go, Helm and serialized config are unchanged, which is why this ships as a minor
Python packages move together: smg-grpc-proto 0.4.22 and smg-grpc-servicer 0.13.0 carry the new proto fields for pairing, media references, cache flush and the prefill media identity. Older servicers keep working with reduced behaviour
Multimodal env vars are deprecated in favour of flags on both the gateway and the vLLM servicer; they still work for one release with a startup warning. An unreadable SMG_MM_PROCESSING now stops startup on every deployment, including HTTP-only gateways that never validated it before
Worker auto-recovery default: with service discovery on and --remove-unhealthy-workers unset, terminally failed workers are now removed so discovery re-registers them. Pass --remove-unhealthy-workers=false (Python: --no-remove-unhealthy-workers) to keep the old behaviour
MCP reconnects: a transport drop during a non-idempotent tool call now fails that call instead of silently running it twice, and every call is bounded by the new pool timeout
json_schema.schema must be a JSON object on every model and backend, including the HTTP relay
PD load accounting: on streamed HTTP PD requests the prefill's load is released when its body drains, not when the client stream ends, so prefill workers report lower load under load-based policies
GLM-4.7 constraints: tool_choice: auto and none send no grammar again; only forced calls are constrained
Routing-key override buffers every request body while enabled
Engine rejections are terminal 400s rather than 500s retried on another worker

🐛 Notable Fixes

PD: an unavailable leg is named with the same 503 code as the other routers; a drained prefill's outcome is recorded once so a prefill whose body keeps failing still trips its breaker; the in-place body editor keeps its fast path when nothing changes
Multimodal: values shared across a batch are kept whole and every batch is checked; the SGLang wire refuses tensor types it cannot name instead of misreading them; a frame budget can never raise sampling; the per-request placement log line is gone
Discovery and Kubernetes: custom KV annotation flags are honoured; PD mode survives service discovery enabling IGW; the Python launcher enables IGW for discovery like the Rust CLI
Tokenizer: skip_special_tokens is honoured in the tiktoken backend, and Kimi-K3 prompts are encoded piece by piece as the checkpoint does
Streaming: reasoning parsers flush buffered text at the end of a stream; Responses streams no longer append [DONE]; an engine's finish_reason: "error" never reaches a client
Tests deflaked: ZMQ handshake and rate-limit port races, the unseen-prefix min-load assertion, HTTP/2 unread-frame tests on macOS, PD router round-trip guards, the in-flight load burst

🏗️ Engines & Infrastructure

Engine pins: SGLang 0.5.20; TokenSpeed advanced to a September build with Torch 2.14.0+cu130 in its lane; vLLM 0.27.1 in CI, with a per-commit vLLM main wheel for the DeepSeek-V4.1 and GLM-5.3-Flash benchmark legs; the TensorRT-LLM lane pinned to cuda-bindings 13.3.1
Nightly artifacts: router images published to GHCR as well as Docker Hub; nightly wheels, sdist and all 19 crate tarballs attached to a dated prerelease on smg-project/artifacts; images, prereleases and tags pruned to a seven-day window with a dry-run default
TokenSpeed CI image: TokenSpeed prebaked into a content-addressed image and cached per node on NVMe, so the eight-minute pull and extract drops to seconds on a warm node; superseded images pruned from GHCR
Resilient runners: apt fails over to Canonical's cloud mirrors, git access to github.com is authenticated on the cluster runners, network steps retry, CPython headers and the host CUDA toolkit are guaranteed on every engine lane, UCX is pinned to the RDMA fabric devices for NIXL workers
Gating policy: the perf benchmark and third-party-dependent vendor tests are advisory, threshold misses surface as warnings, pushes to main cancel superseded runs, lane budgets carry 25 % headroom
Benchmarks: nightly BFCL and τ²-bench legs moved to Qwen3.8-27B, MiniMax-M3 (MXFP8), GLM-5.3-Flash and DeepSeek-V4.1-Flash
Coverage: PD topology sweep and nightly PD workflow, SGLang-over-NIXL lanes, worker-side multimodal lanes, load-report and reasoning-effort e2e tests, vision golden fixtures kept alive under transformers 5, the vLLM lane running the servicer unit tests
Tooling: Claude code review moved to Fable 5; the smg PyPI package links back to this repository; h2 0.4.19 plus routine dependency bumps

🧪 Experimental, Not Part of This Release's Supported Surface

Two pieces landed in the tree during this cycle and appear in the changelog below, but are not ready for use and are not covered by this release's compatibility promises:

RL control plane M1 (smg-rl): compiled in but inert unless the gateway is started with --enable-rl; the /v1/rl/* routes, the selector grammar and the fan-out semantics are all subject to change
smg-radix-tree: a standalone prefix-membership data structure with no SMG dependencies and nothing in the gateway wired to it yet; it is the first layer of a shared prefix-cache index that is still under evaluation

🙏 Welcome New Contributors

12 first-time contributors landed in this release — thank you all! @mvillmow, @syuoni, @PierreLeGuen, @Dogacel, @Hedge-Lord, @jiangwu25, @Arnavsharma2, @StarDuster, @Dovis01, @aniruddh-krovvidi, @chromecast56 and @ywang96.

Full Changelog: v1.10.1...v1.11.0

Upgrade now: pip install smg --upgrade

🐑 OpenAI-compatible was the floor. SMG speaks every frontier vendor's spec, and proves it with their own verifiers.

What's Changed

ci: publish router images to GitHub Container Registry by @slin1237 in #2348
feat(multimodal): support GLM-5.3-Flash by @FlamingoPg in #2349
perf(tokenizer): optimize hot dependencies for runtime by @chenht2022 in #2350
fix(grpc_client): preserve configured HTTP/2 windows by @Moersity in #2351
feat(health): default worker auto-recovery to the service-discovery setting by @hello-alexmcc in #2352
test(routing): deflake the unseen-prefix min-load assertion by @hello-alexmcc in #2354
fix(router): buffer bodies when routing-key override is enabled by @slin1237 in #2355
chore(deps): bump TokenSpeed to 7cd7ca0b by @hello-alexmcc in #2353
chore(ci): swap the nightly BFCL/tau2 Qwen leg to Qwen3.8-27B by @key4ng in #2356
fix(discovery): propagate PD KV metadata from Kubernetes by @ai-jz in #2358
fix(discovery): honor custom KV annotation flags by @ai-jz in #2359
fix(python): enable IGW for service discovery by @ai-jz in #2362
fix(vllm): carry multimodal identity into PD decode-side block hashes by @CatherineSue in #2365
fix(grpc): relay M-RoPE grid tensors to the PD decode leg by @CatherineSue in #2366
feat(parsers): add MiniMax-M3 tool and reasoning parsers by @slin1237 in #1815
refactor(grpc): end request ownership at request building, by construction by @slin1237 in #2251
fix(python): keep PD mode when service discovery auto-enables IGW by @CatherineSue in #2372
fix(tool_parser): treat a blank MiniMax-M3 leaf as an empty container by @key4ng in #2370
chore(pypi): link the smg package to its source repository by @slin1237 in #2379
feat(router): add --mm-per-request-image-limit override for spec image limits by @CatherineSue in #2381
chore(deps): bump lru from 0.18.2 to 0.18.3 by @dependabot[bot] in #2384
chore(deps): bump deadpool from 0.13.0 to 0.13.1 by @dependabot[bot] in #2387
chore(deps): bump uuid from 1.25.0 to 1.26.0 by @dependabot[bot] in #2386
chore(deps): bump str0m from 0.21.0 to 0.23.1 by @dependabot[bot] in #2383
fix(tool_parser): carry Kimi-K3 tool schema $defs into argument grammars by @key4ng in #2392
feat(multimodal): add MiniMax-M3 vision support by @key4ng in #2371
fix(router): preserve disaggregated semantics under IGW by @ai-jz in #2364
fix(router): preserve streaming error content type by @mvillmow in #2404
fix(ci): authenticate github.com git access on cluster runners by @hello-alexmcc in #2405
chore(ci): prebake TokenSpeed into a CI image to cut ~25 min/lane of GPU-idle compile by @key4ng in #2393
feat(router): negotiate HTTP/2 per worker under --upstream-http2 by @slin1237 in #2407
fix(ci): consume the TokenSpeed CI image by extraction — lanes run bare, not in containers by @key4ng in #2409
fix(protocols): accept every OpenAI reasoning effort tier on Responses by @slin1237 in #2410
fix(ci): add python3-dev to the TokenSpeed CI image base packages by @key4ng in #2415
ci: stop gating merges on the perf benchmark and third-party tests by @slin1237 in #2412
fix(ci): install Python dev headers on the runner for TokenSpeed lanes by @slin1237 in #2419
fix(router): distinguish worker overload shed responses by @frankzhouhr in #2417
fix(multimodal): tighten MiniMax-M3 video grid, cap, and limit validation by @key4ng in #2416
test: deflake ZMQ handshake and rate-limit tests; probe PD gateways before testing by @slin1237 in #2414
ci: give GPU lanes timeout headroom and retry network steps by @slin1237 in #2413
refactor(grpc): make transcription a first-class pipeline endpoint by @hello-alexmcc in #2398
fix(grpc): bind scheduler IPC before startup by @Moersity in #2427
fix(vllm): resolve PD connector from MultiConnector config by @Moersity in #2425
ci: install RDMA runtime libraries on the TokenSpeed prebuilt path by @slin1237 in #2430
fix(tool_parser): resolve MiniMax-M3 composite property schemas by @Moersity in #2422
ci: prune superseded TokenSpeed CI images from GHCR by @slin1237 in #2431
ci: install the gh CLI before pruning TokenSpeed images by @slin1237 in #2432
feat(api): serve engine load from the gateway's cached snapshot by @slin1237 in #2418
fix(tokenizer): Kimi-K3 prompt segments and skip_special_tokens in the tiktoken backend by @syuoni in #2421
refactor(tokenizer): replace the Kimi-K3 prompt-segment API with a deferred encode by @CatherineSue in #2434
fix(tool_parser): flush false minimax-m3 tool starts by @Moersity in #2423
feat(router): key cache-aware affinity under the request's cache partition by @hello-alexmcc in #2435
refactor(gateway): move non-routing endpoint handlers out of routers by @hello-alexmcc in #2440
refactor(gateway): serve /v1/models from an endpoint module instead of the router manager by @hello-alexmcc in #2445
refactor(gateway): move the SGLang field list out of the OpenAI provider router by @hello-alexmcc in #2442
fix(mcp): bound tool calls and stop replaying non-idempotent calls on reconnect by @hello-alexmcc in #2441
refactor(gateway): share single-worker placement between the HTTP and gRPC families by @hello-alexmcc in #2444
fix(reasoning-parser): strip MiniMax M3 close markers by @Moersity in #2448
feat(routers): compile provider routers behind per-provider Cargo features by @hello-alexmcc in #2449
refactor(gateway): share prefill/decode pair placement and slice-based selection by @hello-alexmcc in #2446
feat(external-router): move the third-party routers into their own crate by @hello-alexmcc in #2450
refactor(gateway): rename the router manager to gateway.rs with one dispatch by @hello-alexmcc in #2452
fix(protocols): preserve Responses tool namespaces and content-array outputs by @PierreLeGuen in #2451
fix(grpc_servicer): read SGLang's renamed prefill queue counter in GetLoads by @hello-alexmcc in #2458
ci(benchmarks): report threshold misses as warnings instead of failures by @hello-alexmcc in #2457
fix(policies): score a worker without a load report like its peers by @hello-alexmcc in #2454
ci(multimodal): keep the vision golden fixtures under transformers 5 and fail loudly by @hello-alexmcc in #2459
test(e2e): assert every worker delivers load reports to the gateway by @hello-alexmcc in #2460
test(e2e): cover reasoning effort tiers and streaming error shapes by @hello-alexmcc in #2461
fix(router): answer 503 on the gRPC path when a model's workers are all unavailable by @hello-alexmcc in #2465
ci(e2e): run the gRPC PD suite on TokenSpeed by @hello-alexmcc in #2463
ci(e2e): stop pinning TokenSpeed's sequence window in the Qwen3.5-9B spec by @hello-alexmcc in #2467
chore(deps): bump toml from 1.1.4+spec-1.1.0 to 1.1.5+spec-1.1.0 by @dependabot[bot] in #2474
chore(deps): bump tokio-rustls from 0.26.4 to 0.26.5 by @dependabot[bot] in #2470
chore(deps): bump wasm-encoder from 0.255.0 to 0.257.1 by @dependabot[bot] in #2475
fix(worker): keep probing a failed worker so a restart on the same address rejoins by @hello-alexmcc in #2468
fix(pd): fail fast on a failed leg and admit rooms the decode can take by @hello-alexmcc in #2466
feat(protocols): pass through unknown chat content parts by @ekzhang in #2469
test(e2e): size the in-flight load burst to outlast the poll, and widen the gateway lane budget by @hello-alexmcc in #2476
ci: widen the vLLM 1-GPU chat lane budget by @hello-alexmcc in #2477
ci: widen the Go bindings e2e job budget by @hello-alexmcc in #2478
fix(pd): name an unavailable PD leg with the same 503 code as the other routers by @hello-alexmcc in #2479
fix(pd): fan an n>1 request out into one PD pair per sample on room-based engines by @hello-alexmcc in #2482
feat(pd): surface each engine's KV transfer protocol facts as worker labels by @hello-alexmcc in #2484
fix(http): do not relay empty upstream chunks as empty h2 DATA frames by @slin1237 in #2488
feat(pd): pair prefill and decode workers by KV transfer protocol by @hello-alexmcc in #2486
feat(rl): RL control plane M1 — worker discovery, engine-route passthrough, and fan-out behind --enable-rl by @key4ng in #2411
feat(radix-tree): chain-native prefix-membership index by @slin1237 in #2436
chore(deps): bump wat from 1.257.1 to 1.258.0 by @dependabot[bot] in #2471
ci: pin cuda-bindings for the TensorRT-LLM lane by @hello-alexmcc in #2494
fix(pd): re-read a recovered PD worker's KV engine id before it is promoted by @hello-alexmcc in #2492
fix(tool_parser): preserve Qwen XML streamed arguments by @ai-jz in #2490
fix(kv-events): skip stores without dense block-to-token alignment by @ai-jz in #2495
test(e2e): sweep prefill/decode topologies on 4 GPUs for every engine by @hello-alexmcc in #2464
ci: run SGLang over NIXL in the PD lanes by @hello-alexmcc in #2501
feat(pd): let the engine report its pairing protocol from SMG_PAIRING_PROTOCOL by @hello-alexmcc in #2504
fix(policies): keep round robin's rotation per candidate set by @hello-alexmcc in #2507
feat(protocols): provider extension modules for Kimi and MiniMax fields by @CatherineSue in #2374
ci: fail apt over to Canonical's cloud mirrors by @hello-alexmcc in #2510
feat: populate speculative decoding acceptance metrics in OpenAI usage by @Dogacel in #2481
feat(policies): count cache-aware tree decisions by branch and match ratio by @Hedge-Lord in #2515
fix(worker): preserve PD state across property updates by @jiangwu25 in #2516
fix(go): drain buffered completion chunks before EOF by @Arnavsharma2 in #2519
fix(e2e): pin UCX to the RDMA fabric devices for NIXL workers by @hello-alexmcc in #2520
fix(policies): honor the body rid over the header under manual and consistent_hashing by @ighutake-debug in #2517
feat(multimodal): raise short sides to MiniMax-M3's 112 px floor by @CatherineSue in #2513
feat(protocols): provider profiles with Kimi dynamic-tool role rules by @CatherineSue in #2388
feat(protocols): MiniMax tool-history strictness rules by @CatherineSue in #2389
feat(protocols): MiniMax root role with system normalization by @CatherineSue in #2390
fix(worker): resolve removal targets from registered URLs by @StarDuster in #2522
fix(reasoning): keep buffered text at stream end by @jiangwu25 in #2523
fix(responses_api): correct regular-path tool, reasoning, streaming and replay handling by @Dovis01 in #2502
feat(tokenizer): add the DeepSeek-V4.1 native chat renderer with reference parity fixtures by @slin1237 in #2524
feat(multimodal): add the DeepSeek-V4.1 vision preprocessor pinned to the reference image processor by @slin1237 in #2526
feat(parsers): add the DeepSeek-V4.1 reasoning and spaced-DSML tool-call parsers by @slin1237 in #2525
feat(protocols): Kimi sampling defaults and immutable-parameter pins by @CatherineSue in #2512
test(grpc_client): pin the advertised HTTP/2 window and reproduce too_many_data_frames by @vschandramourya in #2531
test(e2e): disable thinking in TokenSpeed PD streaming test by @key4ng in #2533
feat(gateway): DeepSeek-V4.1 encoder input key, renderer capabilities and native continuation by @slin1237 in #2527
feat(tool_call): constrain non-strict GLM47 tool calls with EBNF by @Dovis01 in #2509
perf(server): disable Nagle buffering for plain HTTP streams by @yechank-nvidia in #2534
ci(benchmarks): keep the tokenizer benchmark's cache warm and give it headroom by @slin1237 in #2537
feat(external-router): document the crate and judge discovered models at admission by @hello-alexmcc in #2455
feat(multimodal): support multimodal inputs over the TokenSpeed ZMQ backend by @slin1237 in #2539
chore(deps): bump cc from 1.4.4 to 1.4.5 by @dependabot[bot] in #2543
chore(deps): bump rustls from 0.23.43 to 0.23.44 by @dependabot[bot] in #2544
chore(deps): bump serde_with from 3.22.0 to 3.23.0 by @dependabot[bot] in #2545
chore(deps): bump nixl-sys from 1.4.0 to 1.4.1 by @dependabot[bot] in #2546
fix(tool_parser): send no grammar for a GLM-4.7-family chat request without tools by @slin1237 in #2549
fix(ci): cache the extracted TokenSpeed payload per node on NVMe by @key4ng in #2536
test(grpc_client): use in-memory transport for unread frame tests by @Moersity in #2557
feat(vllm): support gRPC prefix cache flush by @Moersity in #2554
ci(bench): move the BFCL and tau2 DeepSeek legs to V4.1-Flash on a per-commit vLLM wheel by @key4ng in #2551
fix(tool_call): constrain GLM-4.7 tool calls with a structural tag, only when forced by @slin1237 in #2550
ci(workflows): switch Claude code review to Fable 5 by @key4ng in #2560
fix(ci): guarantee CPython dev headers for every engine lane's venv by @key4ng in #2558
fix(ci): put the host CUDA toolkit on PATH so vLLM can detect FlashInfer by @key4ng in #2566
fix(tool-parser): recover missing empty container closing tags by @Moersity in #2567
fix(router): map engine validation errors to 4xx and reject error finish reasons by @CatherineSue in #2565
fix(protocols): reject non-object json_schema.schema with 400 by @CatherineSue in #2562
fix(multimodal): wrap MiniMax M3 video frames with the image markers by @CatherineSue in #2559
ci(bench): move the BFCL and tau2 GLM legs from GLM-5.2-FP8 to GLM-5.3-Flash by @key4ng in #2552
fix(tokenizer): K3 prompt parity with Moonshot by @CatherineSue in #2564
fix(protocols): Kimi stream usage default and dynamic tool validation by @CatherineSue in #2561
ci(bench): move the BFCL and tau2 MiniMax legs from M2.7 to the M3 MXFP8 checkpoint by @key4ng in #2553
fix(kv-index): detach a token-tree node only when the parent still holds it by @hello-alexmcc in #2568
fix(multimodal): cap decoded size in the turbojpeg path by @aniruddh-krovvidi in #2569
feat(protocols): typed thinking field with toggle and effort projection by @CatherineSue in #2563
fix(protocols): honor dynamic tools declared on messages after validation by @slin1237 in #2570
fix(tokenizer): count the tokenizer's own eos_token as an EOS id by @slin1237 in #2571
feat(router): SSE delta re-chunking for MiniMax stream-QoS contract by @CatherineSue in #2391
fix(worker): release the url reservation when AddWorker fails by @aniruddh-krovvidi in #2572
fix(router): scope sticky routing override pins by model by @ai-jz in #2576
feat(multimodal): per-frame MiniMax M3 video budget and exact sampled frame metadata by @CatherineSue in #2575
feat(multimodal): stamp MiniMax M3 video frames with the reference timestamps by @CatherineSue in #2574
feat(vllm-grpc): media_refs wire contract and worker-side media processing by @CatherineSue in #2399
feat(router): forward media references to vLLM workers that process them by @CatherineSue in #2400
test(e2e): worker-side multimodal processing lane and anchor parity check by @CatherineSue in #2530
perf(multimodal): bound ffmpeg decoder threads by the shared decode budget by @slin1237 in #2583
perf(multimodal): preprocess the images of a batch in parallel by @slin1237 in #2582
feat(multimodal): MiniMax M3 video, batched clips, image+video on vLLM gRPC, 404 by @slin1237 in #2581
feat(grpc): add opt-in continuous chat usage stats by @Moersity in #2556
feat(vllm-grpc): Redis media-processing sidecar backend and reference sidecar by @CatherineSue in #2401
perf(multimodal): fetch media named more than once in a request only once by @slin1237 in #2586
fix(ci): let the vLLM lane collect and run the servicer unit tests by @CatherineSue in #2589
feat(multimodal): sample video at the model's reference rate by default by @slin1237 in #2579
fix(multimodal): keep batch-shared values whole and check every batch by @slin1237 in #2588
feat(servicer): accept half-precision media tensors by @slin1237 in #2590
feat(multimodal): place video frames the way the model's reference does by @slin1237 in #2584
feat(router): send vLLM media at the width the worker reads by @slin1237 in #2591
feat(router): cap the preprocessed media bytes held in flight by @slin1237 in #2585
fix(router): give the SGLang test payload its media width field by @slin1237 in #2592
chore(release): smg-grpc-proto 0.4.20 and smg-grpc-servicer 0.12.0 by @slin1237 in #2578
fix(multimodal): batch MiniMax-M3 videos sampled at different rates by @slin1237 in #2593
chore(multimodal): drop the per-request multimodal placement log by @slin1237 in #2594
perf(multimodal): process video clips in a batch in parallel by @slin1237 in #2595
fix(vllm-servicer): retryable media failures, real limits, load shedding, and quieter probes by @slin1237 in #2596
fix(multimodal): keep exact frame selection for longer videos by @slin1237 in #2597
fix(grpc_servicer): keep redis reads alive for the wait they were asked to make by @slin1237 in #2598
fix(multimodal): size the frame-selection check by the frames asked for by @slin1237 in #2600
fix(grpc_servicer): refuse media tensors the SGLang wire cannot name, and report a stuck scheduler once by @slin1237 in #2599
fix(multimodal): accept narrow minimax m3 video frames by @Moersity in #2601
feat(tokenizer): expose cache activity metrics by @Moersity in #2603
refactor(discovery)!: split worker reconciliation and mesh-router discovery by @XinyueZhang369 in #2532
perf(multimodal): cache individual images in small requests by @Moersity in #2602
feat(multimodal): let vLLM workers expand MiniMax-M3 media anchors by @slin1237 in #2609
chore(tokenspeed): advance the engine pin and move the Torch pin with it by @hello-alexmcc in #2610
ci: prune nightly images, releases and tags to a retention window by @hello-alexmcc in #2617
ci: publish nightly wheels and crate tarballs as a dated prerelease by @hello-alexmcc in #2616
chore(sglang): move the engine pin to 0.5.20 by @hello-alexmcc in #2611
fix(tokenspeed): forward explicit sampling seeds by @lightseek-bot in #2622
ci: publish nightly artifacts to smg-project/artifacts by @hello-alexmcc in #2619
fix(worker): serialize hash ring snapshot publication by @lightseek-bot in #2621
fix(worker): guard each DP rank by its own revision on removal by @XinyueZhang369 in #2620
fix(gateway): reject over-long prompts with 400 context_length_exceeded by @ighutake-debug in #2618
fix(sglang): stop server when schedulers exit by @Moersity in #2428
feat(multimodal): let vLLM workers expand Kimi-K3 and GLM-5.3-Flash image anchors by @hello-alexmcc in #2623
fix(multimodal): answer undeliverable sidecar results and stop retrying spent budgets by @hello-alexmcc in #2624
chore(deps): bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #2628
chore(deps): bump deadpool-redis from 0.23.0 to 0.23.1 by @dependabot[bot] in #2633
chore(deps): bump bitflags from 2.13.1 to 2.13.2 by @dependabot[bot] in #2630
feat(gateway): expose the multimodal placement and media settings as CLI flags by @hello-alexmcc in #2625
fix(metrics): preserve colons in /engine_metrics metric names by @chromecast56 in #2636
fix(router): forward client JSON numbers exactly on HTTP and PD paths by @chromecast56 in #2637
fix(pd): serve multimodal requests on language-model-only vLLM decode workers by @ywang96 in #2640
feat(servicer): take the media-processing settings from launcher flags by @hello-alexmcc in #2626
feat(protocols): add the z.ai (GLM) provider profile by @hello-alexmcc in #2634
feat(router): route /v1/messages/count_tokens and forward Anthropic headers by @chromecast56 in #2638
perf(pd): commit the client stream on decode's 2xx, not the prefill body by @chromecast56 in #2639
fix(pd): process worker-side media once, on the prefill leg by @hello-alexmcc in #2627
perf(router): edit DP and PD request bodies as raw slices by @slin1237 in #2641
perf(multimodal): fast-path canonical image data URLs by @Moersity in #2648
fix(protocols): match the z.ai profile's GLM-5.3 family and effort sets to the vendor by @hello-alexmcc in #2657
fix(servicer): lead the media-processor messages with --mm-processor by @hello-alexmcc in #2654
fix(multimodal): keep Kimi-K3 media on the router path by @hello-alexmcc in #2644
fix(router): keep RawBody's fast path when an insert changes nothing by @hello-alexmcc in #2663
fix(servicer): read vLLM's video frame default from vllm.multimodal.media by @hello-alexmcc in #2649
fix(pd): apply the prefill media identity whole or not at all by @hello-alexmcc in #2656
fix(servicer): drop the sidecar's --mm-sidecar-timeout-ms, which applied to nothing by @hello-alexmcc in #2651
fix(servicer): never fail a served request on a media identity it cannot build by @hello-alexmcc in #2652
fix(router): answer count_tokens without a router with the NO_ROUTER 404 by @hello-alexmcc in #2659
fix(router): forward exact numbers on the external-provider paths by @hello-alexmcc in #2658
test(router): hang guards instead of 5 s budgets around the PD router's round trips by @hello-alexmcc in #2660
fix(pd): record the drained prefill outcome once by @hello-alexmcc in #2662
fix(pd): carry the video timing under both of its names in the media identity by @hello-alexmcc in #2653
fix(kv-events): reject unsupported sglang replay cursors by @Moersity in #2672
fix(responses): align tool statuses and streaming terminal events by @key4ng in #2665
feat(kv-events): replay sglang events before resuming live traffic by @Moersity in #2673
fix(responses): guard MCP dispatch and preserve failure details by @key4ng in #2674
feat(protocol): add DeepSeek V4 profile and Chat stream usage by @key4ng in #2671
chore(release): bump versions for v1.11.0 by @slin1237 in #2670

New Contributors

@mvillmow made their first contribution in #2404
@syuoni made their first contribution in #2421
@PierreLeGuen made their first contribution in #2451
@Dogacel made their first contribution in #2481
@Hedge-Lord made their first contribution in #2515
@jiangwu25 made their first contribution in #2516
@Arnavsharma2 made their first contribution in #2519
@StarDuster made their first contribution in #2522
@Dovis01 made their first contribution in #2502
@aniruddh-krovvidi made their first contribution in #2569
@chromecast56 made their first contribution in #2636
@ywang96 made their first contribution in #2640

Full Changelog: v1.10.1...v1.11.0