Shepherd Model Gateway v1.12.0

SMG v1.12.0 adds AMD prefill/decode support contributed by the AMD team and improves performance across gateway streaming, tokenization, engine services, cache-aware routing, and multimodal processing. It also adds Decisions and System One endpoints and fixes streaming, tool-call, and worker-management issues.

This release includes 141 commits from 19 contributors, including 9 first-time contributors.

Benchmarks below are reported in the linked PRs. They measure different workloads and should not be combined into one overall speedup. For latency, p99 means 99% of measurements were at or below that value.

Highlights

AMD prefill/decode support — contributed by the AMD team

The AMD team added support for running prompt processing (prefill) and token generation (decode) on separate AMD GPU workers, using vLLM and MoRI-IO. Thanks to @chunfangamd for the implementation, follow-up fixes, and hardware validation.

• READ and WRITE cache transfers. SMG coordinates both modes between vLLM HTTP workers, fixing the missing transfers that previously caused incorrect output or hanging requests. Streaming and non-streaming /v1/chat/completions and /v1/completions are supported. #2732.
• Data-parallel deployments. SMG assigns each request to the same data-parallel rank on both workers, distributes requests across ranks, and rejects mismatched configurations. This enables models such as DeepSeek-R1 with multiple data-parallel ranks. #2798.
• Optional concurrent WRITE dispatch. Starting both workers together lets the decode worker allocate cache while prefill runs, reducing dispatch delay. Set moriio_write_dispatch=concurrent on the decode worker to enable it. Sequential dispatch remains the default because early cancellation can leave cache memory allocated in the tested vLLM connector. #2742.
• Validated transfers. SMG checks the transfer identity, peer details, ports, cache-block lists, and rank information before accepting a transfer. If validation fails, a sequential request stops before decode begins; a concurrent request fails before its decode response is returned. These checks protect against hung requests, incorrect output, and engine crashes. #2764, #2773.

Validated on two AMD MI355X nodes, including Qwen3-0.6B with tensor-parallel sizes of 1 and 8, and DeepSeek-R1-0528 (FP8) with eight data-parallel ranks and expert parallelism. Tests checked actual cache data transferred over the network, streaming responses, concurrent requests, and failure handling. Validation.

In the reported Qwen3-0.6B test, concurrent WRITE reduced median time to first token from 71.5–72.0 ms to 46.1–46.2 ms at four concurrent requests—about 36% lower—and increased throughput from 9.89–9.98 to 10.62–10.65 requests/s. These measurements used the same two-node setup, approximately 4,350 input tokens and 64 output tokens, with three alternating rounds. Benchmark.

Faster gateway streaming and tokenization

• Less CPU per streamed request. Logging, request counting, router lookup, and cache-aware worker selection avoid repeated locking and bookkeeping. Ordinary log-level settings use the faster filter automatically; advanced span filters keep their existing behavior. #2752.
• Faster follow-up turns in long conversations. The tokenizer now caches the deepest reusable boundary after a cache hit, so each new turn can reuse the growing conversation instead of repeatedly tokenizing its earlier text. Cache storage remains proportional to prompt length. #2869.
• Faster code in release builds. Image processing, tokenization, stop matching, hashing, and routing-tree code are compiled for speed. The reported binary grew from 85.4 to 86.8 MB, or 1.6%. #2768.
• Fairer request queues under bursts. Token-bucket limits wake requests in arrival order instead of waking every waiter whenever tokens return. Cancellation and partial token returns keep the queue moving, and the caller's configured timeout controls how long a request waits. #2733.

HTTP and gRPC figures use CPU-only mock workers, 64 concurrent streams, and 64-token responses; they measure gateway capacity. The gRPC test uses a Qwen2.5 tokenizer and 32 runtime threads. The conversation test generates one token to measure the cost of starting a new turn. These are separate PR experiments with different baselines, so their gains are not combined. Sources: #2752, #2768, #2869.

Rust engine gRPC servicers

vLLM, TokenSpeed, and SGLang can now use a Rust gRPC service between SMG and the engine. Python still starts and manages the engine process. The Rust service is opt-in; Python remains the default.

To enable the Rust gRPC service:

• vLLM: pass --servicer-impl rust to vllm serve <model> --grpc or smg serve --backend vllm --connection-mode grpc. The vLLM argument is provided by the SMG plugin and needs no vLLM source changes. The environment variable SMG_VLLM_SERVICER_IMPL=rust also works with vLLM's entrypoints. #2755.
• SGLang / TokenSpeed: set SMG_SGLANG_SERVICER_IMPL=rust or SMG_TOKENSPEED_SERVICER_IMPL=rust when starting the engine service. In v1.12.0, smg serve --servicer-impl rust accepts vLLM only. Setup.

SGLang also gains a direct ZMQ connection on the same host through a plugin, without modifying SGLang. Both SGLang and TokenSpeed can use this connection through smg serve --backend sglang --connection-mode zmq or smg serve --backend tokenspeed --connection-mode zmq. This connects the Rust gateway directly to the engine, without a gRPC service or a --servicer-impl setting. SGLang connection, launcher.

Rust vLLM and TokenSpeed services now allow 30 minutes for initial model startup by default and clean up child GPU processes when startup fails. Startup and cleanup.

Supported features vary by engine. In particular, the Rust SGLang service does not yet support multimodal requests, and the Rust SGLang and TokenSpeed services do not support separate prefill/decode workers. Setup and supported features.

Reported Python-to-Rust comparisons:

vLLM comparisons ran concurrently on different GPUs of one otherwise idle host; SGLang and TokenSpeed ran sequentially on the same GPU. SGLang measurements used 0.5.20; the release images use 0.5.21. Its worker CPU was 1.6–2.7× lower across six tested workloads. TokenSpeed service CPU was 42–55% lower across three workloads.

Benefits depend on workload. The 512-stream vLLM test improved token latency but increased service CPU 17.8 → 21.5 s; a decode-heavy run lost about 8% throughput. SGLang's 128-request throughput gain increased gateway CPU 20.9 → 30.2 s while reducing worker CPU 30.0 → 13.7 s. TokenSpeed p99 time to first token at 128 concurrent requests was 430 → 439 ms. Sources: vLLM #2727, SGLang #2750, TokenSpeed #2747.

TokenSpeed's Python service also reports the loaded weight version and exposes reinforcement-learning controls and pause status through discovery. The gateway prefers the engine's reported version over a registration label. Encoding-only workers skip unsupported pause checks, and server/discovery metadata hides credentials. The live control features are not yet available in the Rust TokenSpeed service. #2666.

Multimodal improvements

• Image and video processing on the vLLM worker. --mm-processor smg fetches and preprocesses supported image/video inputs inside the Rust vLLM service. It accepts HTTP/HTTPS and data URLs; supported inputs depend on the model. Audio and per-item frame-rate/resolution overrides are outside this mode. The Rust service also supports vLLM's existing in-process and Redis processors. #2727, #2741.
• Correct pixels with less memory. SMG now knows when vLLM normalizes pixels on the GPU and sends raw pixels, fixing images that were normalized twice and interpreted incorrectly. Qwen-VL-family and MiniMax-M3 processors keep pixels as bytes and convert directly to the required format, using 75% less pixel-buffer storage than 32-bit floats. For these processors, the default gateway-to-vLLM pixel payload is also 75% smaller when the engine normalizes on the GPU. #2747.
• Repeated images avoid more work. With the optional pixel cache enabled (SMG_MM_PIXEL_CACHE_MB), a cache hit skips JPEG decoding as well as preprocessing. Repeated media within a request is matched without repeatedly hashing large base64 strings, and repeated image frames are decoded once. #2747, #2767.
• Parallel decoding with controlled CPU use. The gateway can decode multiple images concurrently in a dedicated pool, which defaults to at most eight threads and is configurable with SMG_MM_THREADS. Worker-side image preprocessing runs on the request's thread by default. Release builds also optimize image-processing code for speed. #2747, #2767, #2768.
• Faster and safer image transfer. The Rust vLLM service avoids copying tensors returned by the Python media processor, handles large image transfers more efficiently, and processes shared-memory images once when generating multiple completions. vLLM's in-process media processor warms up at startup. Cancelling a queued request no longer ties up processing capacity, and running work keeps its slot until it finishes. #2727.
• Less repeated work across prefill and decode. For supported vLLM requests with separate prefill and decode workers, the Rust service preserves the media information needed to skip fetching and preprocessing again on the decode worker. #2727, #2741.
• Settings and diagnostics. The SMG worker processor honors mm_processor_kwargs and rejects unsupported settings at startup. Logs and smg_mm_turbojpeg_available show whether the fast JPEG decoder is available. Audio/video data URLs are recognized without scanning the full payload, and HTTP media downloads use the declared length to reduce buffer growth. #2747, #2767.
• Correct vision capability. With enable_mm_embeds, vLLM respects default image/video limits: disabling video no longer hides image support, and audio support does not count as vision. #2776.
• Screenshots returned by tools reach the model. The gRPC path retains images inside Messages tool_result blocks by adding them as user image content in the model prompt; previously they were silently dropped. #2728.
• Multimodal routing fixes. API-registered workers retain their image-transfer and shared-memory settings. Routing policies for separate encoding workers now receive live load and cache state. TokenSpeed skips a pause-status check that could crash those workers. Cache-event relays preserve media identifiers and offsets. #2684, #2680, #2666, #2814.

Reported performance before and after the individual PR changes:

The serving tests used Qwen3-VL-8B-Instruct and 512×512 images. Worker rows compare the same Rust service with the SMG processor before and after #2747; the cached test used a 2 GiB pixel cache. Across the reported worker tests, CPU fell 73–82% without caching and 50–65% with caching. CPU figures are process CPU time over each test. The first three rows measure individual processing steps. The gateway preprocessing comparison also fixes incorrect pixel normalization, which changed answer lengths; request-throughput figures from that comparison are not used here. The PR reports that Qwen3-VL gateway preprocessing checks improved from 1/6 passing to 6/6 after the fix. Sources: #2747, #2767, #2768.

With vLLM's existing media processor, switching the service from Python to Rust used 20–40% less service CPU across #2727's image tests. In the 3,000-request run, CPU fell 14.63 → 9.85 s, throughput was 121.2 → 121.6 requests/s, and median time to first token was 64 → 87 ms. #2727.

Faster, more reliable cache-aware routing

The new chain index speeds up cache lookups and processes more cache updates. Enable it with --kv-index chain; the positional index remains the default and also receives correctness fixes.

Benchmarks reported in PR #2814:

• Compared with SMG's positional index: lookup p99 fell from 438 µs to 7.9 µs, about 55× faster, in an eight-worker mock test. In a separate test using the same CPU cores for each index, throughput rose from 131.6 million to 654.6 million cache-block operations per second, about 5× the throughput.
• Compared with the competitor's cache index: sustained throughput was 44–50% higher on the same host and CPU cores, using a 128-worker Mooncake trace:

At those rates, lookup p99 was 3.7 vs. 13.8 µs across sockets and 3.6 vs. 9.9 µs with local memory.

About 93% less index memory than the competitor. SMG reported 59.9 bytes per cached block per worker, versus 887 bytes—the PR's 1/14.8 comparison. Including allocation overhead, SMG used 62.5 bytes, still about 1/14 as much. The positional index used 333.9 bytes. This test shared each prefix across 128 workers; savings depend on how much prefixes are shared. These are index benchmarks, not model-token throughput or total gateway memory.

The default cache paths are faster too. Token and text-prefix lookup, block hashing, and index-size checks avoid repeated work, without changing cache matches or eviction behavior. Representative operation-level benchmarks from #2766:

Index-size checks now read a maintained total instead of scanning workers. These tests used 64 tenants and a shared 512-token prefix; they measure individual operations.

Worker selection scales better. With 1,024 workers, the chain index's full routing-decision p99 fell 499.2 → 35.6 µs; the positional index fell 591 → 170 µs. The chain run sustained 9,993 decisions/s, up from 3,465. These compare development stages within #2814 on one pinned CPU core, rather than two released binaries. Selection also spreads near-equal choices across idle workers instead of repeatedly favoring a subset. #2814.

Fresher load reports improve decisions under pressure. Rust services send load with cache events and idle heartbeats, with polling as a fallback. Reported load staleness fell 10.55 → 0.81 seconds, at about 7% additional gateway CPU in that hour-long test. In a separate eight-mock-worker Mooncake replay under cache pressure, the existing cache-aware policy's p99 time to first token fell 9,622 → 1,962 ms, goodput rose 6.29 → 6.81 requests/s, and preemptions fell 7 → 0 per run. These are mock pressure-test results, not a hardware or competitor serving comparison. #2814.

Cache reliability and diagnostics also improve:

• Correct cache matches: the positional index no longer overcounts divergent prompts or gaps. Cache events preserve media information and account for cache copies on both the GPU and host.
• Recovery after disconnects: Rust services retain bounded event history and live-cache snapshots. In a four-worker Qwen3-8B test, a restarted gateway resumed routing after 0.64 seconds and rebuilt its cache view to match engine block counts by 1.8 seconds. This restores routing knowledge; it does not rescue requests interrupted by a gateway crash. Python services do not retain this replay history.
• Health and overload protection: new requests move away from stalled or overloaded workers, and workers with empty caches receive warm-up traffic. Overload steering defaults to 80% token use or eight waiting requests. Returning 503 when all eligible workers are overloaded requires --worker-overload-shed. Detection based on stalled token progress covers regular gRPC streams and allows for prefill time; already-running stalled streams may remain until timeout.
• Routing traces: SMG_CACHE_TRACE=1 records why a gRPC worker was selected, including cache estimates, load, sticky routing, and retries. SMG_CACHE_TRACE_HEADER=1 adds a compact response header. Predicted overlap remains separate from engine-confirmed cache hits.

Sources: #2814, #2866, #2872.

Fewer logging stalls and lower idle memory

Logging to stdout no longer blocks serving threads on slow writes. In a mock replay with logs redirected to a file under heavy disk-write pressure, p99 time to first token fell 23,037 → 1,833 ms. This measures a reproduced logging stall, not normal serving latency. Dropped log lines have a counter.

SMG binaries using jemalloc now reclaim unused memory in the background: a matched eight-worker mock soak reported idle gateway memory falling 438–449 → 319 MB, with loaded memory essentially unchanged. This is separate from the chain-index memory savings. #2814.

Prefill queue limits

An optional per-worker prefill limit lets SMG queue excess requests and bound their wait time across HTTP PD, gRPC PD, and gRPC EPD. Enable it with --prefill-max-inflight-requests-per-worker; the queue defaults to 100 requests and a 60-second wait, configurable with --prefill-queue-size and --prefill-queue-timeout-secs. A full queue or expired wait returns HTTP 429 without retry. Streaming Responses now forwards that admission status before opening the stream. Queue depth, wait time, in-flight requests, and rejections have dedicated metrics. Limits apply within each gateway process and cannot be combined with the priority scheduler. Configuration.

New APIs and model support

• POST /v1/decisions supports non-streaming yes/no questions, choices, and scores through compatible HTTP backends or SGLang gRPC. Native HTTP backends can accept inline base64 images; the SGLang adapters support text only, and the gRPC path requires compatible non-reasoning templates with distinct single-token answer labels. HTTP, gRPC.
• POST /v1/systemone forwards native System One requests to HTTP backends. Details.
• Tencent Hy4 gains reasoning and streaming tool-call support, including constrained tool calls. Unrecognized reasoning_effort values follow the model template's default instead of disabling the reasoning parser. Parsers, reasoning settings, tool constraints.

Notable fixes

• HTTP streams finish reliably: hyper 1.11.1 fixes a race that could leave a client waiting after the final data chunk. Three reported 400,000-request tests completed without a hung stream; the older version left one or two unfinished requests in each run. #2751.
• SGLang startup and health: warm-up requests use valid sampling defaults, busy workers' health signals are recognized, and requests whose output cannot be forwarded are cancelled in the scheduler instead of continuing to consume GPU time. #2878, #2879, #2758.
• MCP policy and connection settings: configured approval/deny rules, trust levels, global proxies, connection limits, and call timeouts now take effect. Environment proxy settings are used when the config file does not set a proxy. #2682.
• Unknown routes get normal middleware: 404 responses now receive request IDs, CORS headers, access logs, HTTP metrics, and body limits. #2679.
• Worker attribution: successful gRPC responses include the selected worker ID, or the decode worker for prefill/decode requests, so responses can be matched to worker logs. This does not cover /v1/responses. #2848.
• Tool calls and streaming: Responses accepts continuations containing only tool results. Messages streams close each content block before opening the next and report max_tokens when a tool call is cut short. Valid zero-argument tools stream {}; incomplete MCP tool calls are no longer executed. #2723, #2715, #2718.
• Tool arguments: MiniMax M2 produces complete JSON objects when streaming. GLM preserves indentation and newlines in declared string arguments. #2745, #2744.
• Request compatibility: nullable chat booleans use their defaults, empty user content is accepted where valid, and SGLang HTTP requests preserve explicit false settings for reasoning and special-token handling. #2729, #2847, #2809.
• Errors and metrics: HTTP backend timeouts return 504. Sequential prefill/decode timing includes prompt processing, and streamed /generate responses retain prompt log probabilities. Failed Responses streams get their own counter. #2677, #2760, #2762, #2730.
• Worker configuration: API registration retains provider, retry, cache, prefill/decode pairing, polling, connection-pool, and media-transfer settings. Updating a worker preserves its circuit-breaker settings and live state. #2684, #2681.
• Separate-worker routing: prefill, decode, and encoding policies receive the configured live load, cache-event, and cross-gateway cache-sharing state. An explicit routing-key header takes precedence over a key derived from the request ID. #2680, #2721.
• Discovery: equivalent IPv6, path-prefixed, IPC, and data-parallel addresses resolve consistently. Registered metadata identifies the discovery provider, source instance, and configuration. Kubernetes reconciliation only manages its own workers and removes workers for pods that are no longer Running, even if they still report Ready. #2702, #2722.
• Deployment: Helm history storage starts correctly with PostgreSQL, Redis, and Oracle. TokenSpeed images include the matching SMG release and verify dated engine wheels instead of relying on stale base images. #2683, #2676, #2740.

Upgrade notes

• Sequential PD /generate output: streamed meta_info.finish_reason is now an object, such as {"type":"stop"}, rather than a raw string, matching parallel PD. Requested prompt log probabilities are preserved. #2762.
• Custom retry status lists: HTTP backend timeouts now return 504. Include 504 in a custom retry list if those timeouts should remain retryable. #2677.
• Engine versions used by CI and release images are now vLLM 0.31.0 and SGLang 0.5.21. The release includes smg-grpc-servicer 0.13.3 and smg-grpc-proto 0.4.24; the SGLang servicer requires SGLang 0.5.21 or later. SGLang requirement, release versions.
• The MLX extra now requires mlx-lm>=0.22.0,<0.32.
• vLLM 0.31 on H100: a pip-only installation without the CUDA compiler can fail tensor-parallel startup because the FlashInfer Hopper wheel lacks the required all-reduce kernel. The official vLLM-based SMG image includes the compiler needed to build it. #2612.
• Switching to --mm-processor smg: configure SMG's item and size limits explicitly. This mode does not apply vLLM's --allowed-media-domains restriction or derive its default item limit from --limit-mm-per-prompt. #2741.
• Overload protection and detection of stalled workers are enabled by default. Overload protection redirects traffic; rejecting requests when all workers are overloaded requires --worker-overload-shed.
• MoRI-IO workers must declare moriio_mode=read or write. For data parallelism, register each engine once with matching dp_size values; rank-expanded --dp-aware workers are unsupported. This integration covers HTTP Chat Completions and Completions, not gRPC or Responses. Concurrent WRITE remains opt-in because of the cancellation limitation above. Configuration, data parallelism.
• Rust API change: DiscoveryConfig is now a provider enum, and ServerConfig.service_discovery_config uses Option<RuntimeDiscoveryConfig>. --discovery-provider kubernetes is the new explicit selector; legacy --service-discovery, Python, and Helm usage still works. Legacy serialized configs remain readable, but YAML label values that look like numbers or booleans must be quoted. Migration.
• TokenSpeed release images now include the matching SMG release instead of the old v1.7.0 version.

Parser development and testing tools

• Symphony parser library: one shared format definition handles content, reasoning, and tool calls, and derives tool-call constraints for supported formats, with output adapters for Chat Completions, Responses, and Messages. It preserves tool-argument text and declared types. Symphony is not yet connected to the gateway in this release. Architecture.
• Faster long-tool-call parsing: a reported debug-build Symphony test of a Qwen3-8B SWE-bench 259 KB tool call fell 99.6 seconds → 45 ms with eight-byte chunks, and 800 seconds → 126 ms character by character. Fixture replay and recursive schema handling also avoid repeated work on large inputs. These are parser-library results. #2853, #2852, #2854.
• Broader reference checks: prompt rendering, tokenization, and incremental decoding are compared with Bellwether reference data. Fixing silently skipped compressed datasets expanded one reported run from 63 → 6,124 tokenizer cases and 47 → 3,688 rendering cases. CI runs pinned reference checks; the larger recorded datasets remain optional. #2782, #2817, #2834, #2822.
• Request capture: mock-worker --capture PATH records gRPC Generate prompts, sampling settings, and constraints as JSONL before responding. #2824.
• Reliable evaluation reports: BFCL comparisons read the exact category and current run's output, preventing stale or mismatched scores from appearing as regressions. BFCL and tau2 gain weekly runs with pinned dependencies, and large fixture recordings have dedicated runners. #2808, #2813, #2770, #2885.

Thanks to everyone who contributed, and welcome @ZeldaHuang, @daiyirita, @shikanggao, @garnermccloud, @chunfangamd, @LYMDLUT, @sam123456598, @nperrin-fr, and @minghe-zty!

Full changelog

What's Changed

• ci(release): build the TokenSpeed image against SMG v1.11.0 by @slin1237 in #2676
• fix(server): run unknown routes through the edge middleware by @slin1237 in #2679
• fix(mcp): build the orchestrator from the MCP config file by @slin1237 in #2682
• fix(helm): pass the launcher's history flag names by @slin1237 in #2683
• ci: hourly H100 node health monitor with issue-based alerts by @key4ng in #2587
• fix(ci): use runner 2.337.0 and document ARC schema repair by @key4ng in #2685
• fix(worker): keep the registered spec on API-created workers by @slin1237 in #2684
• fix(router): answer an upstream timeout with 504 on the HTTP path by @slin1237 in #2677
• fix(worker): keep the circuit breaker across a worker PATCH by @slin1237 in #2681
• feat(parsers): add Hy4 reasoning and streaming tool calls by @ZeldaHuang in #2643
• test(tool_parser): move the Hy4 chunk boundaries off the codespell trap by @slin1237 in #2688
• fix(policy): give PD and EPD leg policies the registry's shared state by @slin1237 in #2680
• refactor(parsers): describe reasoning_effort thinking toggles in the tokenizer, not the router by @slin1237 in #2689
• fix(ci): dispatch node health checks from cluster cron by @daiyirita in #2690
• refactor(routing): share expected-wait calculation by @shikanggao in #2691
• ci(workflows): switch Claude code review to Opus 5.5 by @slin1237 in #2697
• feat(router): add per-worker PD Prefill admission by @junliu-mde in #1961
• ci(node-health): stop opening issues for NPD conditions stuck Unknown by @key4ng in #2708
• chore(deps): bump opentelemetry-proto from 0.32.0 to 0.33.0 by @dependabot[bot] in #2713
• chore(deps): bump cc from 1.4.5 to 1.5.1 by @dependabot[bot] in #2712
• chore(deps): bump lru from 0.18.3 to 0.18.5 by @dependabot[bot] in #2710
• refactor(worker): one shared Endpoint parser and canonical EndpointKey by @XinyueZhang369 in #2702
• ci: install TokenSpeed nightly wheels by @lightseek-bot in #2714
• fix(grpc): report max_tokens and skip MCP dispatch for tool calls cut short by @yechank-nvidia in #2718
• fix(grpc): stop the open Messages content block before another starts by @yechank-nvidia in #2715
• fix(router): prioritize explicit routing keys over request ID lineage by @ekzhang in #2721
• fix(responses): accept tool-result-only continuation input by @garnermccloud in #2723
• fix(grpc): render images carried inside Messages tool results by @hello-alexmcc in #2728
• fix(protocols): accept null for the boolean chat extensions by @hello-alexmcc in #2729
• fix(middleware): FIFO notify-one wakeups for token bucket waiters with refill by @ekzhang in #2733
• feat(pd): speak the MoRI-IO KV handoff on the vLLM HTTP PD path by @chunfangamd in #2732
• feat(servicer): Rust vLLM gRPC servicer over the ZMQ engine adapter by @slin1237 in #2727
• feat(responses): type the response.failed terminal and count stream failures by @hello-alexmcc in #2730
• feat(servicer): --mm-processor smg runs smg's own media pipeline inside the Rust servicer by @slin1237 in #2741
• fix: build TokenSpeed engine images from nightly wheels by @lightseek-bot in #2740
• feat(pd): opt-in concurrent dispatch of MoRI-IO WRITE legs by @chunfangamd in #2742
• fix(tool-parser): close MiniMax M2 streamed argument objects by @ai-jz in #2745
• feat: pixels once as bytes, written per destination; the TokenSpeed servicer in Rust by @slin1237 in #2747
• fix(tool-parser): preserve whitespace in GLM string arguments by @ai-jz in #2744
• feat(zmq): SGLang over the direct ZMQ wire without touching SGLang by @slin1237 in #2749
• fix(deps): bump hyper to 1.11.1 for the chunked-terminator flush regression by @slin1237 in #2751
• feat(servicer): the SGLang gRPC servicer in Rust over the ZMQ wire by @slin1237 in #2750
• feat(grpc_servicer): the vLLM servicer switch and its flag without a vLLM change by @slin1237 in #2755
• perf(gateway): lock-free log filter and uncontended streaming hot path by @slin1237 in #2752
• fix(metrics): honest TTFT for sequential prefill/decode by @ywang96 in #2760
• fix(metrics): preserve prefill input logprobs in sequential PD by @ywang96 in #2762
• ci: retire the nightly benchmark and the dead MLX benchmark by @slin1237 in #2761
• fix(grpc_servicer): abort unrelayable SGLang outputs in the scheduler too by @slin1237 in #2758
• ci(evals): run the BFCL and tau2 A/Bs weekly, skip offline Blackwell legs, pin tau2-bench by @slin1237 in #2770
• perf(build): compile the image, tokenizer and routing-tree crates at opt-level 3 by @slin1237 in #2768
• fix(pd): refuse MoRI-IO handoffs the decode engine cannot parse by @chunfangamd in #2764
• ci: show TokenSpeed prebuilt nightly tags in fetch logs by @lightseek-bot in #2774
• perf(kv_index): fewer shared-memory writes and walks per routed request by @slin1237 in #2766
• perf(multimodal): dedup media by comparison and decode a request's images in parallel by @slin1237 in #2767
• fix(vllm): account for default multimodal limits in vision capability by @Moersity in #2776
• fix(pd): accept only integer ranks and grouped block ids in MoRI-IO handoffs by @chunfangamd in #2773
• feat(symphony): add the smg-symphony crate skeleton and its public types by @slin1237 in #2777
• refactor(discovery): provider-neutral reconciler with discovery provenance labels by @XinyueZhang369 in #2722
• feat(tool_parser): structural tag + reasoning prefix for hy_v4 by @LYMDLUT in #2790
• feat(symphony): render events as Chat Completions stream choices by @slin1237 in #2781
• feat(symphony): fold events into one Chat Completions message by @slin1237 in #2785
• feat(symphony): port the partial JSON parser from the old tool crate by @slin1237 in #2786
• feat(symphony): fold events into Responses output items by @slin1237 in #2800
• feat(ci): alert on failed runner pool records by @key4ng in #2775
• feat(symphony): outline a tool-call object as it arrives by @slin1237 in #2789
• feat(symphony): fold events into Messages content blocks by @slin1237 in #2801
• feat(pd): pin MoRI-IO legs to one data-parallel rank by @chunfangamd in #2798
• fix(bfcl): read each category's own score file in run_ab.py by @chunfangamd in #2808
• feat(symphony): stream events as Messages content blocks by @slin1237 in #2802
• feat(symphony): scan marker strings with hold-back of a half-arrived marker by @slin1237 in #2794
• feat(symphony): assemble the events of one tool call from its object by @slin1237 in #2791
• feat(symphony): stream events as Responses output items by @slin1237 in #2803
• feat(symphony): the Qwen3 format as the first Parser by @slin1237 in #2795
• test(symphony): replay bellwether's parse fixtures through the Qwen3 parser by @slin1237 in #2796
• test(symphony): the parser contract as properties over every chunking by @slin1237 in #2797
• test(gateway): replay bellwether render fixtures through the chat request path by @hello-alexmcc in #2782
• feat(symphony): count tokens into the events that carry them by @slin1237 in #2799
• feat(symphony): type a tagged parameter's value by the tool's schema by @slin1237 in #2823
• feat(mock-worker): --capture writes each Generate request as a JSON line by @hello-alexmcc in #2824
• chore(deps): bump tokenizers from 0.23.1 to 0.23.2 by @dependabot[bot] in #2825
• chore(deps): bump serde_with from 3.23.0 to 3.24.0 by @dependabot[bot] in #2827
• chore(deps): bump nixl-sys from 1.4.1 to 1.5.0 by @dependabot[bot] in #2828
• chore(deps): bump toml from 1.1.5+spec-1.1.0 to 1.1.6+spec-1.1.0 by @dependabot[bot] in #2829
• chore(deps): bump libc from 0.2.189 to 0.2.190 by @dependabot[bot] in #2826
• chore(tokenspeed): advance the engine ref to dd70f5dc by @hello-alexmcc in #2812
• chore(sglang): move the engine pin to 0.5.21 by @hello-alexmcc in #2811
• test(tokenizer): check encode and incremental decode against bellwether's fixtures by @hello-alexmcc in #2817
• feat(symphony): tagged::Assembler, the events of one tagged call as it arrives by @slin1237 in #2831
• test(bellwether): read the tree bellwether unpack writes, checked against sets.toml by @hello-alexmcc in #2834
• build(deps): llm-tokenizer's toml entry in Cargo.lock names the locked 1.1.6 by @hello-alexmcc in #2835
• feat(discovery)!: tagged discovery config with one runtime conversion by @XinyueZhang369 in #2833
• feat(symphony): Qwen3 speaks the tagged call syntax, and starts inside a thought the prompt opened by @slin1237 in #2838
• fix(grpc): include routed worker IDs in pipeline responses by @lightseek-bot in #2848
• fix(bfcl): parse only the scores of the current evaluation in run_ab.py by @chunfangamd in #2813
• feat(symphony): formats are data, and one engine runs them by @slin1237 in #2839
• feat(symphony): DeepSeek's DSML through the table, the second tagged assembler by @slin1237 in #2840
• ci: run the bellwether fixture tests on every change they judge by @slin1237 in #2822
• feat(symphony): eleven more recorded families through the tables, Seed-OSS's among them by @slin1237 in #2841
• feat(symphony): keyed arguments, the third tagged assembler, with Hy4, Ling and IQuest by @slin1237 in #2842
• feat(symphony): pythonic calls, the fourth assembler, with Olmo 3 and LFM2.5 by @slin1237 in #2843
• feat(symphony): a bare JSON list of calls, with xLAM by @slin1237 in #2844
• feat(symphony): the MiniMax M3 table, an XML tree of arguments behind a separator the table ignores by @slin1237 in #2850
• fix(protocols): accept empty user message content by @sam123456598 in #2847
• feat(grpc): carry the engine-reported weight_version and TokenSpeed's RL advertisement by @key4ng in #2666
• docs(symphony): the README's models table, written from data and checked by a test by @slin1237 in #2851
• test(symphony): the fixture replay is linear in the output's length by @slin1237 in #2852
• perf(symphony): the JSON assembler reads each byte once by @slin1237 in #2853
• fix(symphony): the DeepSeek V4.1 table is named as the vendor spells the version by @slin1237 in #2855
• fix(symphony): a kind below a wrapped or referenced schema comes from the member that has the shape by @slin1237 in #2854
• feat(kv_index): the chain index, a lossless KV-event relay with state snapshots, and liveness-aware cache routing by @slin1237 in #2814
• docs(symphony): three groups are ready, their whole recorded sets replayed with no difference by @slin1237 in #2856
• test: restrict Linux shared-memory tests to Linux by @hello-alexmcc in #2860
• test(symphony): the parity test replays a sample of each set when asked by @slin1237 in #2857
• test(engine-servicer): fix collapsible match lint in KV wire tests by @hello-alexmcc in #2859
• feat(protocol): add OpenAI Decisions API types by @hello-alexmcc in #2862
• feat(router): support Decisions on the regular HTTP router by @hello-alexmcc in #2863
• feat(cache_trace): trace final cache-aware routing decisions by @lightseek-bot in #2866
• feat(router): support native System One API by @hello-alexmcc in #2865
• feat(symphony): Kimi K3's table and its XTML assembler, whose argument tags type their values by @slin1237 in #2858
• fix(tokenizer): cache the deepest boundary on an L1 prefix hit by @nperrin-fr in #2869
• test(symphony): three recorded Qwen checkpoints join the parity harness, and a trimmed content is a corpus class by @slin1237 in #2861
• feat(symphony): the GLM table, Ling's keyed syntax with nothing between the tags by @slin1237 in #2873
• test(symphony): four more recorded Qwen checkpoints join the parity harness, one opening its thought only when asked by @slin1237 in #2874
• feat(router): support Decisions over SGLang gRPC by @hello-alexmcc in #2867
• feat(symphony): the plain table, for a template that writes the content and nothing else by @slin1237 in #2871
• feat(cache_aware): capture routing inputs before selection by @lightseek-bot in #2872
• test(symphony): the GLM cut-call test checks every byte and the cut value, and the docs name GLM's opener by @slin1237 in #2875
• fix(router): forward an explicit false on SGLang fields that default to true by @slin1237 in #2809
• docs(symphony): the models table from bellwether's five batches and the tables and rows now on main by @slin1237 in #2876
• fix(sglang): send valid sampling defaults in gRPC warmup by @minghe-zty in #2878
• feat(symphony): the structural tag derived from a table, for the JSON, GLM and DSML syntaxes by @slin1237 in #2877
• fix(sglang): handle health signals from the tokenizer socket by @minghe-zty in #2879
• docs(symphony): eighteen more groups are ready, their whole recorded sets replayed with no difference, and batch 4's sets are in by @slin1237 in #2883
• feat(symphony): Kimi K3's structural tag, derived from the table's regions and the old builder's argument shapes by @slin1237 in #2880
• refactor(symphony): the argument shapes a schema pins move out of the Kimi K3 module by @slin1237 in #2881
• ci(runners): a bellwether-record runner scale set for bellwether's recording jobs by @hello-alexmcc in #2885
• feat(symphony): keyed arguments written from the table's tags, for Ling's newlines and Hy4's suffixed tags by @slin1237 in #2882
• feat(symphony): the tagged syntax's structural tag, for Qwen 3.5 and Seed-OSS, with the value's spelling in the table by @slin1237 in #2884
• docs(symphony): the architecture in the README, for readers outside SMG by @slin1237 in #2887
• chore(vllm): move the engine pin to 0.31.0 by @hello-alexmcc in #2612
• chore(release): bump versions for v1.12.0 by @slin1237 in #2888

New Contributors

• @ZeldaHuang made their first contribution in #2643
• @daiyirita made their first contribution in #2690
• @shikanggao made their first contribution in #2691
• @garnermccloud made their first contribution in #2723
• @chunfangamd made their first contribution in #2732
• @LYMDLUT made their first contribution in #2790
• @sam123456598 made their first contribution in #2847
• @nperrin-fr made their first contribution in #2869
• @minghe-zty made their first contribution in #2878

Full Changelog: v1.11.0...v1.12.0