Shepherd Model Gateway v1.14.0

SMG v1.14.0 makes Symphony the default parser, brings the Rust engine servicers to every engine, and serves TokenSpeed over the direct ZMQ lane. On the same hosts it sustains up to 3.3× the request rate of v1.13.0 within the same latency target, and the router uses 21–59% less CPU per request on every model family we measured.

1,066 commits from 17 contributors, 6 of them first-time. Some defaults changed and some deployments need a step before upgrading: read Before you upgrade first.

Before you upgrade

• Symphony is the default parser. --parser legacy (or the request header X-SMG-Parser: legacy) keeps the old parsers. Existing --tool-call-parser and --reasoning-parser names keep working.
• Three flags are removed. --kv-index, --kv-indexer-ttl-secs and --kv-indexer-max-entries are gone; the chain index is the only KV index. A config that still sets them does not start. A per-policy load_check_interval_secs is refused; use load_monitor_interval_secs.
• Engine pods take longer to become Ready, on purpose. Servicers warm up every batch size and prompt length before they report SERVING (about a minute at a 16k-token admission step; SMG_<ENGINE>_SERVICER_WARMUP=fast or off shortens or skips it). The gateway registers a gRPC worker only once its engine is healthy. Raise startup probes and rollout deadlines to match; the chart's router.progressDeadlineSeconds is now 1920 s.
• Upgrade order. Roll gateways first, then engine pods with smg 1.14.0 and smg-grpc-servicer 0.15.0.
• tool_choice: "none" now renders the tools in the prompt, as the engines' own servers do. Prompt token counts and prefix-cache keys for such requests change once.
• The gateway's API key is no longer forwarded upstream. Workers and external providers that relied on it need their own api_key at registration.
• Helm. The API key, the Postgres and Redis URLs and the Oracle credentials come from Secrets (auth.existingSecret, history.<store>.existingSecret). Worker images default to ghcr.io/smg-project/smg. Discovery providers are configured under router.discovery.
• Python bindings. Router(...) and RouterArgs(...) take every parameter after (worker_urls, policy, host, port) and (worker_urls, host, port) as keyword-only.
• Metrics. Six unused metric families are removed, the error counter gains a reason label, and new latency histograms arrive. Details under Upgrade notes.

Highlights

Faster at the shipped defaults

We ran v1.13.0 and v1.14.0 as shipped: one engine at tensor parallel 4 and one router per side, same hosts, same 8k-token chat workload (4k-token shared prefix, 128 output tokens). The table shows the highest request rate each build sustains while time-to-first-token p99 stays under 1 second.

Where the gain comes from:

• Engine step shape. A vLLM gRPC launch now derives the CUDA-graph capture cap from the admission step when the operator sets none. A step that mixes a prefill chunk with decodes replays a captured graph instead of running eagerly. Families whose recipes already captured those steps read equal capacity with lower tails.
• Router cost. A chat request costs 0.37M user-space instructions in the router, down from 2.04M. A tool-calling request costs 3.17M (was 5.98M), a 128-delta stream 1.75M (was 5.28M). About half of the saving is the release profile now optimizing for speed. The rest is per-request work removed: one middleware stack per route group, tool schemas and template values prepared once per tool set, Symphony tables built once per process, batched log writes.
• Engine-side CPU. Under the Rust servicer an engine pod spends 1.3–1.5 ms of CPU per request where the Python servicer spent 7.5–12.2 ms (vLLM). The gap is larger on SGLang prefill/decode pairs and on TokenSpeed image prompts.

Eight engines behind one router

Same workload, eight engines at tensor parallel 4 and one front node per side, against another open-source router with KV-aware routing on its bundled build of the same vLLM release:

• SMG sustains 264–272 req/s within the latency target. The other router sustains 240 on a gentle ramp and none on a warm one.
• At 232 req/s: TTFT p50/p99 76/128 ms against 538/1,627 ms, TPOT p99 26.5 against 62.2 ms, goodput 1.00 against 0.42.
• The SMG front node uses a third of the CPU and a fifth of the memory per request: 8.5–8.7 against 24.4–29.4 ms of CPU, 474–571 MB against 2.7 GB.
• Against the engine's own HTTP server on one node, throughput is equal within 1% and SMG's TTFT p50 is 4–19% lower.

The router's own share of time to first token is about 2 ms at every load step: 1.2–1.3 ms from receive to dispatch and 0.6–0.7 ms from dispatch to the first engine frame.

The configuration behind those numbers

# Engine host: vLLM behind SMG's Rust gRPC servicer
vllm serve <model> --grpc --port 50051 --servicer-impl rust --metrics-port 29100

# Router: cache-aware routing on the chain index, fed by the engine's KV-cache events
smg launch --worker-urls grpc://engine-0:50051 grpc://engine-1:50051 \
  --policy cache_aware --block-size 256

SGLang (sglang serve --grpc-mode --servicer-impl rust), TokenSpeed (python -m smg_grpc_servicer.tokenspeed --servicer-impl rust) and TensorRT-LLM (SMG_TRTLLM_SERVICER_IMPL=rust trtllm-serve <model> --grpc --grpc-protocol smg) take the same switch. Python remains the default servicer in 1.14.0.

Symphony is the default parser

Symphony reads model output through one table per format and emits content, reasoning and tool calls as typed events. It was opt-in in v1.13.0.

• Existing flags keep working. A legacy parser name whose format a table reads selects the table. The twelve names no table answers to keep the legacy parsers (see Known limits).
• Eight new tables: DeepSeek V3, V3.1 and V3.2, DeepSeek V4-Flash, Kimi K2, Llama 4, the Mistral call list, and Muse Spark. Many more families are read by model id with no parser flag.
• Cut calls stay whole. A tool call cut by the token limit keeps its partial arguments and reports finish_reason: length. Nothing leaks into content.
• Tool call ids are unique per request, so multi-turn agents pair results with the right calls.
• Forced calls work on every table. A constraint the engine's backend cannot enforce is refused with 400 before the engine.
• Failures are visible. smg_symphony_outputs_total, smg_symphony_malformed_total{reason} (28 reasons) and smg_symphony_refused_total exist at zero from a model's first registration, with one sampled log line per malformed output.
• No dearer than the legacy parsers. Gateway CPU per request is flat on chat and 20–50% lower on agentic turns and streamed tool calls.

Rust servicers for every engine

• TensorRT-LLM joins (SMG_TRTLLM_SERVICER_IMPL=rust): generation with logprobs, string and token stops, images, guided decoding, LoRA, n > 1, and the disaggregated parameters. What the wire cannot carry is refused by name.
• SGLang gains prefill/decode disaggregation, the language side of encode/prefill disaggregation, multimodal input, hidden states, data-parallel rank pins, and SGLang 0.5.22's request structs.
• TokenSpeed gains prefill/decode disaggregation, Qwen3.5-lineage image and video with the MRoPE position table computed bit for bit, audio, and FlushCache, StartProfile and StopProfile.
• One /metrics endpoint on every wire (--metrics-port): requests by finish reason, refusals, errors, in-flight requests, token counts, TTFT, inter-token and end-to-end histograms, and lifecycle state. The Helm chart scrapes it through workers[].metrics.
• Ready means warm. Both servicer implementations run a planned warm-up before SERVING, so the first request at each new shape no longer pays the engine's kernel tuning. SMG_<ENGINE>_SERVICER_WARMUP sizes it.

TokenSpeed on the direct ZMQ lane

The gateway's same-host engine link now serves TokenSpeed engines in every role (prefill, decode, encode). It carries prompt logprobs, the control calls (/flush_cache, /start_profile, /stop_profile, /v1/rl/*), tcp:// workers beside ipc://, and images. Throughput matches the gRPC servicer path within 1%. The handshake bound is a flag now, --zmq-startup-timeout-secs (3600 s).

Tokenizer

• Tool-calling requests encode with 24.5% fewer instructions and chat requests with 13% fewer. An exact-match cache hit on an 80k-token prompt costs a third of what it did.
• Prompts of 64 KiB and more tokenize on the thread pool with identical ids: a 1,054,000-token prompt takes 39 ms on eight threads instead of 251 ms.
• Streaming decode drops from 25.9 to 17.7 ns per token.
• Exactness fixes: Llama-class legacy conversions, templates shipped as a list, byte-exact template files, Python-style tojson, join and replace, and developer messages on the DeepSeek and Kimi K3 encoders.

Multimodal

• The one-pass wire write is the default: faster in 32 of 33 benchmark rows by 6–60%. The DeepSeek V4.1 preprocess runs 1.5–2.9× faster per image, bit-equal on 440 of 440 images.
• New and corrected families: Muse Spark Open (image, video, audio), native GLM-5.3 and GLM-5.3-Flash renderers (Flash images are no longer normalized twice), GLM 5 video sampling, Nemotron-H Omni, 16-bit greyscale PNGs, and exact frame picks at rates binary32 cannot hold.
• Worker-side processing for every family with --mm-processing worker|auto.
• Bounded and explicit: --mm-fetch-timeout-ms, --mm-video-fetch-timeout-ms, --mm-turbojpeg require, pixel-budgeted video decode, and early refusal of a modality no worker takes. A failing origin is a 502, a bad URL a 400.

Operations

• Readiness follows the engine. Workers register once their engine is healthy. The readiness body names each model's state and reason. Prefill/decode deployments can require minimum ready counts per role.
• Liveness tells a long prefill from a dead engine. SMG_<ENGINE>_SERVICER_SILENCE_TIMEOUT_SECS with an allowance per accepted prompt token.
• A separate control plane. --admin-port serves the admin, worker-registry and RL routes plus the health endpoints on their own listener. --max-connections bounds the data listener. GET /inflight snapshots the accepted requests.
• Secrets off the command line. SMG_API_KEY, SMG_TENANT_API_KEYS, SMG_POSTGRES_DB_URL, SMG_REDIS_URL, --api-key-file and the *_FILE variables. --help no longer prints credentials.
• Shutdown. --shutdown-settle-secs (5 s) keeps accepting after readiness fails. A stopping gateway leaves the mesh before it drains.
• Data parallelism. Under --dp-aware the ranks of one engine share one load poll, each rank indexes its own KV events, and a conversation stays on the rank that holds its prefix.
• Prefill/decode across hosts over TCP. The engine image sets UCX's TCP segment size, so KV pulls run at line rate: TTFT p50 on a two-host pair at 4 req/s went from 33 s to 0.35 s.

Observability

• New log-spaced histograms: smg_router_time_to_first_token_seconds and smg_router_request_time_per_output_token_seconds.
• smg_router_request_errors_total{reason} names the cause of each error. smg_worker_state{reason} says why a worker is out of rotation. Cached prompt tokens, KV capacity and block size are on /metrics.
• The Helm dashboard grows from 4 panels to 41, with ten alert rules under prometheusRule.enabled.
• --access-log-fields extends the access log. Every log line masks worker credentials. The writer threads batch lines. --enable-trace exports spans at every log level.

Discovery

Consul (--discovery-provider consul --consul-service <name>) and Slurm (--discovery-provider slurm) join Kubernetes and file discovery. The chart selects them under router.discovery.

APIs

• Chat Completions: logprobs cover every generated token on the gRPC path. thinking: {"type": "disabled"} wins over reasoning_effort. add_generation_prompt is read. GET /v1/models lists aliases.
• Messages: thinking.display: "omitted" and "updates" are accepted, as is tool_reference in tool results. Upstream errors and broken streams are counted.
• Responses: a long conversation replays its newest items from a turn boundary and keeps the stored tool calls. tool_choice: "auto" without tools and client_metadata are accepted. A stream stopped at max_tool_calls ends with response.completed and is stored.
• Every refusal is a JSON error object with an error.code. Engine "not found", "unavailable", "too many requests" and timeout answers map to 404, 503, 429 and 504.

How this release was validated

• Provider verification suites for DeepSeek V4 and V4.1, Kimi K3 and MiniMax M3, through both parsers: 8 of 8 rows pass with no regression.
• Parity censuses: 793 render-parity models and 816 tokenizer models with zero differences. The 60 images of the 19,556-image corpus that still differed from the reference processors in v1.13.0 now match.
• A 72-hour soak of the release images on four engines and one router under closed-loop agentic traffic: through hour 45, 233,859 requests, 0 request errors, 0 5xx, 0 restarts, router memory flat.
• Nine serving stacks on one node (SMG over the three Rust servicers, SMG over ZMQ, the engines' own servers, another router) at four concurrency levels, with task accuracy equal within noise.
• CI: the chat and router suites run on a CPU mock engine on every PR, the benchmark jobs fail a PR on a regression, and the prefill/decode topology sweep runs nightly.

Notable fixes

Bugs that shipped in v1.13.0. Problems introduced and fixed within this cycle are not listed.

Security

• The gateway's own API key was forwarded upstream when a request carried no Authorization header. It now ends at the auth middleware.
• Worker URLs with credentials appeared whole in logs, headers and health answers. Every rendering masks them.
• External provider error bodies leaked the operator's organization and project identifiers. Relays answer through one sanitized envelope.
• OIDC discovery documents were accepted whatever issuer they declared. JWKS documents without a declared length were read without a bound.
• API keys and store URLs could only be passed as arguments, readable by any process on the host, and the chart rendered the key into the Deployment.
• Unbounded grammar unfolding for wide tool schemas, unbounded XML nesting, a symlink followed under /dev/shm, and the API-key table copied on every request.
• Dependency advisories closed: RUSTSEC-2026-0433 (pulldown-cmark), -0378, -0379, -0420, -0422, -0423 and -0424 (wasmtime 49.0.5), -0385 (ctutils), the zip extraction advisories and the TLS trust-store crates. tokio moves to 1.54.1.

Streaming, APIs and tool calls

• Tool call ids repeated across requests because they were minted from the call index. Ids now come from a per-request nonce.
• Calls cut by the token limit lost members or showed their bytes as content. Gemma and pythonic calls cut before their close returned empty arguments.
• GLM-style <arg_key> fragments, extra function blocks and whitespace leaked into visible content.
• With a reasoning parser on, logprobs covered the content tokens only.
• A client that left a stalled stream, or a stop string the gateway matched itself, let the engine generate on to max_tokens.
• A panic inside the parser cut the stream without a terminal frame. Connection resets were 500. HTTP worker timeouts were replayed on another worker.
• thinking.disabled lost to reasoning_effort. DeepSeek V4.1's medium effort answered 400. tool_choice: "none" dropped the tools from the prompt.

Storage and history

• A Responses conversation longer than 100 items replayed the oldest 100 instead of the newest.
• A continued conversation lost the stored turn's tool calls and outputs.
• Duplicate item ids failed the turn on Postgres and Oracle.
• Redis indexes never expired with their records. Oracle created tables before checking the schema version.

Routing, metrics and multimodal

• Data parallelism: only rank 0's KV events were relayed, loads were polled per rank and misreported, and conversations spread across ranks.
• Prefix-cache accounting credited evicted entries and shrank the shared budget for good. A publisher restart left stores out until resubscription.
• Workers did not follow their engines' restarts (labels, admission window, model card). Demoted workers returned late.
• The bucket prefill policy pinned requests after a join and spread draining workers at random.
• Prefill/decode over TCP pulled KV at 2.2 Gbit/s through 8 KiB segments. Busy prefill workers timed out as silent.
• GLM-5.3-Flash images reached the model nearly black. GLM 5 clips were sampled at 32 frames. Oversize media items were admitted and never scheduled.
• The Messages route did not count upstream errors. Mesh Suspected, Down and Leaving broadcasts never reached peers under mTLS. The Helm dashboard's panels were empty.

Engine servicers and start-up

• Healthy engines were declared dead: long prefills tripped the 300 s silence bound and slow model loads hit the fixed 600 s handshake.
• Servicers reported SERVING before any shape had run, so first requests paid the kernel searches (1–5 s first-token bursts after each load increase).
• TokenSpeed aborts ended as empty completions or 500. Engine 429, 404 and timeout answers reached the client as 500.
• The Python servicer left the engine's KV-event publisher off, so stock cache-aware deployments routed blind.

Upgrade notes

Configuration that now refuses to start

• --kv-index, --kv-indexer-ttl-secs, --kv-indexer-max-entries, and a per-policy load_check_interval_secs.
• A secret given twice (flag plus *_FILE or environment), or an empty or multi-line secret file.
• --admin-host without --admin-port, an admin port shared with another listener, --service-discovery beside --discovery-provider, a Consul provider without --consul-service, a Slurm provider without a port or job name.
• --runtime-worker-threads above 4096, a media fetch timeout of 0, --zmq-startup-timeout-secs 0, invalid prefill/decode readiness thresholds.
• --mm-turbojpeg require on a host where libjpeg-turbo cannot be loaded.
• Engine side: --metrics-port without --servicer-impl rust or equal to the serving port, and --mm-processor smg on the Python servicer.

Changed defaults

• --parser defaults to symphony.
• The chain index is the only KV index.
• Servicers report SERVING after the warm-up (SMG_<ENGINE>_SERVICER_WARMUP=full, 300 s budget, 900 s on TokenSpeed). The gateway registers a gRPC worker once its engine is healthy (--worker-startup-check-interval 30 s, --worker-startup-timeout-secs 1800 s).
• A vLLM gRPC launch derives the CUDA-graph capture cap from the admission step (at most 512) when none is set. SMG_VLLM_CUDAGRAPH_CAPTURE_CAP=engine keeps the engine default.
• KV-cache events are on under the Python servicers too (SMG_<ENGINE>_SERVICER_KV_EVENTS=0 opts out).
• tool_choice: "none" renders the tool definitions. developer messages render as system messages on the native DeepSeek and Kimi K3 encoders.
• The one-pass media wire write is on (SMG_MM_ONE_PASS_WIRE=0 reverts). Pixel-cache entries are now the wire bytes, so raise --mm-pixel-cache-mb to keep the hit rate.
• --request-timeout-secs (1800) also bounds the ZMQ engine link. gRPC worker calls are unbounded unless --grpc-request-timeout-secs is set. --zmq-startup-timeout-secs is 3600 (was a fixed 600).
• Release binaries, the wheel and the images are built with opt-level 3: about half the CPU per request on the gRPC path, for a larger binary.
• Helm: router.progressDeadlineSeconds 1920, worker images default to ghcr.io/smg-project/smg:<tag>, secrets travel in Secrets and a changed secret rolls the router pods.
• The console log is coloured on a terminal only.

API behavior

• Prompts render as the engines render them, so prompt token counts and prefix-cache keys change once for the affected requests: tools under tool_choice: "none", developer messages, tool-call-only assistant turns, byte-exact template files, Python-style tojson.
• New error codes: media_too_large, media_fetch_failed (400 for a bad URL, 502 for a failing origin), reader_panicked, no_media_ref_capable_worker, <side>_worker_failed_to_start, <side>_worker_silent, <side>_stream_error, call_upstream_timeout, upstream_response_too_large, no_control_endpoint.
• Status codes: a connection reset under a call is 502. Engine "not found", "unavailable", "too many requests" and timeout answers are 404, 503, 429 and 504. A constraint the engine cannot enforce is a 400 before the engine.
• Tool calls under Symphony: ids are call_ or toolu_ plus 24 hex characters. A cut call keeps its partial arguments with finish_reason: length. A call to an undeclared function is delivered as named and counted as unknown_tool.
• Responses: the history window begins at a turn boundary with the newest items. The error event is typed, with a code and a sequence_number.
• x-override-priority is forwarded to HTTP and prefill/decode workers. x-smg-parser names the parser that read the answer.

Metrics

• New: smg_router_time_to_first_token_seconds, smg_router_request_time_per_output_token_seconds, smg_worker_state, smg_engine_kv_capacity_tokens, smg_kv_block_size_tokens, smg_http_connections_rejected_total, smg_discovery_skipped_total, the smg_mm_* media series, the smg_symphony_* series, and smg_servicer_* on the servicer's own listener.
• Changed: smg_router_request_errors_total gains reason. smg_router_tokens_total gains token_type="cached". smg_kv_event_missed_batches_total no longer counts replayed gaps.
• Removed: smg_db_connections_active, smg_db_items_stored, smg_db_operation_duration_seconds, smg_db_operations_total, smg_router_stage_duration_seconds, smg_worker_connections_active.

Engine side

• Roll gateways first, then engine pods. A v1.13.0 servicer keeps working with the new gateway. The warm-up, the servicer metrics, the per-rank data-parallel load feed, the TensorRT-LLM Rust servicer and the engine abort statuses need smg-grpc-servicer 0.15.0 with the smg 1.14.0 wheel beside it.
• New settings on every engine: SMG_<ENGINE>_SERVICER_WARMUP, SMG_<ENGINE>_SERVICER_SILENCE_TIMEOUT_SECS (300) with SMG_<ENGINE>_SERVICER_PREFILL_TOKENS_PER_SEC (2000), --metrics-port and --metrics-host, SMG_VLLM_CUDAGRAPH_CAPTURE_CAP, and the SMG_TRTLLM_SERVICER_* family.
• The engine image sets UCX's TCP segment size to 1 MiB. SGLang 0.5.22's request structs are decoded.

Bindings and Rust crates

• Rust 1.98 is needed to build from source.
• Python: keyword-only Router(...) and RouterArgs(...) parameters, the KV-index settings removed, and the launcher gains the new flags listed above.
• Go: not published as a module release this time. The bindings build from the tag.
• Library crates with breaking changes: openai-protocol 1.17.0 (52 deprecated builder constructors removed), kv-index 2.0.1 (positional indexer removed), llm-multimodal 1.15.0 (hf-hub feature, preprocess_with_context), llm-tokenizer 1.11.0 (decoders append to a caller's buffer), smg-grpc-client 1.16.0, engine-servicer 0.3.0, engine-zmq-client 0.4.0, engine-zmq-adapter 0.2.0, smg-symphony 0.2.0, smg-response-template 0.2.0, smg-external-router 0.4.0, smg-rl 0.3.0. Pin exact versions until your code is migrated.

Packages

• smg 1.14.0 on PyPI and crates.io, the image ghcr.io/smg-project/smg:1.14.0, and Helm chart 1.14.0.
• Engine images: 1.14.0-vllm-v0.31.0 (also on the 0.27.1 and 0.26.0 bases), 1.14.0-sglang-v0.5.21, 1.14.0-trtllm-1.3.0rc24 (also rc23 and rc22), and 1.14.0-tokenspeed-<nightly>.
• smg-grpc-servicer 0.15.0 and smg-grpc-proto 0.4.28.
• Library crates at the tag: openai-protocol 1.17.0, llm-tokenizer 1.11.0, llm-multimodal 1.15.0, smg-grpc-client 1.16.0, kv-index 2.0.1, smg-symphony 0.2.0, smg-response-template 0.2.0, engine-servicer 0.3.0, engine-zmq-client 0.4.0, engine-zmq-adapter 0.2.0, smg-external-router 0.4.0, smg-rl 0.3.0, data-connector 2.4.1, smg-mesh 1.4.5, smg-auth 1.2.5, smg-mcp 2.3.6, smg-wasm 1.1.5, wfaas 1.1.1, reasoning-parser 1.10.1, tool-parser 1.10.1, smg-radix-tree 0.1.1, smg-mm-rdma 0.1.1.
• No Go module release this time.

Known limits

• Twelve legacy parser names have no Symphony table and keep the legacy parsers: step3, sarashina, cohere, cohere_cmd, pythonic, kimi, and the reasoning names base, glm45, nano_v3, qwen3_thinking and deepseek_r1. passthrough is the explicit way back for one model. gpt-oss models stay on the Harmony pipeline.
• The Inkling table writes no forced-call tag when the thought is open at the first token; such a call runs unconstrained.
• Python remains the default servicer. The servicer metrics endpoint exists on the Rust servicer only.
• SGLang Rust servicer: the encoders take images only, and a prefill leg of plain data-parallel replicas is refused.
• TokenSpeed Rust servicer: MRoPE families other than the Qwen3.5 lineage are refused, and the encode role stays on the Python servicer.
• TensorRT-LLM Rust servicer: Embed and SubscribeKvEvents answer UNIMPLEMENTED; beam search and the logits-returning flags are refused by name.
• The direct ZMQ lane's tcp:// sockets carry no authentication.
• Hybrid TokenSpeed models publish no KV-cache events, so cache-aware routing on them follows the gateway's own approximate tree.
• Slurm discovery needs a router image with the Slurm clients; the published image has none.

Thanks to everyone who contributed, and welcome @fanyang-real, @NageNalock, @alilxxey, @eyeooo, @BugenZhao and @zRzRzRzRzRzRzR!

Full changelog

What's Changed

1066 commits since v1.13.0 on release/1.14 (the merged pull requests, by type):

Features

• feat(gateway): Symphony is the default parser; --parser legacy and the X-SMG-Parser header keep the older parsers (#4677)
• feat(gateway): TokenSpeed engines serve every role on the direct ZMQ lane (#4649)
• feat(symphony): Llama 3.1 and 3.3 Instruct read with the json table under their own ids (#4655)
• feat(engine_zmq_adapter): a TokenSpeed engine's prompt logprobs, top-k candidates and abort status are read by the column names it declares (#4630)
• feat(gateway): media over the SGLang direct-ZMQ wire is assembled as the gRPC path assembles it, and the ZMQ lane runs the multimodal module (#4580)
• feat(tau2): add shared budget reservations (#4217)
• feat(servicer): a Rust servicer for the TensorRT-LLM gRPC contract, with its binding and launcher (#4494)
• feat(mock-worker): select repeatable output from prompt tokens (#4451)
• feat(servicer): the TokenSpeed servicer's control RPCs ride the wire's control call when the scheduler declares its reply (#4453)
• feat(servicer): the Rust SGLang servicer serves the EPD language side (#4413)
• feat(servicer): the Rust SGLang servicer carries multimodal inputs to the scheduler (#4405)
• feat(health): the readiness body names each served model with its readiness and a reason (#4412)
• feat(servicer): the Rust SGLang servicer serves prefill/decode disaggregation (#4411)
• feat(metrics): first-token and per-output-token latency histograms with their own bucket edges (#4410)
• feat(servicer): the Rust TokenSpeed servicer serves multimodal items of the Qwen3.5 MRoPE lineage (#4415)
• feat(servicer): the Rust TokenSpeed servicer serves prefill/decode disaggregation (#4408)
• feat(observability): the media pipeline's stages, items and fetches are series on the router and the worker side (#4377)
• feat(metrics): the router error counter carries the reason behind each error class (#4345)
• feat(helm): router.discovery.provider selects the worker discovery provider, and the Role exists only for what reads the Kubernetes API (#4391)
• feat(metrics): one per-worker routing-state series names why a worker is out of rotation (#4353)
• feat(metrics): the cached prompt tokens are a token type of the tokens counter and every cache-aware decision observes its match ratio (#4360)
• feat(metrics): the Symphony failure classes of a served model's table exist at zero from its first registration (#4348)
• feat(servicer): StartProfile and StopProfile on the vLLM wire, both servicers (#4369)
• feat(metrics): the KV capacity behind the engine's usage ratio and the block size the index routes by are gauges (#4357)
• feat(helm): a dashboard over the emitted families and a PrometheusRule with ten serving alerts (#4343)
• feat(helm): a gRPC worker can serve the servicer's Prometheus metrics from the engine pod (#4341)
• feat(mock-worker): the mock gRPC worker hosts the standard grpc.health.v1 service (#4338)
• feat(gateway): configure PD readiness capacity thresholds (#4331)
• feat(multimodal): expose media fetch timeouts (#4330)
• feat(servicer): a /metrics endpoint with request, token and latency metrics on every wire (#4149)
• feat(discovery): a Consul provider that discovers a service's passing instances (#4137)
• feat(glm): add native GLM-5.3 rendering and align Flash media processing (#4036)
• feat(multimodal): Nemotron-H Omni images shrink to a request's token budget, as the engine's tiler shrinks them (#4120)
• feat(multimodal): make the Hugging Face Hub download an optional hf-hub feature (#4119)
• feat(cache_aware): opt-in spill pricing for long-session prefill pools (#4089)
• feat(e2e): a mock engine row in the harness, with the answers stated by the test (#4070)
• feat(mock-worker): scripted answers on every wire, from a keyed file or the admin API (#4066)
• feat(discovery): a Slurm provider that discovers the nodes of running jobs (#4013)
• feat(symphony): the DeepSeek R1 line reads with the deepseek_v3 table by model id (#3780)
• feat(gateway): a request configured for Symphony that the legacy parsers read is counted by reason (#4007)
• feat(helm): the Oracle credentials reach the router from a Secret, not in the pod arguments (#3631)
• feat(symphony): a table for the Mistral list of calls, the spelling before the v11 tokenizer (#3770)
• feat(symphony): Llama 3.2 reads with the json table, which drops the python tag the model writes (#3769)
• feat(symphony): the Qwen2.5 Instruct and Coder-Instruct checkpoints read with the qwen2.5 table by id (#3773)
• feat(observability): expose in-flight HTTP request snapshots (#3984)
• feat(multimodal): an opt-in TurboJPEG policy refuses to start without libjpeg-turbo instead of decoding JPEGs with the fallback decoder (#3707)
• feat(symphony): a table for Llama 4's Python list of calls, bare in the turn and every value quoted (#3768)
• feat(symphony): tables for DeepSeek V3 and V3.1, the two spellings the older deepseek parsers read (#3735)
• feat(symphony): a table for DeepSeek V3.2 reads its DSML dialect, the one the older deepseek32 parser read (#3759)
• feat(symphony): every legacy parser name answers to its table or is listed as unanswered, with the reason (#3784)
• feat(symphony): a table for the Kimi K2 line reads its block of calls and the thought its prompt opens (#3772)
• feat(gateway): a Symphony table's malformed runs, suppressed calls and refused outputs are counted per table and written as one sampled line (#3747)
• feat(symphony): the MiMo V2.6 checkpoints read with the qwen3_tagged_direct table, as V2.5 does (#3779)
• feat(symphony): Inkling-NVFP4 joins the Inkling group, whose three checkpoints ship one template (#3778)
• feat(symphony): Muse Glimmer 30B joins the Muse Spark group and Phi-4 multimodal the plain one (#3737)
• feat(helm): the history store URL reaches the router from a Secret, not in the pod arguments (#3608)
• feat(rate_limit): the usage settlement carries the engine's cached and reasoning counts (#3611)
• feat(symphony): a table may bound the runs of whitespace in a JSON answer (#3728)
• feat(symphony): the v1 Nemotron Super and Ultra checkpoints read with the llama_nemotron table (#3777)
• feat(gateway): a bound on the HTTP listener's open connections, --max-connections (#3588)
• feat(cli): --bucket-adjust-interval-secs and the worker mTLS flags reach the Rust CLI (#3570)
• feat(gateway): worker-side media processing forwards every family the gateway's pipeline serves to workers running it (#3652)
• feat(gateway): the access log carries the client address, tenant, key fingerprint, user agent, byte counts and routed worker on request (#3598)
• feat(gateway): an admin listener for the control-plane routes, --admin-port and --admin-host (#3597)
• feat(gateway): the serving keys and the store URLs read from the environment or a file, and no secret is echoed (#3589)
• feat(multimodal): LLaVA, Qwen3-Omni, Qwen3-ASR and DeepSeek V4.1 opt their anchors in for worker-side expansion (#3634)
• feat(symphony): the Muse Spark table answers to the name muse, in the gateway's name checks and the CLI's choices (#3537)
• feat(beam): add day-0 Reflection's Beam support (#3538)
• feat(multimodal): Muse Spark Open images, video and audio (#3526)
• feat(grpc_servicer): the TokenSpeed servicer advertises audio for a checkpoint whose config declares a waveform tower (#3521)
• feat(symphony): the Muse Spark table (#3519)
• feat(router): forward x-override-priority to HTTP and PD workers (#3510)
• feat(symphony): a table for the DSML spelling before V4.1's, for DeepSeek V4-Flash (#3518)

Fixes

• fix(zmq): a prefill- or encode-registered direct-lane worker needs an engine that declares its bootstrap address (#4708)
• fix(servicer): the launchers advertise the structural-tag vocabulary of the engine's xgrammar (#4691)
• fix(zmq): an encode-registered direct-lane worker needs an engine that lists the encode request (#4698)
• fix(rl): a direct-lane worker is controlled over its engine connection only when its handshake declares the control reply (#4699)
• fix(gateway): every log line and failure message that names a worker writes its address masked (#4591)
• fix(gateway): the routed-worker header and the request-sent events show a worker URL with its userinfo masked (#4579)
• fix(gateway): every iteration of a Responses request takes the tenant rate-limit reservation a chat request takes (#4449)
• fix(gateway): a request with tool_choice none still renders its tools, as the engine's own server does (#4129)
• fix(symphony): the structural tag in the older vocabulary for a worker that advertises it (#4688)
• fix(e2e): recognize nixl_rocm in PD transport checks (#4671)
• fix(symphony): the blank line before the calls block is the block's opener on the DeepSeek V4 and V3.2 tables, as on V4.1's (#4073)
• fix(gateway): chat logprobs cover every generated token on the gRPC path, as the engine's own server returns them (#4634)
• fix(gateway): the request deadline on gRPC worker calls is opt-in (--grpc-request-timeout-secs, 0 = no bound by default) (#4581)
• fix(liveness): a busy PD prefill worker's long prompts no longer time out as silent (#4620)
• fix(gateway): make shutdown propagation window configurable (#4631)
• fix(external-router): log lines and the health answer show a worker URL with its credentials masked (#4603)
• fix(servicer): the Rust vLLM servicer withholds the KV-transfer rejection notice wherever a Mooncake connector runs, read off the raw config (#4601)
• fix(config): --dp-minimum-tokens-scheduler together with --dp-aware logs a start-up warning, and the help names its scope (#4595)
• fix(tokenizer): the native encoders render a developer message as the system message the engine hands them (#4598)
• fix(health): a demoted worker returns on consecutive passing probes, not consecutive intervals (#4577)
• fix(gateway): a client that leaves a stalled Harmony Responses stream aborts the engine request at once (#3720)
• fix(servicer): the PD decode side withholds the KV-transfer rejection notice where a Mooncake connector runs (#4281)
• fix(worker): the rank workers of one data-parallel engine share one load poll per tick (#4583)
• fix(mcp): preserve engine HTTP requirements in CI (#4351)
• fix(servicer): the PD decode side's pre-admission notice stops once the engine owns the request (#4260)
• fix(gateway): an explicit thinking.disabled wins over reasoning_effort (#4284)
• fix(tokenizer): the opt-in v1 pipeline cuts a Metaspace tokenizer's added tokens before the word marker is prepended (#4574)
• fix(gateway): a worker URL's userinfo is masked whole in every rendering, the start-up lines and the configuration refusals included (#4572)
• fix(engine): the engine image sets UCX's TCP segment size so a prefill/decode pair over NIXL across hosts pulls KV at line rate (#4276)
• fix(loads): preserve native DP rank attribution (#2429)
• fix(e2e): the harness's dependency list names pyyaml, which its worker module imports (#4569)
• fix(gateway): Messages, completions and harmony streams anchor PD TTFT at the prefill dispatch (#4497)
• fix(gateway): a client that leaves a stalled Responses stream aborts the engine request at once (#3706)
• fix(gateway): --request-timeout-secs bounds a request to a gRPC worker from its dispatch to its last message (#4001)
• fix(servicer): a TokenSpeed encode worker refuses FlushCache, StartProfile and StopProfile by name (#4507)
• fix(worker): a data-parallel rank worker indexes its own rank's KV events (#4499)
• fix(chat): the request's add_generation_prompt is read, so false renders the conversation without a generation header (#4280)
• fix(gateway): a client that leaves a stalled stream aborts the engine request at once (#3632)
• fix(gateway): a gRPC worker whose model info names the model as model_id only gets a tokenizer job (#4493)
• fix(worker): a data-parallel rank worker's HTTP load probe goes to its engine's address (#4486)
• fix(servicer): GetServerInfo and the zero-filled loads count the connected ranks (#4485)
• fix(python): declare the protobuf floor the generated proto modules check at import (#4484)
• fix(tests): scope the shared-memory path import to Linux (#4450)
• fix(servicer): the negative video frame count refusal names its grid row (#4474)
• fix(servicer): a warm-up prompt render that raises is a WARNING with its traceback (#4472)
• fix(servicer): a TokenSpeed engine abort ends the stream with the engine's status and words (#4444)
• fix(gateway): the response reader's panic guard holds the reading's future once (#4470)
• fix(servicer): a negative video frame count is refused by name, not as an overflow (#4466)
• fix(servicer): the Rust SGLang servicer serves a prefill or decode leg of more than one scheduler rank (#4460)
• fix(gateway): a parser panic ends its own request with an error frame instead of a cut stream (#4135)
• fix(metrics): a KV event replay request counts no missed batches (#4440)
• fix(servicer): the planned warm-up counts a request the engine ended with an abort as failed (#4434)
• fix(servicer): the Rust SGLang servicer places a request on the data-parallel rank the Router pins (#4414)
• fix(servicer): the Rust SGLang servicer reads an unset top_p and top_k as their neutral values (#4459)
• fix(router): the HTTP router's worker-address error log fields are written log-safe (#4462)
• fix(metrics): the registration step publishes a Symphony table's failure classes once the worker is in the registry (#4382)
• fix(servicer): the SGLang language-only worker runs the planned warm-up before it reports SERVING (#4426)
• fix(servicer): the TokenSpeed wire pins the seed of a greedy request as the engine does (#4420)
• fix(servicer): the planned warm-up stops short of what the engine's scheduler refuses, and a one-process engine plans the ladder alone (#4438)
• fix(external_router): the Messages tool loop counts an upstream error status and a broken stream (#4441)
• fix(servicer): the SGLang Python servicer reads unset sampling scalars as their neutral values (#4416)
• fix(external_router): the Messages route counts a send that fails below HTTP (#4429)
• fix(gateway): the PD count-tokens route logs the worker and endpoint of a call that failed below HTTP (#4387)
• fix(servicer): SGLang hidden states are served on the Rust servicer; the Python servicer refuses them by name (#4407)
• fix(metrics): a collected answer records its token and generation families as a streamed one does (#4399)
• fix(kv_index): the string tree's snapshot merge compares a remote edge that continues below a local child against that child's own children (#4394)
• fix(worker): the worker API refuses the smg.ai/discovery-* labels, and a PUT or a label PATCH of a discovered worker (#4392)
• fix(tokenizer): a request without tools renders the processor's chat template where the checkpoint ships one that differs (#4373)
• fix(tracing): the trace layer keeps its own filter, so --enable-trace exports the spans at every log level (#4379)
• fix(gateway): the SSE re-chunker keeps a tool call's tail for its closer (#4365)
• fix(tokenizer): tojson(indent=...) renders as Python does for a negative, zero, string or bool indent (#4356)
• fix(gateway): a worker's address stays out of the client's answer for a call that failed below HTTP (#4368)
• fix(servicer): a readiness hold admits only the warm-up's own requests (#4250)
• fix(chat): a history tool call's arguments may be a JSON object, and the template gets the object the engines hand it (#4279)
• fix(helm): a worker Deployment's rollout deadline leaves 600 s over its startup probe's budget for the node and the pull (#4364)
• fix(gateway): a media host that does not deliver is a 502 under the media's code, the request's own URL a 400 (#4355)
• fix(e2e): Worker.stop() releases the log handle and the ports of a process that already exited (#4262)
• fix(symphony): the tagged-XML assembler refuses a call nested past the JSON reader's bound instead of growing its stack (#4354)
• fix(helm): the router Deployment's progress deadline covers the engine's bring-up now that /readiness follows the engine (#4286)
• fix(tokenizer): a chat template file reaches the renderer byte for byte: no escape translation of its backslash-n sequences (#4232)
• fix(servicer): the load report's aggregate sums saturate (#4346)
• fix(symphony): the Qwen-tagged and XML assemblers refuse a call's leftovers, never content (#4303)
• fix(symphony): the cut token in the unnamed-invoke test is marked for the spell check (#4339)
• fix(tests): the prefill-decode side-failure test reads the shared harness crate, not a module that is gone (#4340)
• fix(e2e): a worker that dies during the gateway's readiness wait fails it at once with its death report (#4336)
• fix(e2e): a gRPC worker is healthy on SERVING only; the engines' ports lie below the ephemeral range (#4325)
• fix(servicer): the warm-up's end line carries why the engine's cache was not flushed (#4321)
• fix(gateway): GET /v1/models lists every alias of a served model (#4289)
• fix(gateway): a demoted worker returns to routing on a health read, not on its first contact (#4249)
• fix(tokenizer): a chat template file is read as written, so the newline a template opens with reaches the prompt (#4212)
• fix(gateway): a prefill or decode side that fails is named the same way by the HTTP and the gRPC prefill-decode paths (#4205)
• fix(cache_aware): gate the miss path and ignore idle throughput gauges (#2810)
• fix(symphony): a call the token limit cuts inside a string value's closing tag keeps the value and none of the tag (#4307)
• fix(symphony): a keyed call the token limit cuts keeps the cut member in its arguments (#4223)
• fix(e2e): an HTTP worker that dies during start-up fails the wait at once with the death report (#4261)
• fix(gateway): a connection reset under a call is answered 502 like other transport failures, not 500 (#4309)
• fix(servicer): the TokenSpeed warm-up also visits every prompt length up to the chunk size before SERVING (#4231)
• fix(symphony): bytes a tagged call's syntax cannot read are refused, never content (#4291)
• fix(kv-events): a clear resets the worker's block of record for its next cache incarnation (#4300)
• fix(symphony): a Gemma call the token limit cuts stays open and keeps the cut member (#4186)
• fix(tokenizer): the legacy-normalizer test takes only the space-to-marker replacement (#4301)
• fix(symphony): a pythonic call the token limit cuts keeps the cut argument in its arguments (#4179)
• fix(tokenizer): a Llama-class checkpoint encodes as its reference class, not as its legacy file pipeline (#4248)
• fix(tokenizer): the string filters a chat template applies to a list or a dict start with Python's str, as {{ }} does (#4247)
• fix(kv-events): a hybrid engine's prefix-tail entries are left out of the index, not read as parent-less stores (#4170)
• fix(gateway): embeddings, classify and transcription take the tenant rate-limit reservation every other gRPC endpoint takes (#4202)
• fix(multimodal): a GLM 5 family clip is sampled by the family's own loader rule, not the generic loader's (#4203)
• fix(tokenizer): a chat_template shipped as a list of named templates is read: default for a plain request, tool_use when tools are declared (#4230)
• fix(health): discovery's default removes only the failed workers it discovered (#4297)
• fix(symphony): a call the token limit cuts inside a held value keeps the value in its arguments (#4167)
• fix(tokenizer): an explicit null special token in tokenizer_config.json is no token, and a guess names only added special tokens (#4240)
• fix(servicer): the Rust vLLM servicer withholds a PD decode side's rejection notice on the Mooncake connector (#4287)
• fix(servicer): the TokenSpeed servicer maps an engine abort's status with the shared table (#4173)
• fix(gateway): a tool-call-only assistant turn reaches the chat template as empty content, as the engine hands it (#4210)
• fix(gateway): a Messages response refused for a tool_use input no value can carry is counted under its table (#4252)
• fix(deps): upgrade tokio to 1.54.1 (#4296)
• fix(tokenizer): a DeepSeek V4 architecture checkpoint that ships a chat template renders through it; the encoder port stands in only where there is none (#4236)
• fix(response-template): \w takes the 12,860 word characters the regex module adds past Unicode 17 (#4227)
• fix(symphony): a string value left open at the call's closing marker is closed there and said (#4207)
• fix(engine_zmq_adapter): a stale ipc socket that vanishes between the stat and the unlink counts as removed (#4215)
• fix(servicer): the KV-event relay joins the engine's publisher within a second of its bind, and the planned warm-up waits for the subscription (#4204)
• fix(tokenizer): a list or a dict a chat template prints is written as Python's repr, the bytes the engine's prompt holds (#4220)
• fix(data-connector): an item created again under a stored id is the stored item on every history backend (#4194)
• fix(servicer): a request that reaches a closing connector is UNAVAILABLE, not INTERNAL (#4193)
• fix(data-connector): the Redis history store's item and identifier indexes expire with the records they list (#4180)
• fix(servicer): the KV-event relay keeps the learned cache groups across a clear, so its snapshot carries the prompt stored with the reset whole (#4178)
• fix(gateway): an output whose decoder fails under Symphony is counted as the table's failed output (#4175)
• fix(protocols): Kimi K3 and GLM-5.3 models served under an alias keep their series' rules under every name (#4169)
• fix(multimodal): the video sampling rate is carried as a double from the config and the request to the frame picks (#4163)
• fix(observability): close_logging waits for the writer threads past the guards' bounded moment (#4166)
• fix(helm): worker images default to the repository the engine images are published under; README and examples on the current release tags (#4188)
• fix(grpc_servicer): a recipe's capture-cap ceiling applies when it leaves --max-num-seqs to the engine; the README says what a ceiling buys per running window (#4158)
• fix(protocols): a DeepSeek V4.1 model served under an alias keeps its reading of the reasoning effort under every name (#4164)
• fix(symphony): a Llama 4 quoted value that ends in a backslash closes at its quote (#4168)
• fix(servicer): a long prefill on a token-batch wire is not a dead engine, and the silence bound is configurable (#4068)
• fix(gateway): a transcription answer is read under the same payload cap as every other worker body (#4143)
• fix(engine): one prompt admission check on the three ZMQ wires (#4142)
• fix(servicer): a MultiConnector is projected onto the child that moves the KV, a store or MoRI-IO child included (#4155)
• fix(gateway): a stop the gateway matches itself aborts the engine request instead of draining it (#4040)
• fix(grpc-client): a chat request's no_stop_trim reaches vLLM and TRT-LLM as it reaches the other engines (#3915)
• fix(symphony): a string value the model leaves open at its own end of output is closed there and said (#4134)
• fix(liveness): a PD decode side's first output counts as its prefill worker's answer (#4136)
• fix(gateway): the console log is coloured on a terminal, not in a file or a pipe (#4130)
• fix(servicer): the servicer reports SERVING only after a planned warm-up over every running-batch size and prefill length class the engine admits (#4132)
• fix(engine): an engine's grammar abort is one explicit error the gateway does not retry, where the engine attributes it (#4112)
• fix(zmq): a JSON-schema answer with thinking on lands in content over the gateway's own engine link (#4110)
• fix(symphony): text where a call's syntax has structure keeps the model's whitespace between its words (#4125)
• fix(symphony): a value held whole from a CDATA section keeps its bytes when stray text follows the closer (#4127)
• fix(symphony): text released as content keeps the model's whitespace between its words (#4116)
• fix(protocols): a DeepSeek V4 forced tool choice with thinking on is served instead of refused (#4109)
• fix(tokenizer): a DeepSeek V4.1 request's medium reasoning effort renders the checkpoint's budget instead of a 400 (#4100)
• fix(tool_parser): a surrogate pair escape decodes to one character (#4009)
• fix(gateway): the worker HTTP client keeps its connect timeout below the request timeout (#4075)
• fix(tokenspeed): forward PD rank pins and correct per-rank capacity (#4131)
• fix(external-router): a streaming MCP tool call with non-JSON arguments is counted as an errored call, and both loops word the error alike (#4115)
• fix(servicer): finish draining when admitted requests complete (#4117)
• fix(symphony): prose after a bare call list keeps the model's whitespace between its words (#4123)
• fix(servicer): the KV-events default follows the engine's own word and leaves TokenSpeed's publisher off for a model whose State groups span no tokens (#4107)
• fix(zmq): log unexpected input monitor events (#4087)
• fix(multimodal): a Qwen2-VL image anchors on the template's <|image_pad|> and expands to image_token_id (#4121)
• fix(servicer): the start-up lines logged during the launcher's argument parse reach the engine log on a base that configures logging after the parse (#4096)
• fix(symphony): a refused call block that follows the answer's text goes out as its bytes before the finish (#4090)
• fix(zmq): bound queued payloads with upstream transport fixes (#4082)
• fix(tokenizer): the DeepSeek V4.1 renderer writes the effort budgets of the checkpoint's encoder revision (#4067)
• fix(vllm): reject oversized prompts before engine admission (#4076)
• fix(symphony): a JSON answer's tag derives for the selected worker's start, so a pool of mixed starts gives each worker its own (#4071)
• fix(servicer): a constraint the engine's grammar backend cannot enforce is refused before the engine on every servicer (#4069)
• fix(tokenizer): a control-token spelling typed in a message stays text under the renderers that encode their own prompts (#4072)
• fix(symphony): the blank line between a text preamble and the calls block is the block's opener on the DeepSeek V4.1 table, not content (#3982)
• fix(protocols): accept tool_reference in tool results and unsigned thinking (#4028)
• fix(gateway): a request timeout on an HTTP worker ends the request once instead of replaying it on another worker (#4064)
• fix(gateway): a gRPC worker registers only once its engine reports healthy (#4065)
• fix(zmq): a request over the gateway's ZMQ engine link ends at --request-timeout-secs when the engine goes silent (#4060)
• fix(servicer): SGLang sampling params ride as the API prefix and the handshake checks the scheduler's field order (#4062)
• fix(symphony): a Messages call whose arguments stop being JSON gets the prefix's value in the whole message as in the stream (#4059)
• fix(tokenizer): a template that writes a history call's arguments as text gets them as the request sent them (#4058)
• fix(symphony): a call to an undeclared function is counted as unknown_tool and its line names the function (#4056)
• fix(placement): keep cache-aware PD prefill placement clear of the overload veto (#4024)
• fix(liveness): give a PD prefill worker's new run of prompts its backlog's time to answer (#4015)
• fix(symphony): a released refused call block goes out from its opener, with the bytes between two blocks (#4049)
• fix(symphony): an answer of nothing but a refused call block goes out as its bytes before the finish (#4044)
• fix(gateway): a template that takes content parts gets a message's media where the request put it (#4043)
• fix(protocols): accept tool_choice auto without tools on responses (#4038)
• fix(zmq): a tool schema the backend cannot enforce is refused inside a forced call's structural tag too (#4041)
• fix(symphony): an answer of whitespace alone goes out as content and is counted as empty (#4033)
• fix(symphony): an answer of whitespace alone goes out as content and is counted as empty (#4033)
• fix(zmq): every structured request carries the backend structural tags compile on unless the engine's configuration pins another (#4030)
• fix(protocols): accept client_metadata on Responses requests (#4032)
• fix(zmq): include deferred requests in scheduler waiting loads (#4023)
• fix(zmq): the startup handshake bound is a flag, an hour by default, and a handshake takes the READY of an engine an earlier one greeted (#4021)
• fix(symphony): a JSON answer on an open thought is the schema alone where the engine does not say where it starts (#4019)
• fix(symphony): the malformed-run counter labels every reason by the crate's own name; other only for a reason the build does not know (#4018)
• fix(symphony): the inkling table starts the answer at the text marker written inside a thought, and the tag derivation tells a kind said again from a close (#4011)
• fix(routing): ignore output throughput for prefill waits (#4017)
• fix(symphony): a forced list with no closer ends its tag with its own bracket, and every table's tag is checked for one (#4016)
• fix(external-router): a Responses stream stopped at max_tool_calls ends with response.completed and its record, as the non-streamed response does (#4010)
• fix(gateway): a decode leg the engine fails before its first token probes the pair's bootstrap port, and names a dead one to the client, in the log and in the rendezvous family (#3913)
• fix(grpc-client): a plain generate request's no_stop_trim reaches vLLM and TRT-LLM as it reaches the other engines (#3909)
• fix(responses): a history window cut inside a turn begins at the next turn boundary (#3722)
• fix(gateway): a worker's labels follow its engine when it comes into service (#3410)
• fix(symphony): a call to a function the request did not declare is announced by an UnknownTool event of no bytes before its start, on every syntax (#4020)
• fix(symphony): the inkling table reads a kind marker inside a body as the kind, so a JSON answer written behind an engine's reasoning gate comes back bare (#4006)
• fix(symphony): the json table reads every object of a sequence written on blank lines as a call (#4012)
• fix(responses): a tool result with no call before it is refused under the contract profiles, as an unknown call id is (#3592)
• fix(symphony): the Qwen3, GLM 4.6, MiniCPM5 and dots3 tables carry their templates' newlines after the thought's close, so a forced call there does not begin with the call marker (#4008)
• fix(placement): place a fully over-threshold PD pool by its policy, not by load (#4014)
• fix(servicer): one table from an engine's HTTP status to the gRPC code, behind every engine (#3726)
• fix(grpc_servicer): an engine's workers are tracked from its spawn, so an early exit leaves none behind (#3653)
• fix(external-router): a provider's error body loses the operator's account identifiers on the realtime paths too, through one shared answer (#3930)
• fix(symphony): a pythonic parameter without a value is reported and skipped, not written as an empty string (#3880)
• fix(responses): a conversation longer than the history window replays its newest items (#3600)
• fix(multimodal): worker-side media placement with no worker that processes media references is refused in preparation and holds readiness (#3584)
• fix(symphony): a pythonic call the region's close cuts inside its parentheses says so and closes nothing (#3792)
• fix(gateway): the engine's reported silence on a step-reporting wire is judged against the prefill its servicer reports in hand (#3981)
• fix(routing): isolate pushed loads by virtual DP rank (#3983)
• fix(symphony): a forced call on the DeepSeek V3.1 table is written by the table's own block and read back as a call (#3992)
• fix(symphony): the fuzz crate names its events' lifetimes (#4005)
• fix(symphony): the Kimi K2 table's test helper names its events' lifetime (#4004)
• fix(gateway): a failed Responses stream ends the same way on both routers (#3997)
• fix(symphony): a forced call on the Kimi K2 table is written by the table's own tag and read back as a call (#3990)
• fix(servicer): the tokenizer archive is built off the event loop and once (#3989)
• fix(golang): Close frees the stream's converter handle after a request that ended on its own (#3568)
• fix(symphony): the scanner's buffers come back empty when a scan is reclaimed (#3996)
• fix(servicer): the Python servicer turns the engine's KV event publisher on by default, as the Rust path does (#3558)
• fix(multimodal): the cap on one video decode's work is measured in pixels, so a low-resolution clip hours long is no longer refused when the budget fits it (#3629)
• fix(policies): the bucket's adjustment stays in the size domain (#3842)
• fix(servicer): an SGLang health probe that fails at once is answered unhealthy at once, not at the 20 s deadline (#3988)
• fix(gateway): the auth middleware's key map is shared, not cloned, on every request (#3904)
• fix(symphony): a JSON answer on the inkling table comes back without a second text marker, and a marker the table did not read is counted (#3969)
• fix(multimodal): a modality no worker of the model takes is refused before the media is fetched or decoded (#3979)
• fix(symphony): a keyed call block the syntax cannot read is never content, and the failure is counted (#3977)
• fix(tokenizer): a chat request's tools reach a DeepSeek V3.1 prompt as the block its template never writes (#3946)
• fix(symphony): a forced call after the thought's close begins with the template's newline, not the call marker (#3978)
• fix(multimodal): an item above the engine's encoder budget is refused before the engine, which would never schedule it (#3948)
• fix(symphony): the tagged reader reads every function block of a call region, not the first alone (#3971)
• fix(servicer): the load entries before the engine's first snapshot carry the servicer's own figures (#3963)
• fix(gateway): a data-parallel rank whose event stream is quiet follows the fallback poll instead of its last pushed record (#3968)
• fix(symphony): the json table reads a call behind Llama 3.1's python tag and the calls a semicolon separates (#3964)
• fix(symphony): the inkling table's no-tools variant keeps the message framing out of the content (#3962)
• fix(servicer): a data-parallel worker's load feed carries each rank's own hit rate and generation throughput (#3959)
• fix(servicer): a data-parallel worker's KV events reach the gateway from every rank under every servicer (#3941)
• fix(symphony): a pythonic call the stream's end cuts before its close says so before its end (#3791)
• fix(rl): slime data-plane parity, TokenSpeed e2e lane and a trainer-side refit example (#2668)
• fix(gateway): an origin that does not deliver a request's media in time is the request's 400, and a size refusal has its own code (#3947)
• fix(config): the per-policy load_check_interval_secs goes, and a configuration that still sets it is refused by name (#3838)
• fix(gateway): the completion API reads a backend's finish reason through one normalizer, and the engines' signals pass through (#3879)
• fix(gateway): a gateway API key ends at the auth middleware and is never forwarded to an upstream as the caller's credential (#3918)
• fix(symphony): a call whose arguments are not JSON keeps them as the model wrote them and says so once, not a call cut at the bad token (#3945)
• fix(external-router): the realtime WebSocket relay strips the operator's account identifiers from a provider's error event (#3954)
• fix(multimodal): 16-bit greyscale PNGs clip each sample to 255, as the engine's own loader does (#3344)
• fix(ci): the mixed-line-ending hook leaves the recorded SSE streams as recorded (#3936)
• fix(external-router): a provider's error body loses the operator's account identifiers on the chat and Messages paths too (#3919)
• fix(gateway): the WASM middleware answers an unreadable body in the error envelope every other gateway error uses (#3912)
• fix(gateway): the selection queue bound is read on the runtime's clock, and its later-run test on a paused one (#3867)
• fix(gateway): a stopping gateway leaves the mesh before it drains (#3854)
• fix(mesh): the status broadcasts dial with the node's TLS identity, so Suspected, Down and Leaving reach the members under mTLS (#3853)
• fix(release): the version tool's git calls read the root they are given, and its files are read and written as UTF-8 (#3887)
• fix(grpc-client): a plain generate request that names two structured constraints is refused on every engine alike (#3895)
• fix(mcp): the inventory refresh keeps a server's tools when its listing fails, awaits outside the map's lock and sees a recovered server; the four tool listings share one function (#3905)
• fix(auth): the JWKS provider bounds a response body as it reads it, for the key set and for the discovery document alike (#3901)
• fix(symphony): a JSON call closes whole only with arguments a strict reader takes (#3790)
• fix(symphony): a tool_use input that cannot carry the model's arguments is named, and the gateway refuses the message instead of {} (#3885)
• fix(symphony): a JSON call the end or the close cuts right after its header says so before its end (#3876)
• fix(policies): a draining prefill worker's size range goes to its neighbour at once (#3840)
• fix(servicer): the TokenSpeed servicer reads /dev/shm tensors through the shared reader, which refuses a symlink (#3865)
• fix(tests): building an application context applies its configuration's liveness budgets (#3850)
• fix(grpc-client): a plain generate request's seed reaches vLLM and TRT-LLM, and its stop token ids TRT-LLM (#3846)
• fix(mesh): the join peer is dialed until the membership table has another member, not once in the first round (#3847)
• fix(symphony): the one-call-only rule keeps the bytes of the calls it drops, so the content ranges after them hold (#3844)
• fix(mesh): the peer-connections gauge counts a peer's open streams, so an older stream's close does not zero a live one (#3851)
• fix(mesh): the peer-connections gauge carries the dialer's name, never an empty peer label (#3839)
• fix(bindings): the Go binding's chat responses carry the engine's weight version as system_fingerprint, none without one (#3832)
• fix(policies): a prefill worker listed after the first split takes its share of the sizes (#3774)
• fix(symphony): the Gemma syntax bounds its nesting where the JSON prefix parser does (#3760)
• fix(gateway): Symphony's registry reads an unlabeled model it knows before a fleet-wide parser name (#3787)
• fix(symphony): a Gemma member without a value is reported and skipped, not written as an empty string (#3788)
• fix(deps): zip 9.0 closes the extraction advisories and the audit's ignore list records every remaining warning (#3805)
• fix(symphony): a raw control character inside a JSON string reaches the client as its escape (#3755)
• fix(symphony): a Gemma member without a key is reported and skipped, as the tagged readers leave one (#3781)
• fix(policies): a removed prefill worker leaves the bucket policy's size partition, and removal reaches every policy (#3763)
• fix(auth): OIDC discovery refuses a document whose issuer is not the configured issuer (#3806)
• fix(symphony): a call the closing marker cuts inside a string value is left open and says so (#3776)
• fix(gateway): a window the operator set by hand is the worker's only window, at registration and on a label update (#3446)
• fix(symphony): a tool call's id counts from a nonce of the request's own, never from the call's index (#3705)
• fix(symphony): a tagged call the stream's end cuts says so before its end (#3767)
• fix(symphony): the chat adapter's test builds its malformed reason in the borrowed form (#3794)
• fix(symphony): prose after a JSON value alone in the turn keeps its whitespace (#3643)
• fix(bindings): the chart render test runs helm by name, and the Python lint job passes on main again (#3798)
• fix(kv_index): the string tree's insert replay re-validates each edge before it attaches a tenant (#3785)
• fix(router): every answer a router gives on its own is the JSON error object, under a code (#3743)
• fix(servicer): a health probe closes the stream it leaves at its first output (#3753)
• fix(servicer): the default KV-event publisher takes a free port, and the e2e job ends what it started (#3786)
• fix(helm): the Grafana dashboard reads the families the gateway emits (#3756)
• fix(kv_index): the insert replay of the fused walk re-validates each edge before it attaches a tenant (#3782)
• fix(symphony): a call cut inside its arguments value says so before its end (#3754)
• fix(liveness): an engine building a kernel for a prompt shape it has not run is not a wedge (#3730)
• fix(grpc): a retry derives the table's auto tag for the start of the worker it takes (#3727)
• fix(messages): thinking.display "omitted" returns the thinking blocks without their text (#3731)
• fix(symphony): a turn the prompt leaves unopened starts inside the thought on the DeepSeek V4 table (#3711)
• fix(worker): the mesh health hint and the drain step decide a worker's status under its mutation lock (#3614)
• fix(servicer): the vLLM launcher switch refuses a launcher it cannot bind at the argument parse, before the engine is built (#3569)
• fix(launcher): the six CLI flags the binding could not carry reach the launcher (#3565)
• fix(messages): a system-role message sent after the first is a user turn for a renderer whose reference writes no turn for it (#3712)
• fix(symphony): a table's tag takes the engine's grammar start into account where the prompt leaves the thought open (#3648)
• fix(multimodal): the anchor parity checker follows the engine's current processing API (#3713)
• fix(golang): a request that fails after its worker is selected gives the worker's load back (#3714)
• fix(messages): thinking.display takes the documented set, updates included (#3694)
• fix(workflow): a replacement that changes a worker's group refreshes the pool size of the group it leaves (#3704)
• fix(symphony): a call whose object has no arguments member is a call with an empty object (#3702)
• fix(install): the toolchain version lines in install-smg.sh run after PATH is exported and stop the build when a tool is missing (#3664)
• fix(symphony): a response template's strip: false field keeps whitespace alone as written (#3697)
• fix(grpc): Symphony keeps a table's calls wherever the legacy path runs a tool parser (#3572)
• fix(symphony): the whitespace a content stay begins with is content only when text follows it (#3676)
• fix(grpc): filter workers by advertised context capacity (#3670)
• fix(grpc_servicer): a vLLM gRPC launch derives the CUDA-graph capture cap from the admission step (#3642)
• fix(responses): the kinds the history does not replay are named one by one, and the left-out line costs a name (#3661)
• fix(grpc_servicer): load_record lists poll_timeout_for among its exports (#3662)
• fix(multimodal): an empty image batch is refused as an empty batch by every vision processor (#3659)
• fix(cli): bound clap construction stack usage (#3663)
• fix(tokenizer): enforce the L1 cache memory budget (#3657)
• fix(grpc): a retry stays on workers that decide the constraint alike (#3638)
• fix(glm): the media-reminder gate probes the part kinds a template is handed, and its doc and test name the family it checks (#3644)
• fix(responses): the history a request continues keeps a stored turn's tool calls and their outputs (#3582)
• fix(multimodal): a seek run of a long clip ends right after its last pick, so a missed pick falls back to the sequential decode at once (#3621)
• fix(grpc_servicer): the host:port rendezvous constructor takes its flag in either spelling (#3630)
• fix(servicer): the Rust vLLM servicer takes audio media references for the engine's own processors (#3633)
• fix(grpc_servicer): the SGLang and TokenSpeed health Watch answer an unknown service without failing the stream (#3635)
• fix(config): the listen IP must parse and the worker-thread count has a bound (#3594)
• fix(grpc): a request's constraint is decided once, on the worker selected for it (#3624)
• fix(data-connector): the Redis conditional writes store the output-item keys and the expiry with the record (#3580)
• fix(glm): render media reminders without preprocessing (#3587)
• fix(symphony): a word under way in a call's arguments is held until it is spelled or refused (#3593)
• fix(grpc_servicer): the engine's --mm-processor-kwargs reach smg's own media pipeline on the Rust servicer (#3585)
• fix(symphony): an optional call the stop closes has no tag again where more than one may come (#3619)
• fix(symphony): a Gemma member that follows a value without a separator keeps its first byte (#3601)
• fix(symphony): a value's held newline leaves before the start of a tag the stream cut (#3603)
• fix(symphony): an ignored terminal inside a held call keeps its place among the bytes (#3591)
• fix(multimodal): a long clip sampled sparsely is decoded by seeking to its picks, and a decode over its budget ends at once (#3578)
• fix(servicer): the vLLM launcher switch binds the launcher run as main, on every supported release (#3577)
• fix(gateway): the auto-constraint test asserts the engine's schema again for a registry table's JSON response_format (#3576)
• fix(grpc_servicer): the worker-side media path raises a tiny image to the family's short-side floor, as the router's own pipeline does (#3575)
• fix(grpc_servicer): the Python servicer refuses --mm-processor smg before the engine is built, not after the model loaded (#3571)
• fix(gateway): a failed start-up flushes and shuts the tracer down before the error returns (#3566)
• fix(discovery): the file provider's lost-watch retry fires under a short check interval (#3564)
• fix(data-connector): the Redis conditional writes treat a record without an object field as open, as the SQL stores do (#3563)
• fix(gateway): the gRPC chat gates read the tools a request declares anywhere, so a tool declared on a message keeps its call markers (#3562)
• fix(gateway): a worker's model card follows its labels when they are updated (#3561)
• fix(responses): a retried terminal persist whose earlier write landed links its items (#3560)
• fix(gateway): the Harmony encoding's first load runs on a thread of its own (#3559)
• fix(symphony): a call written as Python where a keyed block's name goes names no call (#3555)
• fix(golang): the gRPC request path keeps the engine's special tokens for the tool parser (#3552)
• fix(launcher): --drain-settle-secs and the CLI's second spellings reach the launcher, a bare --storage-context-headers entry is refused, and a test pins the flag parity (#3548)
• fix(data_connector): the Oracle store checks the schema version before it creates any table (#3545)
• fix(mock-worker): honor gRPC aborts and close the listener on shutdown (#3511)
• fix(gateway): a JSON response_format leaves a tag that is the schema alone to the engine (#3543)
• fix(multimodal): the exact audio resampler admits the odd rates real headers carry (#3542)
• fix(symphony): the walks behind an invoke's number leaves have an allowance of their own, and size what is sent (#3533)
• fix(tool_parser): the walks behind an invoke's number leaves have an allowance of their own (#3534)
• fix(multimodal): follow-ups to the Muse Spark media review (#3532)
• fix(gateway): a Harmony chat choice of header tokens only still opens with its role chunk at its Complete (#3531)
• fix(gateway): a Harmony chat choice whose only engine output is its Complete opens with its role chunk and ends with its finish chunk (#3529)
• fix(symphony): the XML call grammar's leaves are bounded in bytes (#3528)
• fix(grpc_servicer): the MRoPE fact reads both rope sections of the checkpoint config (#3530)
• fix(gateway): a JSON response_format takes the table's answer tag under Symphony for every table (#3527)
• fix(engine_servicer): a vision grid is refused on the TokenSpeed ZMQ wire only where it feeds MRoPE positions (#3525)
• fix(tool_parser): a MiniMax forced-call grammar's leaves are bounded in bytes, and a fragment carries only the definitions it reaches (#3524)
• fix(gateway): a chat choice whose only engine output is its Complete opens with its role chunk (#3523)
• fix(grpc_servicer): TokenSpeed's Generate keeps a choice's empty first Chunk and closes the engine iterator before its finished check (#3520)
• fix(symphony): a shaped call's refused JSON value ends where it was refused, so its trailer can close the call (#3507)
• fix(server): disable Nagle buffering for TLS streams (#3516)
• fix(tool_parser): a tool schema's wide fan-out no longer unfolds the MiniMax forced-call grammar without bound (#3512)
• fix(symphony): the XML call grammar is bounded by an element budget (#3515)

Performance

• perf(multimodal): resize MiniMax M3 images once from source dimensions (#4681)
• perf(gateway): a worker's constrained-decoding labels are parsed once at construction, not on every request (#4718)
• perf(symphony): a streamed call's argument fragments borrow the delta (#4706)
• perf(gateway): a Messages stream keeps one buffer for the content-block machine's events (#4703)
• perf(worker): a worker's client lookup no longer carries the connect path's state on every call (#4683)
• perf(gateway): the protected routes' middleware is one stack in every configuration (#4673)
• perf(tenant): the copies of a request's route meta share its stamps instead of cloning them (#4672)
• perf(tokenizer): the byte-level table knows at load which tokens are whole, so a step reads them without validation (#4323)
• perf(engine): the decoded output batch of a ZMQ engine message is sized to its outputs, not to the allocator minimum (#4645)
• perf(tokenizer): a prepared tool keeps its tojson text, so a repeated tool set is not serialized per request (#4597)
• perf(registries): the per-request name lookups allocate nothing (#4623)
• perf(servicer): the Rust SGLang servicer sends large multimodal tensors as auxiliary zmq frames the scheduler's plugin reads in place (#4582)
• perf(symphony): a format's scanner markers are built once and shared, not copied per request (#4611)
• perf(gateway): the HTTP layers are applied as one stack per group, not one boxed level per layer (#4596)
• perf(gateway): the request deadline is applied to a stream's wrappers in place (#4570)
• perf(http): the streaming relays keep one wait for the client across the chunks (#4552)
• perf(gateway): the stalled-stream watch registers once per stream, not once per delta (#4550)
• perf(metrics): the per-request records that still built a key per call go through kept handles (#4529)
• perf(engine_zmq_adapter): a request's one sub-stream is polled without the merge (#4522)
• perf(engine_zmq_client): the scheduler stats decode the load fields and skip the rest (#4492)
• perf(engine_zmq_client): an engine output's rare payloads ride boxed (#4487)
• perf(gateway): the ZMQ submit future is boxed at the backend dispatch, so a gRPC request's future is not sized by it (#4468)
• perf(gateway): the template value of a request's tools is prepared once per distinct tool set (#4427)
• perf(tokenizer): the stop decoder checks each token against sorted stop ids instead of two hash sets (#4282)
• perf(tokenizer): the matcher's ASCII class tests read a membership table (#4337)
• perf(tokenizer): the decoders append a token's text to the caller's buffer instead of returning a string per token (#4292)
• perf(metrics): the per-worker gauges set on every request go through kept handles (#4264)
• perf(stream): a streamed chat delta is written from the buffer it was read into, not copied into a chunk of its own (#4295)
• perf(tokenizer): the pre-tokenizer matcher runs a compiled program with an explicit choice stack (#4245)
• perf(tokenizer): the pre-tokenizer matcher skips the alternatives a character cannot begin and walks one-way nodes in place (#4228)
• perf(tokenizer): the prefix cache learns the boundary after a hit, so a shared prefix behind a leading token is served (#4209)
• perf(multimodal): the Pillow-parity resize runs one 32-bit RGB pass pair, and the DeepSeek V4.1 pad and patchify stop copying (#4219)
• perf(gateway): the stream-or-buffer decision runs as a lean route layer, boxing a future only for a request it may stream (#4144)
• perf(gateway): the legacy admission runs as a lean route layer, boxing a future only for a request that waits for a token (#4148)
• perf(engine): large tensors ride as auxiliary zmq frames on every typed-tensor wire (#4145)
• perf(gateway): the request body timeout layer runs lean, and the legacy concurrency layer is installed only with a rate limiter (#4114)
• perf(gateway): the chat and generate dispatcher stages box their inner stage once, not twice (#4113)
• perf(gateway): a request's context moves into its pipeline future once, not four times (#4111)
• perf(gateway): the text log lines are written with the stock format's bytes and a third of its work (#4105)
• perf(tokenizer): the chat template's tojson filter writes a value straight into its JSON text, not through a tree (#4103)
• perf(gateway): the tenant route-meta layer runs lean, and the serving auth and wasm layers are installed only when they have work (#4101)
• perf(symphony): a streamed delta is built without the role the stream drops (#4095)
• perf(vllm): send large tensors as auxiliary zmq frames (#4093)
• perf(protocols): a typed JSON body is parsed once; only a body that fails is parsed again to name the failing field (#4086)
• perf(gateway): the Messages envelope, SSE keep-alive and trace-context layers cost nothing on requests they have nothing to do for (#4083)
• perf(symphony): the reader keeps a shared table's no-calls and no-reasoning variants instead of rebuilding them per request (#4084)
• perf(gateway): a UUID v7 takes its random bits from the thread's generator, not from a getrandom call each (#4080)
• perf(symphony): a step's events fill the caller's buffer and borrow the delta's text (#3764)
• perf(symphony): the scanner reads a delta after held bytes without allocating, seeks a marker's first byte a word at a time, and is not rebuilt at the end of a stream (#3987)
• perf(protocol): a chat request's tools are shared with the response spec, not copied into it (#3925)
• perf(symphony): a reader reads the request's tool schemas in place instead of indexing them first (#3916)
• perf(gateway): where a Symphony table reads the response, no legacy reasoning parser is built to answer for it (#3874)
• perf(policies): the registry keeps whether a policy wants the request text where policies change, not per request (#3910)
• perf(gateway): the one-pass wire write is the default now that it beats the two passes at every image size (#3852)
• perf(gateway): the body limit caps the request alone and the load guard rides in the body's own stream (#3627)
• perf(symphony): a table is built once and shared by every request and engine that reads with it (#3721)
• perf(symphony): a streamed delta is read in place, without the copies and allocations each delta made (#3723)
• perf(router): the selection snapshot keeps no copy of the prompt text next to token ids (#3675)
• perf(symphony): the decoder's special-token check is decided once per model, tokenizer and markers (#3719)
• perf(router): Harmony detection reads a worker's model cards in place (#3674)
• perf(metrics): the router's per-request records keep their handles per worker instead of rebuilding the key on every record (#3599)
• perf(gateway): the HTTP edge wraps a response body once, not four times (#3590)
• perf(gateway): a chat request's tool schemas are shared, not copied, between preparation, the spec and the reader (#3703)
• perf(responses): a tool-call delta copies the call's id and name only when it announces the item (#3701)
• perf(build): the release profile optimizes for speed (opt-level 3) (#3715)
• perf(responses): a previous-response chain is read in place, not copied out of the memory store on every turn (#3708)
• perf(symphony): the registry's repository names are worked out once, not on every lookup (#3689)
• perf(tokenizer): the direct encoder encodes a long prompt unit by unit on the thread pool (#3447)
• perf(tokenizer): the BPE word-cache bound goes on the model as it is read, not on a copy of it (#3583)
• perf(gateway): the log writer threads write queued lines in batches instead of one write per line (#3573)
• perf(tokenizer): the exact-match cache hashes a request once, with a keyed hash at memory speed, and compares the text on a hit (#3550)
• perf(multimodal): the one-pass wire write beats the two passes at every image size, with the same bytes (#3549)

Documentation

• docs(gateway): the EPD selection comment says what the encode pool holds since the direct lane took the encode role (#4711)
• docs: three private rustdoc links read as code and -D warnings gates the workspace docs (#4690)
• docs(router): the streamed request path's help text says the body reaches the worker as the client sent it (#4687)
• docs(servicer): the subdirectory shape of the shared socket directory says the mount must be writable by the gateway's uid (#4628)
• docs(servicer): the headless-over-ZMQ section says why a shared ipc socket directory is refused and how a pod hands it to the gateway (#4616)
• docs(gateway): the rendered-prefill reasoning doc names its crate-private helper as code (#4575)
• docs(gateway): the test trap's doc comment names its cfg(test) arming function as text (#4482)
• docs(gateway): the two logprobs conversion docs write their return type as code (#4464)
• docs(servicer): the warm-up refusal comments point at UnservedRequest for the stop rule (#4473)
• docs(servicer): the plain-DP prefill refusal sentence belongs to SGLang (#4469)
• docs(gateway): the policy registry's key-order comment says the header comes before the rid (#4465)
• docs(engine_zmq_adapter): the encode dispatch sink's doc names the crate-private notice as code (#4455)
• docs(gateway): the tenant rate-limit module's docs describe the wired state (#4458)
• docs(engine_zmq_adapter): the TokenSpeed module's four private-doc links resolve or read as code (#4446)
• docs(engine_zmq_adapter): six rustdoc links name their private targets as code, and one names the method it means (#4268)
• docs(gateway): five private-doc links resolve or read as code (#4428)
• docs(crates): fourteen private-doc links in seven sibling crates resolve or read as code (#4423)
• docs(symphony): eleven private-doc links name their targets by path or read as code (#4425)
• docs(crates): two rustdoc links in the tokenizer and the gRPC client name their private targets as code (#4409)
• docs(multimodal): the rate-walk sampler's doc names its private helper as code (#4400)
• docs(gateway): the least-load policy's doc names its private helpers as code, and the model-id-from flag's placeholders are code spans (#4378)
• docs(gateway): the streamed assistant choice's doc names the opening-choice helper that carries the role (#4380)
• docs(crates): eight rustdoc warnings in the tokenizer, the protocol crate and Symphony read as code or sit in one code span (#4329)
• docs(gateway): the start-up launcher's doc names where it is declared and reached, and the mock MCP server drops its old-path line (#4313)
• docs(crates): five rustdoc links in the servicer and the multimodal crate name their private targets as code (#4312)
• docs(tokenizer): twelve rustdoc lines name their private targets and a generic as code (#4275)
• docs(symphony): sixteen rustdoc links name their private targets as code, and one drops its redundant path (#4274)
• docs(tool_parser): seven reference URLs are hyperlinks and two rustdoc links name their private targets as code (#4273)
• docs(protocols): eleven reference URLs are hyperlinks and two rustdoc links name their private targets as code (#4272)
• docs(engine_zmq_client): ten rustdoc lines name their private targets as code, and the SGLang request type by its path (#4270)
• docs(crates): fourteen rustdoc links in six crates name their private targets as code (#4267)
• docs(multimodal): eight rustdoc lines name their private targets and tensor indices as code (#4269)
• docs(grpc_client): the media reference's proto comment reads its URL schemes as code (#4278)
• docs(crates): three rustdoc lines in the auth, MCP and reasoning-parser crates read their examples as code (#4277)
• docs(gateway): two rustdoc links to the private liveness module read as plain code (#4246)
• docs(servicer): the NIXL in-place restart recipe carries its measured cost and the sweep's real timing (#4213)
• docs(tau2): the validation note says why TP=1 held for the 27B model (#4201)
• docs(servicer): restarting a prefill/decode leg in place on one host over NIXL (#4200)
• docs(symphony): the mistral notes say which template a request with tools takes (#4199)
• docs(tau2): the validation note drops an environment-specific detail (#4189)
• docs(wasm): the crate README's layout tree and interface links name the paths the crate has today (#4185)
• docs: drop two pointers to design documents that are in neither repository (#4184)
• docs(contributing): drop the SMG_RUNNER_GPU_8 row, a runner variable no workflow reads (#4183)
• docs(readme): the two feature links that answered 404 point at the pages' published routes (#4182)
• docs(gateway): eight doc comments in the worker and observability modules name private items as code, and one ambiguous link names its target (#4176)
• docs(gateway): six doc comments in the worker and config modules name private items as code, not as links (#4108)
• docs(gateway): the middleware's ten doc comments name private items as code, not as links (#4104)
• docs(gateway): the routers' ten doc comments name private items as code, not as links (#4106)
• docs(gateway): twenty-one doc comments name private items as code, not as links (#4102)
• docs(gateway): ten doc comments name their items so rustdoc resolves them (#4099)
• docs(gateway): the crate root carries a module map (#4097)
• docs(gateway): the routing snapshot's doc line stands on RoutingState again (#4025)
• docs(multimodal): the worker-mode README names the encode-pipeline exception to the 503, and the registration warning says media references, not multimodal requests (#4026)
• docs(multimodal): the decoder report's comment says the gauges are set on every report and the log lines once (#3980)
• docs(symphony): the DeepSeek V3 table intro says the table reads a thought its template never writes (#3972)
• docs(mesh): the README describes what an operator can read of the mesh, not HTTP routes the gateway does not serve (#3834)
• docs(symphony): the architecture document (#3732)
• docs(servicer): the test fixture's notes say that teardown puts the shell's selection back (#3793)
• docs(symphony): only the Chat Completions stream shows a kept state's whitespace alone (#3699)
• docs(symphony): a kept state's stream can carry whitespace the whole answer drops (#3698)
• docs(tokenizer): the configured BOS is added on request, and a cross-check compares with the same pipeline (#3579)

Maintenance

485 maintenance changes (dependency updates, CI and build) are listed in the full changelog.