Shepherd Model Gateway v1.13.0

SMG v1.13.0 makes the gRPC serving path faster, lighter and closer to the engines behind it. On the same 18 engines as v1.12.0, it serves nearly twice the image traffic and finishes those requests 3.3× sooner, with 38–56% less CPU per request and about half the router memory. It also serves 1.63× the image traffic of another open-source router on the same hosts. The tokenizer, media pipeline and chat templates now produce what the engines' own processors produce, and the OpenAI and Anthropic routes answer the way the public APIs do. The gateway refuses misconfiguration and overload up front instead of failing later, and this release closes several security gaps, listed first under Notable fixes. Symphony, a single table-driven parser for reasoning and tool calls, now runs inside the gateway as an opt-in beta.

This release includes 567 commits from 11 contributors, including 2 first-time contributors. Some defaults changed, and some deployments need a step before upgrading: see Before you upgrade.

The fleet figures come from this release's validation run, which compared v1.12.0 and v1.13.0 on the same hosts, engines and traffic. Other figures come from benchmarks of the individual changes. They measure different workloads and should not be combined into one overall speedup. For latency, p99 means 99% of measurements were at or below that value.

Before you upgrade

Postgres or Oracle history store: this release adds schema migration 4. Set auto_migrate: true in the store's schema config (or DB_AUTO_MIGRATE=true), or apply the statements the gateway prints, and create the sweep index on existing tables (see Storage).
Browser clients: CORS is closed unless --cors-allowed-origins is set. Pass '*' to keep v1.12.0's open policy.
Media from internal hosts: URLs that resolve to private or loopback addresses are refused. Set --mm-allow-private-media-hosts on the gateway, and SMG_MM_ALLOW_PRIVATE_MEDIA_HOSTS=1 on workers that run SMG's media pipeline.
Helm: the readiness probe moved to /readiness, so a router whose workers register only through the admin API never becomes Ready; set router.readinessProbe.path: /health for that setup. A custom router image must run as UID 65532 or override router.securityContext.runAsUser.
Stricter start-up: an unreachable Postgres or Redis store, a mesh that cannot start, and a list of malformed settings now stop the gateway before it binds a port. Check your configuration against Configuration that now refuses to start.
Engine pods on the Rust servicer: roll gateways first, then install smg 1.13.0 together with smg-grpc-servicer 0.14.0. The new servicer fails with TypeError against the 1.12.0 smg wheel.
Python Router(...) users: metrics now bind all interfaces when prometheus_host is unset (v1.12.0 bound 127.0.0.1).
Dashboards and API clients: two metrics became histograms, error labels and error envelopes changed, and some requests that used to pass are now refused with 400. See Upgrade notes.

Highlights

Nearly twice the image traffic on the same GPUs, at less than half the CPU per request

v1.12.0 and v1.13.0 ran in turn on the same 18 GPU hosts, with the same vLLM 0.31.0 engines and the same traffic:

Traffic	Requests/s	Median request time	CPU per request	Router memory
One image per request	61.4 → 120.0 (1.95×)	2.59 → 0.77 s (3.3× faster)	56% less	47% less
Short prompts, shared prefixes	65.4 → 66.6	2.69 → 2.67 s	38% less	55% less
Long multi-turn conversations	76.2 → 76.1	0.58 → 0.57 s	48% less	43% less

Text traffic is paced by the engines, so its throughput holds. The gain there is cost and tail latency: TTFT p99 is 6–9% lower.
The tokenizer encodes a 1M-token prompt 14.6× faster (1,536 → 105 ms) and decodes streams 19–44× faster per token, with identical output.
vLLM engine pods on the Rust servicer use about 1 GB less memory each.
On a single engine with no routing, SMG serves 6% more short-prompt requests per second than vLLM's own server (3.75 vs 3.55), with half the CPU per request and 60% less memory outside the engine.

Each cell is a single 30-minute run with one router in front of 18 engines at tensor parallel 4, under cache-aware routing, with no server errors on either build. CPU counts the router and the engine-side servicers together. In the single-engine comparison both sides ran the same engine flags, and CPU and memory outside the engine count SMG's gateway and servicer against vLLM's API server. v1.12.0 preprocessed images in the router, because its engine image predates engine-side preprocessing.

1.63× the image throughput of another open-source router

Another open-source router with KV-aware routing ran on the same 18 hosts and engine settings as v1.13.0:

1.63× the image throughput (120.0 vs 73.7 requests/s), with image requests finishing 2.5× sooner (median 0.77 vs 1.91 s).
8% more throughput on long multi-turn conversations (76.1 vs 70.4 requests/s).
31% more throughput at the top of a load ramp, past saturation, and more throughput with a lower median TTFT at every step.
2–4× less CPU per request and less than half the memory in the process that fronts each engine (4.1–10.0 vs 15.2–27.9 ms per request; 1.2–1.3 vs 2.7–2.8 GB).

As above, each cell is a single 30-minute run. Each router used its own KV-aware routing, and the other router ran on its own vLLM build. Its frontend's CPU was not measured, so the CPU and memory figures compare only the process inside each engine pod.

Running the configuration behind these numbers

Python remains the default servicer and positional the default KV index in v1.13.0, so the configuration measured above is opt-in:

# Each engine host: vLLM behind SMG's Rust gRPC servicer and SMG's media pipeline.
# The Rust servicer switches on the engine's KV-cache event publisher by itself.
vllm serve <model> --grpc --port 50051 --servicer-impl rust --mm-processor smg

# The router: cache-aware routing on the chain index, fed by those KV-cache events,
# with images and video preprocessed on the engines.
smg launch --worker-urls grpc://engine-0:50051 grpc://engine-1:50051 \
  --policy cache_aware --kv-index chain --mm-processing worker

TokenSpeed (python -m smg_grpc_servicer.tokenspeed --servicer-impl rust) and SGLang (sglang serve --grpc-mode --servicer-impl rust) take the same switch. The SGLang Rust servicer does not carry images or video yet.

A faster tokenizer that matches the reference exactly

Native encode for byte-level BPE vocabularies (Qwen, GPT-2, GLM, DeepSeek, MiniMax and their relatives), on by default. It produces the same ids as the tokenizers library for about a sixth of the router CPU on 32k-token prompts (33.5 → 5.6 ms per request). A 1M-token prompt encodes in 105 ms instead of 1,536 ms, and router memory under 64 concurrent 32k-token streams falls from 2.3 to 1.0 GB. Requests that add special tokens, such as embeddings, take the same path.
Streaming decode without re-decoding. Each stream keeps an incremental decoder: 456 → 17 ns per token for byte-level vocabularies, 1,064 → 24 ns for tiktoken vocabularies, and 1,124 → 59 ns for sentencepiece-style vocabularies with byte fallback (the Llama 2, Mistral and Gemma generations).
Exact. 5,578,285 of 5,578,285 encode and incremental-decode fixture cases match the reference tokenizers across 78 recorded models. The optional L1 prefix cache now returns the same ids as a plain encode on every vocabulary, with 0 mismatches in 176,904 comparisons.
Bounded memory. The L0 cache has a byte budget (--tokenizer-cache-l0-max-memory, default 256 MiB), and the per-thread BPE word cache is capped, which took a router's live heap in a steady run against 46 mock workers from about 930 MB to about 290 MB.
Opt-in tokenizers-v1 build feature, off in the published wheels and images: 6.6–11.1× faster encodes of 80k-token prompts and about 17× faster full byte-level decode on MiniMax M3, GLM-5.3, Qwen3-8B and DeepSeek-V3.2.

Symphony: one parser for reasoning and tool calls, now in the gateway (opt-in beta)

Symphony reads a model's output through one table per format and emits content, reasoning and tool calls as typed events. The same table derives the grammar that constrains a forced tool call. v1.12.0 shipped Symphony as a library; v1.13.0 connects it to the gateway.

Switch it on per gateway or per request with --parser symphony (default legacy) or the request header X-SMG-Parser: symphony. The response header x-smg-parser names the parser that read the response. Chat Completions and Messages read through Symphony; Responses run through the Chat pipeline.
No parser flag needed for known models. A registry picks the table from the model id. Existing --tool-call-parser names such as qwen, glm47_moe and deepseek_v41 resolve to their tables, and a table can be pinned per model by name or worker label.
Twelve new formats: GLM 4.5 and 4.6, K2-Horizon, MiniMax M2 and M2.7, Inkling, MiniCPM5, dots3, Gemma 4, Mistral (Devstral 2, Ministral 3, Mistral Medium 3.5, Mistral Small 4), Llama Nemotron, FunctionGemma, bare JSON calls, and arguments-only forced calls.
A model's own response template. The parser name hf compiles the response_template a checkpoint declares, in the transformers 5.18 format, into a Symphony table through the new smg-response-template crate.

smg-symphony 0.1.0 and smg-response-template 0.1.0 are now published on crates.io. Known gaps are listed under Known limits.

Forced tool calls that keep the model's reasoning

Shaped for the engine that serves them. With tool_choice: "required" or a named function on a reasoning model, the servicers now tell the gateway where their engine starts a grammar, and the gateway shapes the constraint for the worker it picked, again on a retry. On a prompt the model would otherwise answer in text, the forced call was honored 7 of 8 times instead of 0 of 8.
MiniMax M3 forced calls use the model's own call syntax and let it think first: sampled answers went from 9 of 20 invalid with no reasoning to 0 of 20 invalid, matching the engine's own server, on both parser paths.
GLM-4.7 and Hunyuan 4 forced calls on thinking prompts no longer continue with invented text after the call.
Always-on constraints per worker. A worker can declare constrained_decoding: always so that tool_choice: "auto" requests with tools are constrained too.

OpenAI and Anthropic APIs that behave like the public ones

Error envelopes clients can key on. OpenAI-shaped routes return the public error.type vocabulary with a string error.code and error.param, and refusals carry the public codes such as missing_required_parameter, unknown_parameter and invalid_type. /v1/messages returns Anthropic's error envelope, and unknown routes answer with JSON.
Complete response objects. Responses, Chat Completions and Messages objects carry every always-present public field, so clients generated from the public schemas parse them. Streamed usage now equals the usage of the same request answered without streaming.
Validation before generation. Requests the public APIs refuse get a 400 naming the parameter before any generation. Rules specific to OpenAI's own models apply only to OpenAI model names, so self-hosted models keep their vendor extensions.
Anthropic Messages end to end on gRPC workers: POST /v1/messages/count_tokens (equal to the request's usage.input_tokens), GET /v1/models/{id}, Anthropic-shaped model lists, max_tokens: 0, document blocks, assistant prefill and output_config.effort.
Responses API background mode. background: true answers a queued response at once (4.9 ms instead of 3 s against a mock worker). Poll it with GET /v1/responses/{id}, cancel it with POST /v1/responses/{id}/cancel, or stream it. Gateways that share a history store agree on its state, and --max-background-responses (default 1,024) bounds how many run at once.
SSE keep-alives. Streams send a keep-alive every 15 seconds while the engine is silent, so proxies with idle timeouts no longer cut long prefills. Set --sse-keepalive-secs to change the interval.

Leaner, sturdier engine servicers

One process per engine. The Rust servicers start the engine's cores (vLLM) or scheduler ranks (TokenSpeed, SGLang) in their own process instead of a second Python interpreter. On vLLM that saves about 1 GB per engine pod, and the engine cores start 6 seconds sooner.
Cache-aware routing without extra configuration. The Rust launchers turn on the engine's KV-cache event publisher unless the launcher passes its own events configuration; SMG_<ENGINE>_SERVICER_KV_EVENTS=0 opts out.
One switch on every engine: --servicer-impl {python,rust} on the vLLM, TokenSpeed and SGLang launchers, with the environment variable as a fallback.
Less CPU per token. The Rust servicer's runtime defaults to one worker thread: servicer CPU per output token fell from 28 to 20 µs at 128 streams, with client inter-token latency unchanged.
Failures stay local. A stalled engine core fails only the request it cannot accept, with gRPC UNAVAILABLE, instead of taking the whole servicer down. A request still open when the engine dies ends with an error instead of a quiet finish_reason: "abort". Launchers validate their settings before binding a port and exit with a clear code.
Engines describe themselves. Servicers report the engine's running window, per-request media limits and encoder dtype. Prefill/decode admission now engages on vLLM and default-flag SGLang decode workers, and media limits are enforced before any fetch.
Reinforcement-learning control for TokenSpeed over gRPC. /v1/rl/* reaches TokenSpeed engines served over gRPC through their rl.control_url worker label.

Routing that holds up under load and failure

A safe default for in-flight requests. With --max-concurrent-requests unset, the router bounds in-flight requests at 1,024 per core (at least 4,096) and answers the surplus with 429 and Retry-After, instead of queueing until memory runs out.
Circuit breakers and retries that agree. A request charges a worker's breaker at most once, a worker's own 500 is not replayed, and retries move to engines not yet tried. A worker whose health probes pass again is routable at once: after an engine restart, workers rejoined in 1.9 seconds instead of 50.5.
Silent workers leave routing quickly. A worker whose pushed load reports stop is excluded after --worker-stale-secs (default 15 seconds) instead of the 40-second transport timeout.
Fair spreading. Consistent hashing no longer keys on the Authorization header. On 46 engines with 480 clients sharing one credential, throughput rose from 34 to 163 requests/s and TTFT p99 fell from 12.9 s to 614 ms. The load scorer also stops herding onto workers whose load reports lag.
File-based discovery. --discovery-provider file --discovery-file PATH follows a JSON manifest that another system maintains, and rereads it on change. In-place edits to a worker's manifest record or Pod annotations now apply without a restart.
Connection and body timeouts. Connections that never send a request head close after 30 seconds and idle keep-alive connections after 90 (--header-read-timeout-secs, --idle-connection-timeout-secs). A stalled request body gets 408.
Disaggregated prefill/decode. A silent prefill worker is given up on within seconds with 504 prefill_worker_silent, instead of holding the request for up to 30 minutes, and a stalled decode leg is charged to the decode worker. During a leg outage a pair answers 503 with Retry-After instead of 404 model_not_found. Admission is first come, first served, which cut TTFT p99 from 11.6 s to 456 ms in the starvation case at unchanged throughput. Image and video requests pair with vision-capable legs, cached_tokens reports the prefill's real prefix-cache hit, and KV-store connectors (Mooncake Store, SharedStorage, FlexKV, HF3FS) get their own profile.

Models and multimodal

Gemma 4: images bit-identical to the reference processor and video sampled the way the engine's loader samples it, with prompt token counts equal to the engine's on 46 of 46 media requests; tools, reasoning and audio references over gRPC.
NVIDIA Nemotron-3-Nano-Omni (Nemotron-H Omni) images, video and audio on TokenSpeed.
Qwen3-VL video through router-side processing: 14 of 14 test clips with the engine's own prompt tokens, where every clip failed before.
Engine-exact preprocessing. Router-side image preprocessing for the Qwen2-VL, Qwen3-VL, Qwen3-Omni and MiniMax-M3 processors is bit-identical to the reference processor on 19,496 of 19,556 corpus images (18,843 of 18,903 for MiniMax-M3), and greedy answers that matched the engine's went from 17 of 40 to 40 of 40. Transparent images follow the serving engine's own rule.
A faster media pipeline: resize 23.2 → 5.0 ms per 1296×972 image, the pixel write 12.35 → 1.78 ms, a WebP image 81.3 → 21.9 ms end to end, and a 600-second video 773 → 393 CPU-ms. The gateway image now ships libjpeg-turbo, so JPEGs decode the way the engine decodes them.
GLM: GLM-5.3 accepts 64 images and 2 videos per request, and a GLM 5 video is sampled and budgeted the way the engine does it: a 9-second clip costs 30,037 prompt tokens against the engine's 30,045, instead of 90,361.
Chat templates render as the engine renders them: DeepSeek V4 thinking defaults, Kimi K3 thinking and long-prompt windows, developer messages, Olmo 3, Step 3, and templates that use transformers' Jinja extensions (IQuest-Q1, K2-Horizon-36B, Laguna-XS.2, LFM2.5-1.2B-Instruct, llava-1.5-7b-hf, Trinity-Mini, Apertus).

Operations and observability

Configuration errors stop the start. About fifteen misconfigurations that used to start, including a control plane left open by a malformed key list, now refuse with the field named before any port binds.
Mesh mTLS that forms (--mesh-tls-ca-cert, --mesh-tls-cert, --mesh-tls-key), and IPv6 throughout: listeners, the metrics endpoint, trace export, worker URLs, the servicers and the realtime WebRTC bridge.
Storage that fails fast. Every Postgres and Redis call is bounded (5 seconds by default). An unreachable store answers 503 storage_unavailable, or stops the start, instead of hanging. Postgres calls take one round trip.
Reproducible builds: a single rustls backend (ring) in every binary, with the aws-lc C library no longer built, and the Rust toolchain and container base images pinned.
Helm: readiness on /readiness, a 210-second termination grace so long streams drain, and a non-root router container.
Traces into the engine. The gateway passes W3C trace context to vLLM and SGLang, so engine spans join the gateway's trace, and the upstream span covers the whole stream.
Metrics and profiling: smg_worker_requests_total{worker,model}; a removed worker's series disappear; families exist at zero from start-up; every family has help text; PD time to first token splits into phases that add up; POST /heap_profile writes a jemalloc heap profile (--jemalloc-prof-dir).
Less memory: jemalloc with 16 arenas cut router resident memory from about 1,186 MB to 476 MB at the same live heap, and the in-memory history store is now bounded.

How this release was validated

Provider verification suites for MiniMax and Kimi (the vendors' public kits), DeepSeek and GLM were replayed against the release build over the Rust servicer. Nothing regressed against the previous candidate, and the remaining misses trace to test-deployment media limits, harness connection cuts and sampling.
Failure drills on 46 real engines with the release candidates (engine kill, engine-core freeze, router restart, router-to-engine partition): unreachable engines left routing within 15–17 seconds and returned after the heal, with no hung requests. A 24-hour soak at 46 engines stayed clean.
Parity sets: 5,578,285 tokenizer cases across 78 models and a 19,556-image corpus across three processor families.

Notable fixes

These bugs shipped in v1.12.0. Problems introduced and fixed during this cycle are not listed.

Security

Control plane left open by a configuration typo. A malformed --control-plane-api-keys entry, or --jwt-issuer without --jwt-audience, was dropped with a warning, and the gateway then served POST /workers, DELETE /workers/{id} and POST /flush_cache to callers with no credential. Both are now start-up errors.
Cross-tenant access to stored state. With tenant identity on, a caller who knew another tenant's response or conversation id could read, continue, write into and delete it. Records now carry their tenant, and another tenant's id answers as unknown.
Media fetches could reach internal addresses. An allow-listed host that redirected could send the gateway or the worker anywhere, loopback and private ranges included, and router-side processing had no allow-list at all. Every redirect hop is now checked, and private addresses are refused by default.
JWT. A JWT setup failure could leave the admin routes unprotected, a token with a future nbf was accepted, and an unknown kid made the gateway refetch the issuer's JWKS on every request.
Open CORS by default. Without --cors-allowed-origins, browsers could call the bearer-token APIs from any origin.
Secrets and identifiers in logs. An API key sent in the query string was written to the access log, and an x-request-id of any length was echoed and logged.
Dependencies. 29 RustSec advisories against the v1.12.0 lockfile are closed, among them the critical RUSTSEC-2026-0327, by moving to rustls 0.23.46, wasmtime 49.0.3 and quinn-proto 0.11.18.

Streaming, API output and tool calls

One message per token in some clients. gRPC chat streams set the assistant role on every chunk; it is now sent once, in the opening delta.
Cut streams looked successful. A stream the engine ended mid-way reached the client as HTTP 200 with an internal_error event, or through the HTTP relays as a closed connection without [DONE], and nothing counted it. It now ends with a typed stream_error event and counts against the worker and its breaker.
Wrong usage numbers. Streamed usage differed from non-streamed usage, and through a prefill/decode pair cached_tokens read prompt_tokens - 1 on every request.
Tool markup in visible text. tool_choice: "none" still rendered the tools, so models wrote call markup into content. A streamed DeepSeek DSML call leaked its closing tag, and a Kimi K3 answer cut by max_tokens ended with <|close|>message.
Requests that changed on the way. max_tokens reached HTTP workers renamed to max_completion_tokens, so an upstream that reads only max_tokens used its own default length. parallel_tool_calls: false still returned every call.
Reasoning settings. GLM-5.3 returned its reasoning as content when a request asked to turn thinking off, which its template cannot do. On MiniMax models, reasoning_effort: "none" turned thinking off, streams without stream_options had no usage chunk, and a root message landed in the wrong template slot.

Storage and history

Memory history grew without bound. The default memory backend kept every Responses record, about 12 MB per 1,000 responses, until the process was killed. It is now bounded; see Upgrade notes.
Responses lost during store outages. With the history store down, POST /v1/responses answered 200 completed, and the id later answered 404. A response the client expects to be stored is now stored, or answered 503 storage_unavailable.
Postgres start-up. A fresh empty database was refused with "Schema migration required", two gateways starting together raced, and a rolled-back image started on a newer schema and then failed reads.
Conversations. On Redis, deleting a conversation answered 500 after deleting it, and a message added as plain string content was left out when the conversation was replayed.

Routing, metrics and multimodal

Requests that hung. An image sent to a model whose workers have no vision encoder never got an answer, and in prefill/decode mode an image could land on a prefill worker without one. Such requests now go to vision-capable workers, or are refused at once with 400 multimodal_not_supported.
A bad media URL took a worker out. Unfetchable client media opened a healthy worker's circuit breaker, so its text requests failed too. Such failures are now answered 400 or 502 media_fetch_failed and are not charged to the worker.
Stalled-worker detection misfired. It missed a frozen worker under light load, flagged a cache-heavy worker during a burst of cold prompts, and took a healthy worker out of rotation behind one long prompt.
KV events lost after a gateway restart against an engine with a large KV-state snapshot, which left cache-aware routing blind.
Metrics told the wrong story. A client that closed the connection counted as a 200 or not at all, the gateway's own 503s counted as backend_error, and a worker's 429 counted as internal_error.
Log storms. A worker outage with retrying clients wrote about 1.7 million ERROR lines in five minutes. Each refusal is now one rate-limited WARN per model.

Engine servicers and start-up

Engines that never came up. Under the Rust servicers, a vLLM model that took longer than 30 minutes to load, or a slow TokenSpeed or SGLang scheduler, restarted from zero forever. The deadline now follows the engine's liveness.
GPU time spent on cancelled work. Cancelling a streamed n > 1 request on the SGLang Python servicer let all n completions run to the end.
Start-up failures and crashes. --tls-cert-path with Kubernetes discovery could not start, a host without a native CA root store refused to start, --health-check-interval-secs 0 busy-looped a core, and a --log-dir naming a file or --runtime-worker-threads 0 panicked.

Upgrade notes

Storage

Schema migration 4 (Postgres, Oracle) adds a nullable tenant column to responses, conversations and conversation_items. A version-3 store without auto-migration refuses to start with Schema migration required (current version: 3, latest version: 4) and prints the statements to run. v1.12.0 replicas keep starting against a version-4 store, so a rolling upgrade works, and rows written before the upgrade stay readable by every tenant.

Create the sweep index on existing tables. v1.13.0 deletes expired store: false background records at start-up and every 10 minutes, and creates the index that sweep reads only with a new table. On an existing table, create it once (default schema shown; adjust table and column names for a custom schema), or each sweep scans the whole table:

-- Postgres
CREATE INDEX CONCURRENTLY IF NOT EXISTS responses_unstored_text_idx ON responses
  ((replace(raw_response::text, $$\u0000$$, $$\u0020$$)::json->>'status'))
  WHERE replace(raw_response::text, $$\u0000$$, $$\u0020$$)::json->>'store' = 'false';

-- Oracle (add ONLINE where your edition supports it, to keep writes flowing)
CREATE INDEX RESPONSES_UNSTORED_IDX ON RESPONSES
  (CASE WHEN JSON_VALUE(RAW_RESPONSE, '$.store') = 'false' THEN JSON_VALUE(RAW_RESPONSE, '$.status') END);

The memory history backend is bounded at 10,000 responses, 100,000 conversation items and one day, oldest evicted first; an evicted id answers 404. Raise --history-memory-max-responses or --history-memory-max-items, set --history-memory-retention-secs 0 for no time limit, or use a persistent backend.
The store must answer. A Postgres or Redis store that refuses the connection or does not answer within 5 seconds (--postgres-op-timeout-secs, --redis-op-timeout-secs) stops start-up, and at run time such a call answers 503 storage_unavailable.

Configuration that now refuses to start

Each of these used to start and now exits with the setting named, before any port binds, from the CLI, a config file or the Python binding:

A malformed --control-plane-api-keys (or CONTROL_PLANE_API_KEYS) or --jwt-role-mapping entry, or --jwt-issuer without --jwt-audience and the reverse.
With --priority-scheduler-enabled: a scheduler YAML that cannot be read, has unknown keys, or names an unknown --priority-scheduler-default-max-class.
A missing or unreadable --tokenizer-path, --model-path, --chat-template or --log-dir, and 0 for any of the four health-check settings (use --disable-health-check to turn probing off).
--cors-allowed-origins entries not written as a browser sends them (no path, no default port), a --health-check-endpoint without a leading /, invalid header names in --request-id-headers or --storage-context-headers, an empty --api-key, unsorted --prometheus-duration-buckets, a --prometheus-host that is not an IP address, NaN or out-of-range routing thresholds, and --redis-retention-days of 0 or below -1.
A mesh that cannot start, such as a --mesh-port already in use or unusable mesh TLS material.

Changed defaults

Admission: an unset --max-concurrent-requests (also the Helm default of -1) now means 1,024 in-flight requests per core, at least 4,096, with the surplus answered 429 admission_queue_full. Set 0 for unbounded admission.
Connections: a connection that sends no request head closes after 30 seconds, and an idle keep-alive connection after 90. Keep the idle timeout above that of any proxy in front of the gateway, or set either flag to 0.
SSE keep-alives: after 15 seconds of engine silence, OpenAI-shaped streams send a : ping comment and /v1/messages streams a ping event. --sse-keepalive-secs 0 turns them off.
Silent workers: a worker whose pushed load records stop is excluded after --worker-stale-secs (15). 0 turns this off.
Selection timeout: on gRPC-served models, --queue-timeout-secs (60) also bounds the wait for worker selection, answered 503 selection_queue_timeout.
Circuit breaker: a worker is charged one failure per request instead of one per retry attempt, and its own 500 is not replayed on it. Revisit a --cb-failure-threshold raised to compensate.
Rate limit: --rate-limit-tokens-per-second is now an enforced sustained request rate, with --max-concurrent-requests as the burst. Re-tune deployments that set it.
Consistent hashing no longer uses the Authorization header as an implicit routing key. Send an explicit routing key for per-credential affinity.
Silent prefill/decode legs fail with 504 after the wedge bound (--worker-wedge-secs, 3 seconds, stretched by the measured prefill rate up to 120) instead of waiting for --request-timeout-secs. --worker-wedge-secs 0 turns the bound off.
Background responses now run asynchronously. --max-background-responses 0 keeps v1.12.0's foreground behavior.
Python Router(...) metrics bind the unspecified address of the serving host's family when prometheus_host is unset, as the CLI does. Pass prometheus_host="127.0.0.1" to keep them local.
Helm: pods get up to 210 seconds to drain (router.shutdownGracePeriodSeconds plus router.terminationGraceMarginSeconds), and the router runs as UID 65532 with all capabilities dropped.
Media: video URLs get a 30-second fetch budget (images and audio keep 10), and cache_aware scales its spill margin with the pool's load, so small pools spread some hot-prefix traffic.

API behavior

Error envelopes: on OpenAI-shaped routes error.type follows the status (invalid_request_error, permission_error, rate_limit_error, server_error) instead of the HTTP reason phrase, error.code is a string with the public API's names, and error.param is always present. Messages routes, and any route called with anthropic-version, return Anthropic's envelope. Status codes are unchanged.
Requests refused that used to pass: Responses refuses unknown top-level fields (unknown_parameter), oversized metadata, unknown input roles and local_shell tools. Messages refuses a malformed anthropic-version, temperature or top_p outside 0–1, unpaired tool blocks and other shapes the public API refuses, and answers output_format with 400 unsupported_output_format instead of ignoring it. Chat refuses non-object tool parameters and non-token logit_bias keys, and applies OpenAI's own rules only to OpenAI model names.
Response shapes: a chat stream sends the role once, in its opening delta, and omits reasoning_content when there is none; response objects carry the public API's always-present fields; Responses echo tool_choice as sent instead of as a JSON string; conversation items drop the gateway-only created_at. Strict decoders must accept these.
Status codes: store outages answer 503 storage_unavailable; a tokenizer still loading answers 503 tokenizer_not_ready with Retry-After; a worker's gRPC CANCELLED and HTTP/2 faults answer 502 and are retried; a prefill/decode pair missing a leg answers 503 instead of 404; a stream the engine abandons ends with an error instead of finish_reason: "abort"; media fetch failures answer 400 or 502 media_fetch_failed.
Prompts render as the engines render them, so prompt token counts and prefix-cache keys change once for affected models. tool_choice: "none" renders no tool definitions (Kimi K3 keeps them, as its vendor does), a request-level reasoning_effort overrides chat_template_kwargs, and developer messages become system messages on templates without a developer branch. DeepSeek V4 now thinks by default with high effort; send thinking: false, enable_thinking: false or reasoning_effort: "none" to keep chat mode.
Images are preprocessed the way each engine does it, so answers on affected images, video prompt token counts and media cache keys change once. Custom gateway images should install the libturbojpeg runtime package for the same JPEG output.
Tool calls: parallel_tool_calls: false returns one call, a Responses function call with cut-off arguments is incomplete, streamed Messages end with stop_reason: "end_turn" on the engine's own stop token, and MiniMax streams without stream_options end with a usage chunk whose choices is empty.
Authentication: a token with a future nbf is rejected, tokens from a just-rotated key can be refused for up to a minute while JWKS refetches are throttled, and serving-plane 401s carry WWW-Authenticate and a JSON body.
Bounds: an inbound x-request-id over 128 characters is replaced with a generated id, worker admin calls reject malformed fields with 400, and conversation item ids over 64 characters get 400.
Prefill/decode cached_tokens reports the prefill's real prefix-cache hit, so cache-hit ratios computed from usage drop to the real value.

Metrics, logs and traces

smg_kv_event_lag_seconds and smg_worker_retry_backoff_seconds are now histograms. Rewrite quantile queries as histogram_quantile() over _bucket.
In smg_router_request_errors_total, 429s move from internal_error to rate_limit_error, the gateway's own 503s from backend_error to unavailable, and engine-reported stream failures use engine_error.
smg_http_responses_total counts a response once its body is out: a client disconnect is status_code="499", a body that fails after its head carries error_code="stream_error", and refusals carry their code instead of "". Success rates keyed on status_code="200" drop by those shares.
The model label is a served model id or other, never the client's string.
A removed worker's series disappear at the next scrape, and many families now exist at zero from start-up.
smg_kv_event_tail_depth is removed.
The access log records path instead of uri, No available workers refusals log one rate-limited WARN per model instead of ERROR lines, and several servicer start-up and readiness lines are reworded.
grpc_execute spans now last until the stream ends, so they show generation time rather than dispatch latency.

Engine side

Upgrade order: roll gateways first, then engine pods. A v1.12.0 servicer still registers with the new gateway, but the prefill/decode admission window, engine media limits, encoder dtype and engine-aware forced calls need smg-grpc-servicer 0.14.0. In Rust mode that servicer needs the smg 1.13.0 wheel in the same environment.
Prefill/decode admission engages once decode workers advertise their running window, so bursts beyond vLLM's --max-num-seqs, or the SGLang scheduler's running bound, are shed with 503. Size those settings for the bursts you expect.
Exit codes: launchers exit 2 on an invalid SMG_<ENGINE>_SERVICER_* value before binding a port, and the Python TokenSpeed and SGLang servicers exit 1 when their engine dies.
Start-up deadlines: SMG_<ENGINE>_SERVICER_STARTUP_TIMEOUT_SECS now bounds silence while the engine is alive, and the new SMG_<ENGINE>_SERVICER_STARTUP_CEILING_SECS (4 hours) bounds the whole start.
Rust servicer defaults: one runtime worker thread (set SMG_SERVICER_WORKER_THREADS to 2–4 above about 512 streams per engine), the engine's KV-cache event publisher on (SMG_<ENGINE>_SERVICER_KV_EVENTS=0 opts out), and engine cores or scheduler ranks started in the servicer's own process.
The SGLang Rust servicer reports no vision support, so image and video requests to it get 400 multimodal_not_supported.

Bindings and Rust crates

Rust 1.98 is needed to build from source.
Go: MessageDelta.ToolCalls is now []ToolCallDelta, reassembled by index. Rebuild libsmg_go with the package.
Python: Router(...) and RouterArgs keep every v1.12.0 positional slot; new parameters are appended or keyword-only.
Library crates with breaking changes in minor versions: openai-protocol 1.16.0, llm-multimodal 1.14.0, smg-grpc-client 1.15.0, llm-tokenizer 1.10.0 and data-connector 2.4.0 change public types, for example openai-protocol's ProviderProfile::Generic default and typed Responses tool_choice, llm-multimodal's MediaConnector::new, and smg-grpc-client's Bytes tensor fields. A caret requirement picks them up on cargo update, so pin exact versions until your code is migrated. In smg, RouterConfigBuilder drops discovery_config, enable_l0_cache and enable_l1_cache.
rustls: the crates link only the ring provider. A process that builds its own reqwest client without a rustls provider must install one first.

Packages and engines

smg 1.13.0 on PyPI and crates.io, with container images and Helm chart 1.13.0. Release images build on vLLM 0.31.0, the latest TokenSpeed nightly, SGLang 0.5.21 and TensorRT-LLM 1.3.0rc24.
smg-grpc-servicer 0.14.0 and smg-grpc-proto 0.4.27. Roll gateways first, then engine pods: the engine-aware forced-call shape, the prefill/decode admission window and the media limits come from the new servicer, which in Rust mode needs the smg 1.13.0 wheel beside it.
New crates: smg-symphony 0.1.0 and smg-response-template 0.1.0. The library crates moved to new versions, among them openai-protocol 1.16.0, llm-tokenizer 1.10.0, llm-multimodal 1.14.0, smg-grpc-client 1.15.0 and engine-servicer 0.2.0.
Building from source needs Rust 1.98.

Known limits

Symphony is opt-in and in beta. DeepSeek V4 models that use the DSML spelling from before V4.1 have no table yet, so keep them on legacy. A tool call that appears only in the conversation history, with no tools declared on the request, is dropped.
Forced-call grammars for XML tool syntaxes (MiniMax and Symphony's XML tables) are bounded in depth and recursion but not yet in total size, so a tool schema with very wide fan-out can produce a very large grammar.
The SGLang Rust servicer does not carry images or video. Serve multimodal SGLang models through the Python servicer.
KV-cache events are on by default only under the Rust servicer launchers. The Python servicers still need an explicit events configuration.
With --mm-processing worker and no worker that processes media, multimodal requests fail with 503 when they arrive instead of being refused at registration.
An engine core that freezes while holding a single pending request is not yet excluded from routing.
60 of the 19,556 corpus images, all 16-bit greyscale PNGs, still differ from the reference processor, and a very long inline video (30 minutes) does not complete.

Thanks to everyone who contributed, and welcome @AMD-yanfeiwang and @mrn3088!

Full changelog

What's Changed

fix(zmq): gate the /dev/shm tensor read and stale-socket unlink to unix by @slin1237 in #2892
fix(auth): reject private IPv4-mapped JWKS addresses by @hello-alexmcc in #2893
fix(auth): retain control-plane protection when jwt initialization fails by @hello-alexmcc in #2895
fix(tokenspeed): correct KV pressure and running-work scoring by @lightseek-bot in #2891
fix(servicer): launch the engine cores from the servicer process by @hello-alexmcc in #2898
fix(policies): consistent_hashing no longer hashes on the Authorization header by @hello-alexmcc in #2899
fix(auth): enforce the JWT nbf claim by @hello-alexmcc in #2900
fix(auth): refetch the JWKS at most once a minute on an unknown kid by @hello-alexmcc in #2901
fix(metrics): register descriptions after the recorder is installed by @hello-alexmcc in #2902
feat(launcher): the tenancy flags of the Rust CLI reach the Python launcher by @hello-alexmcc in #2903
fix(metrics): export the KV event lag as a histogram by @hello-alexmcc in #2904
fix(gateway): decode KV event messages larger than tonic's 4 MiB default by @hello-alexmcc in #2906
fix(gateway): back off a KV event stream that fails before its first batch by @hello-alexmcc in #2907
fix(policies): cache_aware keeps deep continuations on an idle holder during warm-up by @hello-alexmcc in #2908
feat(metrics): count requests dispatched per worker by @hello-alexmcc in #2910
fix(worker): name the worker in circuit breaker transition logs by @hello-alexmcc in #2911
perf(tokenizer): decode byte-level vocabularies incrementally without re-decoding by @hello-alexmcc in #2912
fix(servicer): cap KV snapshot chunks by encoded bytes as well as by block count by @hello-alexmcc in #2913
fix(metrics): publish protection, retry and breaker families at start-up by @hello-alexmcc in #2914
test(tokenizer): compare the bellwether fixtures for tiktoken vocabularies too by @hello-alexmcc in #2923
fix(launcher): run the Python-launched router with the gateway's jemalloc options by @hello-alexmcc in #2920
fix(observability): sample and cap the cache-trace evidence line by @hello-alexmcc in #2925
perf(tokenizer): bound the L0 cache by bytes and sample evictions uniformly by @hello-alexmcc in #2924
fix(admission): make --rate-limit-tokens-per-second a request rate under the concurrency cap by @hello-alexmcc in #2932
feat(servicer): enable vLLM's KV event publisher by default in Rust mode by @hello-alexmcc in #2934
fix(protocols): answer 413 for a chunked body over the payload limit by @hello-alexmcc in #2928
fix(admission): time out stalled request bodies on the buffered handler path by @hello-alexmcc in #2930
fix(gateway): zero a model's KV index gauges when its last worker leaves by @hello-alexmcc in #2929
fix(admission): bound in-flight requests by default instead of queueing without limit by @hello-alexmcc in #2927
feat(servicer): enable TokenSpeed's KV event publisher by default in Rust mode by @hello-alexmcc in #2940
feat(servicer): enable SGLang's KV event publisher by default in Rust mode by @hello-alexmcc in #2939
fix(middleware): log the trace id with each request and echo traceparent by @hello-alexmcc in #2933
fix(cli): bracket IPv6 worker hosts in smg serve and probe ports on both families by @hello-alexmcc in #2938
fix(servicer): bracket IPv6 hosts in the gRPC bind and warm-up targets by @hello-alexmcc in #2937
fix(grpc): keep token logprobs on streamed reasoning deltas by @hello-alexmcc in #2919
fix(grpc): return no top_logprobs when the request asked for none by @hello-alexmcc in #2918
fix(tracing): carry the trace context over gRPC to the engine's tracer by @hello-alexmcc in #2936
feat(gateway): a gauge for workers whose backend serves no KV events by @hello-alexmcc in #2935
fix(config): one bind-host rule for every listener, IPv6 with or without brackets by @hello-alexmcc in #2941
fix(mock-worker): accept a long prompt's Generate, with the message limit a flag by @hello-alexmcc in #2963
fix(middleware): let browser clients read the echoed trace context by @hello-alexmcc in #2945
fix(tracing): keep the upstream span open to the end of the response stream by @hello-alexmcc in #2944
fix(config): validate the OTLP traces endpoint as the URL the exporter dials by @hello-alexmcc in #2943
fix(config): default the metrics host to the serving listener's family by @hello-alexmcc in #2942
fix(pd): bracket an IPv6 bootstrap host in the Mooncake decode params by @hello-alexmcc in #2947
test(servicer): a worker reaped mid-read counts as dead in the lifecycle tests by @hello-alexmcc in #2948
fix(gateway): account a relay snapshot's blocks on the per-worker KV gauges by @hello-alexmcc in #2949
fix(tokenizer): decode tiktoken streams byte by byte instead of re-decoding a window by @hello-alexmcc in #2951
chore(gateway): gate the test-only WorkerIndexState re-export by @hello-alexmcc in #2952
fix(servicer): the --mm-* flags on the stock gRPC launcher, the env fallback quiet without a flag by @hello-alexmcc in #2953
fix(servicer): the media pipeline's refusal names the model type, the served name and the supported families by @hello-alexmcc in #2955
feat(multimodal): a Gemma 4 spec for the media pipeline, images and video by @hello-alexmcc in #2959
fix(servicer): the engine startup deadline follows the engine's liveness by @hello-alexmcc in #2972
fix(auth): case-insensitive Bearer scheme, plain audit fields and a serving 401 challenge by @hello-alexmcc in #2970
fix(metrics): export the worker retry backoff as a histogram by @hello-alexmcc in #2968
test(tokenizer): move the bellwether pin to the sets encoded with tokenizer.json by @hello-alexmcc in #2967
test(servicer): the KV events default read by its fields in either form by @hello-alexmcc in #2961
perf(grpc): serialize a chat chunk's invariant head once per stream by @hello-alexmcc in #2964
fix(gateway): start without a native CA root store when no TLS is configured by @hello-alexmcc in #2973
fix(observability): capture routing evidence for the response header without the log switch by @hello-alexmcc in #2950
fix(servicer): advertise the vLLM engine's running window, so PD admission engages by @hello-alexmcc in #2926
ci(fixtures): install git-lfs on the runner before unpacking the bellwether sets by @hello-alexmcc in #2982
fix(auth): admit the user role to read-only control-plane routes by @hello-alexmcc in #2969
test(servicer): the stub gRPC contexts answer invocation_metadata() by @hello-alexmcc in #2990
fix(servicer): the smg media pipeline honours the engine's --limit-mm-per-prompt by @hello-alexmcc in #2978
perf(grpc): send a text-only prompt's token ids without a second copy of its text by @hello-alexmcc in #2971
ci(pre-commit): pass on main again (script mode, recorded SSE fixtures) by @hello-alexmcc in #2984
test(gateway): the spec-less media refusal test uses a family without a spec by @hello-alexmcc in #2987
perf(servicer): one runtime worker by default, so per-token sends stop contending on the connection lock by @hello-alexmcc in #2977
perf(middleware): keep the request span out of the response body's per-frame polls by @hello-alexmcc in #2976
fix(mock-worker): serve the gRPC worker's connections with TCP_NODELAY by @hello-alexmcc in #2957
fix(replay): count reasoning deltas for TTFT and ITL, forward chat_template_kwargs by @hello-alexmcc in #2960
feat(servicer): --servicer-impl on the TokenSpeed launcher, the environment variable as its fallback by @hello-alexmcc in #2999
feat(servicer): the SGLang servicer advertises --limit-mm-data-per-request as mm_item_limits by @hello-alexmcc in #2995
feat(metrics): hits, misses, evictions and occupancy of the pixel cache by @hello-alexmcc in #2993
feat(mock-worker): fault hooks that hold or throttle admission by @hello-alexmcc in #2983
fix(middleware): answer 413 for an over-limit body behind the request-body watchdog by @hello-alexmcc in #2980
feat(servicer): --servicer-impl on SGLang's launcher, the environment variable as its fallback by @hello-alexmcc in #2998
fix(gateway): router-side media processing holds a request to the engine's advertised media limits by @hello-alexmcc in #2997
fix(router): the circuit breaker counts failed requests, not attempts by @hello-alexmcc in #2974
perf(grpc): drain the engine's ready chunks before yielding the chat stream by @hello-alexmcc in #2975
feat(worker): a liveness veto for a worker whose pushed load records stopped by @hello-alexmcc in #2988
feat(servicer): the vLLM servicer advertises the engine's per-prompt media limits as mm_item_limits by @hello-alexmcc in #2996
feat(gateway): one startup warning names every surface an unauthenticated start leaves open by @hello-alexmcc in #3000
feat(mock-worker): a fault hook that answers a status, stalls or cuts a stream by @hello-alexmcc in #2958
fix(replay): an opt-in connection per request, so a stall hunt never queues behind another stream by @hello-alexmcc in #2966
perf(tokenizer): a native encode path for byte-level BPE tokenizers by @hello-alexmcc in #2921
fix(tokenizer): DeepSeek V4 renders the engine's defaults, thinking on with the high effort by @hello-alexmcc in #2962
fix(multimodal): send Retry-After with the in-flight budget's 429 by @hello-alexmcc in #2992
fix(multimodal): convert video frames to RGB the way the engine's loader does by @hello-alexmcc in #3001
fix(protocols): a chat request keeps max_tokens as the client wrote it by @hello-alexmcc in #3002
fix(tokenizer): Kimi K3 renders the engine's thinking toggles, enable_thinking and reasoning_effort=none by @hello-alexmcc in #3004
fix(tokenizer): load a checkpoint that ships vocab.json + merges.txt by Hub name and in the fixture harnesses by @hello-alexmcc in #3006
fix(gateway): the chat role chunk goes out on the engine's first output by @hello-alexmcc in #3018
fix(e2e): the harness and the mock worker run on an IPv6 loopback by @hello-alexmcc in #3019
perf(http): poll the upstream send without a dispatcher so pooled connections inherit no span by @hello-alexmcc in #2981
fix(tokenizer): a chat template's generation blocks render their body, as transformers renders them by @hello-alexmcc in #3013
fix(symphony): a tagged value written between the tags directly keeps the newline it ends with by @hello-alexmcc in #3008
fix(tokenizer): encode Kimi checkpoints in the pieces their tokenizer cuts a text into by @hello-alexmcc in #3007
fix(router): a retry re-selects away from the engines the request already tried by @hello-alexmcc in #3003
fix(tokenizer): the tokenizer's own chat template outranks the processor's legacy chat_template.json, as the engine's tokenizer resolves it by @hello-alexmcc in #3014
fix(tokenizer): strftime_now in the chat template context, as transformers provides it, bound to an instant the render can name by @hello-alexmcc in #3015
fix(multimodal): sample metadata-less video the way the engine's loader does by @hello-alexmcc in #3017
fix(realtime): the WebRTC bridge gathers candidates in the bind address's family, IPv6 included by @hello-alexmcc in #3026
fix(gateway): media of a model without a registry spec passes through to a worker that processes it by @hello-alexmcc in #2954
fix(servicer): the smg media pipeline samples video to the engine's own frame budget by @hello-alexmcc in #2979
fix(parsers): stream Qwen XML string arguments as they are generated by @hello-alexmcc in #2915
fix(grpc): count reasoning tokens in usage when the reasoning parser ran by @hello-alexmcc in #2916
fix(protocols): serialize a tool-call message's content as null by @hello-alexmcc in #2917
fix(servicer): a worker-pod liveness probe that fails when the leader's store is gone by @hello-alexmcc in #2991
fix(tokenizer): string content takes the parts branch of an openai-format template, as on the engine by @hello-alexmcc in #2965
fix(router): the all-unavailable 503 carries Retry-After, like the gateway's other refusals by @hello-alexmcc in #2994
feat(observability): the routing evidence header on request, with its worker, and the KV integrity series from zero by @hello-alexmcc in #3023
feat(observability): POST /heap_profile dumps the gateway's jemalloc heap profile on request by @hello-alexmcc in #3033
ci(fixtures): register the git-lfs filters, and list what the 78-model pin's harnesses know to differ so the job judges the rest by @hello-alexmcc in #2985
ci(fixtures): the sccache stats step passes with a notice when sccache is not on PATH, so the step that failed before it stays the one that failed by @hello-alexmcc in #3045
test(fixtures): the two harnesses compare the models in parallel, one thread per CPU, the output in the manifests' order by @hello-alexmcc in #3036
fix(gateway): learn the KV block size as the largest stored block, not the first by @hello-alexmcc in #3029
fix(symphony): a Kimi K3 tag the end of the output cut short is dropped, not released as content by @hello-alexmcc in #3046
fix(symphony): a forced call's derived prefix is free text with nothing owed at its end by @hello-alexmcc in #3038
fix(tokenizer): a chat template's dict() builds a map from key/value pairs, as Jinja2's does by @hello-alexmcc in #3032
fix(gateway): the Kimi K3 reasoning parser is armed from the decision the prompt was rendered with by @hello-alexmcc in #3037
ci(fixtures): the test step renders the pin's recording day, so templates that write today's date match their references by @hello-alexmcc in #3025
fix(tokenizer): a Jinja template without a developer branch renders developer messages as system messages by @hello-alexmcc in #3022
fix(protocols): a tool definition keeps the fields beyond the function schema, so the chat template renders the tool as the request carried it by @hello-alexmcc in #3005
fix(cache_trace): record discovered TokenSpeed cache epochs by @lightseek-bot in #2956
fix(mesh): an IPv6 peer's TLS server name is its address, not the bracketed URL host by @hello-alexmcc in #3016
fix(metrics): drop a removed worker's series from /metrics by @hello-alexmcc in #2909
fix(grpc): send the chat stream's role once, in an empty first content delta by @hello-alexmcc in #2905
test(symphony/tool-parser): the three test literals of Function name the extra field, so both crates' tests compile again by @hello-alexmcc in #3057
docs(runners): the bellwether-record scale set's placement notes say what the pods prefer, not which node class takes them by @hello-alexmcc in #3052
fix(responses): the Response object carries every field the public API always returns by @hello-alexmcc in #3055
ci(fixtures): the three tests that read bellwether run at once, so the step takes the longest of them, not their sum by @hello-alexmcc in #3028
docs(servicer): the selected-token logprobs test says it runs without an engine by @hello-alexmcc in #3053
test(fixtures): the known-differences list follows the merged fixes: the tool-field class goes, and the two date-dependent templates list their remaining cases by @hello-alexmcc in #3059
fix(tokenizer): bound the BPE model's per-thread word cache by @hello-alexmcc in #3060
fix(tokenizer): detect the content format by the engine's rule, so a string-format template keeps its string content by @hello-alexmcc in #3050
test(symphony): the replay over every recorded model runs the models in parallel, one thread per CPU, each report in the table's order by @hello-alexmcc in #3048
test(fixtures): a harness thread that meets an error or panics stops the others after their model, and the fan-out has its own tests by @hello-alexmcc in #3056
docs(symphony): the comments name the serving engine's parsers, not a product by @hello-alexmcc in #3067
docs(tool_parser): the parser comments name the serving engines' parsers, not a product by @hello-alexmcc in #3068
docs(gateway): the shared modules' incidental engine mentions say "the engines" or "one engine" by @hello-alexmcc in #3070
fix(protocols): the Message object carries every field the public API always returns by @hello-alexmcc in #3072
fix(gateway): errors on the Messages API leave in the Messages envelope by @hello-alexmcc in #3073
fix(gateway): document blocks render as prompt text, unreadable sources are refused before the template by @hello-alexmcc in #3074
ci(test): the Rust tests build in a step of their own, so the 30-minute test timeout bounds the tests and not a cold compile by @hello-alexmcc in #3078
test(policies): the emptied-index warm-up slice test stores its holder as blocks, so the slice decides by @hello-alexmcc in #3079
fix(servicer): the SGLang and TokenSpeed worker pods get the leader-store liveness probe by @hello-alexmcc in #3081
perf(tokenizer): the direct encode path serves a request for special tokens too by @hello-alexmcc in #3082
docs(protocols): the field docs say "an engine extension", not which product by @hello-alexmcc in #3069
docs(tokenizer): comments, doc comments and the fixture script's docstrings name no specific serving engine by @hello-alexmcc in #3066
fix(responses): item_reference input items resolve against the response store by @hello-alexmcc in #3065
fix(tokenizer): a request without tools renders with tools and documents as none, as transformers passes them by @hello-alexmcc in #3021
fix(gateway): refuse a request held past the queue timeout before worker selection by @hello-alexmcc in #2989
feat(mesh): --mesh-tls-ca-cert/--mesh-tls-cert/--mesh-tls-key turn mesh mTLS on, and a plaintext mesh is named at start-up by @hello-alexmcc in #3024
fix(tokenizer): Kimi K3 keeps a <|media_pad|> spelling typed in a message as text, as the checkpoint's encoder does by @hello-alexmcc in #3035
fix(responses): refuse the request shapes the public API refuses, accept top_logprobs without include by @hello-alexmcc in #3063
fix(protocols): a served model aliased under a vendor name keeps that vendor's contract profile after the alias is resolved by @hello-alexmcc in #3077
test(gateway): the RL-flag-off test expects the JSON not-found envelope, not an empty body by @hello-alexmcc in #3089
ci(cache): the Rust cache is saved by complete runs on main only, and a scheduled warmer builds the unit-tests entry by @hello-alexmcc in #3085
fix(responses): the Response object echoes tool_choice as sent, not as a JSON string by @hello-alexmcc in #3083
fix(policies): the router's own in-flight count floors a worker's expected-wait cost by @hello-alexmcc in #2986
fix(mesh): the listener serves TLS and the dialer presents the node certificate when mTLS is on by @hello-alexmcc in #3020
fix(grpc): the chat stream opens with the public API's first delta by @hello-alexmcc in #3064
fix(tokenizer): a dropped byte-level BPE tokenizer leaves no word cache on the threads that encoded with it by @hello-alexmcc in #3076
fix(vllm): report multimodal encoder dtype by @Moersity in #3084
fix(tool_parser): a forced call on a thinking prompt follows free text, with no owed to the grammar by @hello-alexmcc in #3034
test(responses): the MCP tool_choice e2e case runs against the suite's mock MCP server, not a live one by @hello-alexmcc in #3090
fix(gateway): a named tool_choice renders every tool, and the request-level reasoning_effort is written over a kwargs entry, as the engine renders them by @hello-alexmcc in #3041
fix(reasoning-parser): a Kimi K3 tag the end of the output cut short is dropped, not flushed as content by @hello-alexmcc in #3010
fix(protocols): the chat completion object carries every always-present field by @hello-alexmcc in #3054
fix(gateway): the OpenAI-shaped error envelope speaks the public API's vocabulary by @hello-alexmcc in #3062
fix(tokenizer): long pieces are cached by the encoder too, within a byte budget by @hello-alexmcc in #3093
perf(tokenizer): keep the L0 memory setting at the positional tail and insert plain ids by @hello-alexmcc in #3094
perf(ci): overlap worker startup on independent GPUs by @ai-jz in #3042
perf(ci): reuse compatible ZMQ serving pairs by @ai-jz in #3043
perf(ci): reuse prepared vLLM and SGLang environments by @ai-jz in #3047
ci(e2e): the PD lane's job budget sits above its setup and test step budgets by @hello-alexmcc in #3100
ci(e2e): drop the MMMU, PD MMLU and duplicate MMLU accuracy tests by @key4ng in #3009
fix(vllm): the Rust servicer's launcher leaves the multimodal encoder dtype to the servicer by @hello-alexmcc in #3103
ci(test): the finish gate fails on a job cancelled by its timeout, not only on a failed one by @hello-alexmcc in #3102
fix(media): a video fetched by URL gets its own budget, and a fetch that runs out of time names its URL by @hello-alexmcc in #3101
fix(gateway): the TLS crypto provider is installed at start-up, so the mesh listener binds with mTLS on by @hello-alexmcc in #3096
test(macos): fix platform assumptions in native tests by @Moersity in #3087
fix(data-connector): the persistent history backends resolve an item_reference by output item id by @hello-alexmcc in #3086
fix(protocols): Messages API request shapes the public API refuses by @hello-alexmcc in #3075
fix(multimodal): a GLM 5 video is sampled, budgeted and stamped the way the engine's own server does it by @hello-alexmcc in #3106
fix(multimodal): the GLM-5.3 spec takes the 64 images and two videos the vendor's API serves by @hello-alexmcc in #3104
fix(tokenizer/gateway): a template without a thinking switch keeps its reasoning parser armed; clear_thinking is not the switch by @hello-alexmcc in #3027
fix(protocols): Chat Completions request shapes the public API refuses by @hello-alexmcc in #3058
fix(gateway): count_tokens on every worker type, GET /v1/models/{id}, and the models list in the caller's API shape by @hello-alexmcc in #3071
ci(fixtures): the bellwether pin moves to the re-recorded render sets of fifteen groups, and the two harness lists follow by @hello-alexmcc in #3112
ci(e2e): the TokenSpeed lanes pin the scheduler package to the nightly's lower bound by @hello-alexmcc in #3113
test(symphony): the fan-out's panic test waits for the stop flag rather than 200 ms by @hello-alexmcc in #3118
fix(servicer): the Python servicers push the load record on their KV-event stream by @hello-alexmcc in #3119
fix(servicer): the SGLang and TokenSpeed start-up deadlines follow the scheduler's liveness by @hello-alexmcc in #3117
fix(gateway): max_tokens: 0 on the Messages API answers an empty message, as the public API does by @hello-alexmcc in #3114
fix(e2e): the harness reports a router that exits at launch and re-checks its reserved ports by @hello-alexmcc in #3110
fix(gateway): a request's unfetchable media is answered for the media, not charged to the worker by @hello-alexmcc in #3108
ci(test): the finish gate fails when detect-changes failed or was cancelled by @hello-alexmcc in #3107
fix(liveness): the prefill-rate estimate behind the wedge bound counts busy time, not idle gaps by @hello-alexmcc in #3099
ci(e2e): the Rust-servicer lane runs the vLLM chat suite too by @hello-alexmcc in #3105
test(fixtures): the harnesses print one line per model and keep the per-case report as a file the job uploads by @hello-alexmcc in #3116
feat(grpc): the servicers advertise where the engine starts a grammar, and a forced tool call takes the shape that engine needs by @hello-alexmcc in #3044
feat(discovery): file provider reading a JSON worker manifest by @XinyueZhang369 in #2897
ci(e2e): the Rust-servicer lane runs the TokenSpeed chat suite too by @hello-alexmcc in #3115
fix(models): GET /v1/models/{id} refuses an unknown id in the caller's error envelope by @hello-alexmcc in #3129
fix(grpc): a retried forced tool call takes the shape of the worker that gets the retry by @hello-alexmcc in #3131
fix(gateway): a stream the engine ends mid-way is a stream_error, counted and charged to the worker by @hello-alexmcc in #3128
test(servicer): the TokenSpeed launcher tests fake the smg package first by @hello-alexmcc in #3121
fix(metrics): the PD time to first token splits into prefill, relay and decode phases that add up by @hello-alexmcc in #3127
fix(liveness): the prefill-rate estimate learns the tokens the engine computed, not the prompt's full length by @hello-alexmcc in #3130
fix(gateway): the max_tokens: 0 answer checks the window, counts as a request, and picks a gRPC router by @hello-alexmcc in #3124
build(deps): one rustls backend in every binary: the HTTP client takes the process provider instead of pulling in aws-lc-rs by @hello-alexmcc in #3098
fix(conversations): the public field shapes of the conversation objects (empty metadata, listed items, string content) by @hello-alexmcc in #3125
fix(mock-worker): a clean exit on SIGTERM, with the requests in flight drained by @hello-alexmcc in #3120
feat(responses): background mode answers a queued response and runs the request behind its id by @hello-alexmcc in #3095
fix: restore CUDA runner compatibility and require Mistral tool calls by @ai-jz in #3111
test(mock-worker): the SIGTERM test builds its HTTP client through a provider-installing helper by @hello-alexmcc in #3142
perf(gateway): jemalloc keeps 16 arenas instead of four per CPU by @hello-alexmcc in #3141
fix(gateway): a 429 the worker answers is counted as a rate_limit_error, not an internal error by @hello-alexmcc in #3140
perf(grpc): every engine message leaves the chat stream as its own frame again by @hello-alexmcc in #3138
fix(responses): preserve cached usage in gRPC terminal events by @ai-jz in #3137
fix(conversations): list the items of one add call in the order they were added by @hello-alexmcc in #3136
perf(grpc): drop the per-token hops of the chat stream that serve nothing by @hello-alexmcc in #3132
fix(servicer): the SGLang and TokenSpeed Rust launchers start the scheduler ranks from the servicer process by @hello-alexmcc in #3123
fix(servicer): a request a frozen engine does not accept fails alone and the ZMQ client stays up by @hello-alexmcc in #3109
fix(protocols): a chat request that sets both max_tokens and max_completion_tokens is accepted; max_completion_tokens wins by @hello-alexmcc in #3146
fix(gateway): a decode leg the engine fails is an engine_error of the decode worker, a lost prefill leg a failure of the prefill worker by @hello-alexmcc in #3133
fix(gateway): the SSE re-chunker's idle flush keeps a remainder shorter than the tiny band by @hello-alexmcc in #3147
test(symphony): the Qwen3 table reads Qwen3-Omni, and its two recorded groups replay by @slin1237 in #3151
test(responses): the terminal-event test passes the response id the stream takes by @hello-alexmcc in #3152
feat(symphony): the GLM 4.5 and 4.6 table, Ling's newlines under GLM's turn opener, and GLM-4.6 replays by @slin1237 in #3153
feat(symphony): the K2-Horizon table, GLM 4.6's keyed spelling under ifm tags in a block, and its recorded set replays by @slin1237 in #3160
feat(symphony): the MiniMax M2 table, invoke blocks whose parameters name their key in an attribute, and its two recorded groups replay by @slin1237 in #3163
test(mock-worker): the zmq drain test abandons its request once the stop is in effect, and drains a request queued behind it by @hello-alexmcc in #3150
docs(symphony): seven groups whose whole recorded sets replayed with no difference are ready by @slin1237 in #3162
feat(symphony): the grammars for pythonic calls, xLAM's list and MiniMax M3's XML calls by @slin1237 in #3165
docs(symphony): the formats module says each table once, and the probe comment names MiniMax M2 by @slin1237 in #3164
docs: fix the DeepWiki badge image by @ai-jz in #3170
fix(grpc): preserve ByteFallback text at stream completion by @ai-jz in #3144
feat(grpc): support Gemma 4 tools, reasoning and audio references by @ai-jz in #3143
fix(servicer): the SGLang Python servicer ends a request's trace on every exit of its handler by @hello-alexmcc in #3171
ci: skip GPU tests for Markdown-only changes outside tests and fixtures by @ai-jz in #3174
ci: automate prepared vLLM and SGLang runner image builds by @ai-jz in #3049
test(gateway): the ByteFallback Messages specs carry the reasoning_in_prefill field by @hello-alexmcc in #3178
fix(conversations): a reference to an item already in the conversation is refused, and the memory store keeps one link per item by @hello-alexmcc in #3155
fix(tracing): the Router's trace context reaches the SGLang servicers' tracer by @hello-alexmcc in #3126
fix(gateway): pair a multimodal request's decode leg with a vision-capable worker when the model has one by @hello-alexmcc in #3159
fix(gateway): refuse image input to a model whose workers run language-model-only by @hello-alexmcc in #3051
fix(gateway): a no_available_workers refusal writes one rate-limited WARN line, not three at ERROR by @hello-alexmcc in #3148
fix(symphony): the pythonic grammar writes each dialect's values and quotes a referenced enum; xLAM's prose and M3's trigger follow their tables by @slin1237 in #3187
fix(tokenizer): a decode that no longer starts with the streamed text streams the remainder instead of failing the request by @hello-alexmcc in #3176
fix(gateway): PD admission hands a freed decode room to the request that has waited longest by @hello-alexmcc in #3168
fix(router): a stream the worker cuts mid-way ends with the typed error event and is counted, as on the gRPC path by @hello-alexmcc in #3183
fix(gateway): probe a prefill leg's bootstrap port at registration and count the rendezvous failures the engines report by @hello-alexmcc in #3158
fix(protocols): a body the deserializer refuses answers the public API's error code by @hello-alexmcc in #3167
perf(tokenizer): a ByteFallback run streams each complete character as it closes instead of waiting for a regular token by @hello-alexmcc in #3177
feat(symphony): the Inkling table, a turn of control-token messages with JSON calls, and its recorded set replays by @slin1237 in #3166
fix(chat): the logprobs object carries refusal and the message carries annotations by @hello-alexmcc in #3172
fix(servicer): the load records start at the subscription, so an engine frozen on its first request is judged too by @hello-alexmcc in #3182
fix(gateway): the streamed usage of a request is the usage of its non-streamed answer by @hello-alexmcc in #3175
docs(symphony): twenty-one more groups whose whole recorded sets replayed clean are ready, and three rows that were never replayed are by @slin1237 in #3193
feat(symphony): the MiniCPM5 table, attributed calls with no block and CDATA values, and its recorded set replays by @slin1237 in #3197
fix(gateway): the error_code metric label of a request the JSON extractor refused by @hello-alexmcc in #3185
feat(symphony): the dots3 table, attributed calls in a block each with every value on its own line, and its recorded set replays by @slin1237 in #3200
feat(symphony): the attributed grammar writes MiniCPM5's calls without a block and dots3's block per call with framed values by @slin1237 in #3201
test(symphony): laguna-xs.2 reads with the GLM 4.5 table under its own turn opener, and its recorded set replays by @slin1237 in #3202
feat(symphony): the Gemma 4 table and its call syntax, read as JSON while it arrives, and its recorded set replays by @slin1237 in #3204
fix(config): --runtime-worker-threads 0 is a configuration error, not a runtime panic by @hello-alexmcc in #3189
fix(config): an unusable --priority-scheduler-config refuses to start instead of falling back to legacy admission by @hello-alexmcc in #3196
fix(protocols): the MiniMax profile keeps the root role, which the chat template renders in its system slot by @hello-alexmcc in #3161
test(protocols): the rejection-code test expects invalid_json for a syntax error by @hello-alexmcc in #3209
perf(tokenizer): the stop decoder's common path returns the decoded text in one call again by @hello-alexmcc in #3206
fix(grpc-client): asking for load records before the first batch is the caller's choice by @hello-alexmcc in #3203
fix(gateway): a worker promoted by its health probes puts its open circuit breaker on trial instead of waiting out the timeout by @hello-alexmcc in #3192
test(gateway): the metric-label test expects invalid_json for a truncated body by @hello-alexmcc in #3217
feat(policies): power_of_two can compare candidates by in-flight requests by @AMD-yanfeiwang in #3092
fix(messages): tool_use content blocks carry the caller the public API reports by @hello-alexmcc in #3169
fix(config): skip the power_of_two worker-count check under --dp-aware by @AMD-yanfeiwang in #3091
fix(responses): background mode accepts store: false and keeps the response for its polling window by @hello-alexmcc in #3179
fix(protocols): top_p, stop-sequence count and empty messages name the bound crossed by @hello-alexmcc in #3180
fix(gateway): the JSON envelope for a declared over-limit Content-Length on the OpenAI routes by @hello-alexmcc in #3188
fix(data-connector): the Redis conversation delete reads the two counts its pipeline answers by @hello-alexmcc in #3198
feat(symphony): the structural tag for Gemma 4's call syntax, in the template's spelling with the keys sorted by @slin1237 in #3207
fix(router): a decode stream cut mid-way through the prefill/decode HTTP relay ends with the typed error event and is counted by @hello-alexmcc in #3210
fix(gateway): /generate reports a local stop found mid-run; the raw decode step does not count a recovery by @hello-alexmcc in #3216
fix(gateway): PD rendezvous failures in their own metric family, counted for the unary decode leg too, probed once the leg is registered by @hello-alexmcc in #3219
fix(golang): the first chunk of a choice carries the role with its text instead of a role-only delta by @hello-alexmcc in #3220
fix(grpc): end a Messages stream's turn on the engine's stop token by @yechank-nvidia in #3233
fix(responses): the terminal write of a background response is retried while the store is unreachable by @hello-alexmcc in #3195
test(config): a start-up test for --runtime-worker-threads 0, and the flag's help names the minimum by @hello-alexmcc in #3215
fix(gateway): header-read and idle keep-alive timeouts for the HTTP edge by @hello-alexmcc in #3211
fix(data-connector): the Postgres schema bootstrap runs once under the schema lock and a fresh database starts at the latest version by @hello-alexmcc in #3226
fix(servicer): the SGLang servicer advertises no vision until it carries media by @hello-alexmcc in #3229
test(responses): the two completed-response test helpers of background.rs get distinct names by @hello-alexmcc in #3257
fix(helm): the router's readiness probe reads /readiness, and the pod's termination grace covers the gateway's drain by @hello-alexmcc in #3250
perf(data-connector): the Postgres store's statements are prepared once per connection by @hello-alexmcc in #3225
fix(helm): run the router container as the image's non-root user with capabilities dropped and the default seccomp profile by @hello-alexmcc in #3255
fix(gateway): a served model whose tokenizer is still loading is answered 503 tokenizer_not_ready with a Retry-After, not a 500 by @hello-alexmcc in #3280
fix(responses): a response the client expects stored is answered as stored or not at all by @hello-alexmcc in #3278
feat(discovery): replace a worker whose configuration fingerprint changed by @XinyueZhang369 in #3135
fix(pd): a prefill leg that produces nothing is given up on within its liveness bound by @hello-alexmcc in #3149
test(data-connector): the repeatable-read case drops its scratch database before it panics by @hello-alexmcc in #3267
fix(docker): the gateway image ships libjpeg-turbo, so JPEGs decode as the engine's processor decodes them by @hello-alexmcc in #3271
fix(data-connector): a database whose schema version is ahead of the build is refused at start-up by @hello-alexmcc in #3224
fix(gateway): keep an image request's prefill leg on a prefill worker with a vision encoder by @hello-alexmcc in #3173
fix(gateway): a client that closes the connection is a 499 in the request metrics, not a 200 or nothing by @hello-alexmcc in #3199
fix(multimodal): the media fetch policy holds on every hop, and private media origins are refused by default by @hello-alexmcc in #3218
refactor(router): the event that ends a cut SSE body is framed by one helper for both relays by @hello-alexmcc in #3238
fix(config): a malformed control-plane API key entry or a half JWT configuration refuses to start instead of leaving the control plane open by @hello-alexmcc in #3240
perf(tokenizer): a batch of prompts takes the direct byte-level encoder input by input by @hello-alexmcc in #3245
fix(servicer): a cancelled n>1 stream aborts the sub-requests it leaves running by @hello-alexmcc in #3248
refactor(protocols): one copy of the validation-error helper, and a sentence-code example that still exists by @hello-alexmcc in #3249
fix(servicer): the launchers validate their knobs before the gRPC bind and the Python servicers exit when the engine dies by @hello-alexmcc in #3253
fix(cli): a flag's companion is named when it is absent, and a malformed value is refused by name by @hello-alexmcc in #3270
test(config): the start-up test keeps the bind and the health signals apart by @hello-alexmcc in #3251
fix(config): a health-check knob set to 0 is a configuration error, not a busy loop by @hello-alexmcc in #3254
perf(tokenizer): a ByteFallback run that cannot become valid streams its replacement characters as they come, with a bounded decode window by @hello-alexmcc in #3258
fix(scheduler): a small pool is shared by every class, its capacity counts every activated worker, and an abandoned queue wait is counted by @hello-alexmcc in #3260
ci(bench): the tokenizer and request-processing benchmarks link the gateway's global allocator by @hello-alexmcc in #3261
fix(golang): the FFI library runs jemalloc with the gateway's options by @hello-alexmcc in #3262
fix(config): equal listener ports are a configuration error, and nothing binds before a late configuration error by @hello-alexmcc in #3265
fix(gateway): an empty model inventory answers the JSON error envelope by @hello-alexmcc in #3276
fix(protocols): the deserialization error code is read at the start of serde's reason, and the e2e helper tells the parser's 400 by its prefix by @hello-alexmcc in #3232
feat(symphony): the registry picks a model id's table, and the prompt's replay is the table's to cut by @slin1237 in #3282
fix(symphony): the Gemma tag stays within its bounds for a union chain, a wide DAG and heavy definitions, and a type list's branches are the known types once each by @slin1237 in #3283
feat(symphony): Inkling's structural tag, triggered by its call marker as the gateway's is by @slin1237 in #3212
fix(grpc): non-stream chat logprobs cover the message content only by @yechank-nvidia in #3156
fix(install): the engine images' gateway gets the TurboJPEG runtime library, so JPEGs decode as the engine's processor decodes them by @hello-alexmcc in #3284
fix(gateway): a conversation item id over 64 characters is refused before any store is asked by @hello-alexmcc in #3228
fix(gateway): a worker's status before the first token counts in smg_worker_errors_total by @hello-alexmcc in #3205
fix(responses): unstored background records are swept past their window by the stores that outlive the gateway, and are no previous response by @hello-alexmcc in #3275
fix(responses): a background response deleted while its run is in flight stays deleted by @hello-alexmcc in #3273
fix(cli): the documented -1 sentinels parse in the space-separated spelling, and an unsigned flag refuses -1 by name by @hello-alexmcc in #3274
fix(router): a streamed head's success is recorded before the relay task can sample a cut by @hello-alexmcc in #3247
fix(protocols): accept up to 20 logprobs on /v1/completions, as chat's top_logprobs by @yechank-nvidia in #3292
fix(grpc): render no tools for tool_choice none by @yechank-nvidia in #3291
feat(gateway): a keep-alive on every open SSE stream while the engine is silent by @hello-alexmcc in #3241
fix(ci): provide defaults for nightly GPU runner image builds by @ai-jz in #3264
docs(gateway): the idle keep-alive timeout's help says to keep it above the fronting proxy's idle timeout by @hello-alexmcc in #3287
fix(gateway): the streamed usage of a Harmony chat completion is the usage of its non-streamed answer by @hello-alexmcc in #3214
fix(config): a --log-dir the gateway cannot use is a configuration error, not a panic by @hello-alexmcc in #3263
fix(tokenizer): a streamed literal U+FFFD counts as its three bytes when a ByteFallback run collapses by @hello-alexmcc in #3281
fix(symphony): a JSON schema leaf carries only the definitions its pointers reach, without annotations by @slin1237 in #3290
fix(multimodal): the Qwen families resize with the torchvision kernel their reference processors run by @hello-alexmcc in #3243
fix(gateway): the streamed usage of a Harmony Responses answer is the usage of its non-streamed answer by @hello-alexmcc in #3246
fix(data-connector): a configurable bound on every history store call, answered 503 when exceeded by @hello-alexmcc in #3239
fix(gateway): a leg stream that fails mid-way during a non-streamed disaggregated answer counts against its worker by @hello-alexmcc in #3227
fix(gateway): judge a prefill/decode leg's veto fallback and verdict over the workers the request may use by @hello-alexmcc in #3194
test(startup): one launcher shared by the start-up tests, and the runtime's size reaches the log by @hello-alexmcc in #3295
fix(pd): usage.cached_tokens is the prefill leg's prefix-cache hit, not the decode leg's KV transfer by @hello-alexmcc in #3286
fix(history): stored responses and conversations are scoped to the tenant that created them by @hello-alexmcc in #3222
ci(codeowners): smg-project-bot owns every path, so its review counts as a code owner review by @slin1237 in #3299
fix(policy): cache_aware's spill gate sizes its absolute margin to the pool's load, so small pools balance too by @hello-alexmcc in #3294
test(symphony): every table's structural tag is written on request, and a script compiles them with xgrammar by @slin1237 in #3289
fix(gateway): keep image and video requests off workers without a vision encoder in a mixed pool by @hello-alexmcc in #3122
fix(router): the PD decode relay records the head's success before it can sample a cut by @hello-alexmcc in #3296
fix(multimodal): a video batch of the Qwen3-VL family is sliced by its own sizes tensor by @hello-alexmcc in #3231
fix(gateway): worker-mode media processing applies per-item hints on the router instead of refusing them by @hello-alexmcc in #3298
fix(multimodal): pixel bytes normalize with the reference processors' fused arithmetic by @hello-alexmcc in #3242
fix(multimodal): CMYK and YCCK JPEGs decode through libjpeg-turbo and convert to RGB the way PIL does by @hello-alexmcc in #3285
fix(test): the overload-steering test registers its workers with the helper the module has by @hello-alexmcc in #3302
build(deps): quinn-proto 0.11.17 -> 0.11.18 in the lockfile (RUSTSEC-2026-0338, -0340, -0343) by @hello-alexmcc in #3301
fix(data-connector): a second link of a conversation item moves it on every backend by @hello-alexmcc in #3223
fix(data-connector): the start-up contact with a Postgres or Redis store is bounded and a verdict by @hello-alexmcc in #3252
fix(history): a store that is not available is answered 503 storage_unavailable on every stateful endpoint by @hello-alexmcc in #3268
fix(config): a NaN or infinite threshold, a share outside 0.0-1.0 or a zero-sized budget is a configuration error at start-up by @hello-alexmcc in #3279
fix(gateway): a served model whose tokenizer load was skipped or failed for good is answered 503 tokenizer_load_failed without a Retry-After, not the not-ready 503 forever by @hello-alexmcc in #3288
fix(multimodal): apply an image's EXIF orientation when decoding, as the engine's own loader does by @hello-alexmcc in #3213
build(deps): rustls 0.23.44 -> 0.23.45 in the lockfile (RUSTSEC-2026-0285) by @hello-alexmcc in #3300
build(deps): wasmtime and wasmtime-wasi 45.0.3 -> 49.0.2 (RUSTSEC-2026-0327 and seven more) by @hello-alexmcc in #3303
fix(config): a tokenizer, model or chat-template path the gateway cannot read is a configuration error by @hello-alexmcc in #3266
fix(gateway): the access log carries the request path, not the query string by @hello-alexmcc in #3306
perf(multimodal): pixel bytes convert to the wire dtype per channel block and on the vector unit by @hello-alexmcc in #3307
fix(history): the memory history backend is bounded by record counts and a retention by @hello-alexmcc in #3244
test(gateway): the usage parity tests drive an engine-reported reasoning count on both shapes by @hello-alexmcc in #3221
fix(data-connector): pass the memory bounds in the start-up timeout test helper by @key4ng in #3317
fix(servicer): a data URL the worker's fetcher does not take is the caller's error, not a retry by @hello-alexmcc in #3230
ci: cargo audit of the lockfile on dependency changes and nightly by @hello-alexmcc in #3304
build: pin the Rust toolchain and the container base images by @hello-alexmcc in #3305
fix(gateway): the worker admin endpoints refuse a bad body in their JSON envelope and validate the labels and fields the routers rely on by @hello-alexmcc in #3277
fix(gateway): the request metrics' model label is bounded to the served models by @hello-alexmcc in #3259
fix(gateway): the proxy routers' refusals and the model_not_found 404 write one rate-limited WARN line, not ERROR lines per request by @hello-alexmcc in #3157
fix(gateway): warn at start-up when a TLS private key file is readable by others by @hello-alexmcc in #3310
build(deps): bump thiserror from 2.0.20 to 2.0.21 by @dependabot[bot] in #3334
build(deps): bump wasm-encoder from 0.257.1 to 0.261.0 by @dependabot[bot] in #3330
perf(multimodal): accelerate rgb long-side cap resizing by @Moersity in #3269
perf(multimodal): the torchvision uint8 bicubic resize runs its passes in 16-bit taps and 32-bit sums on the vector unit by @hello-alexmcc in #3311
build(deps): bump tiktoken-rs from 0.12.0 to 0.12.1 by @dependabot[bot] in #3331
build(deps): bump indexmap from 2.14.0 to 2.14.2 by @dependabot[bot] in #3332
build(deps): bump uuid from 1.26.0 to 1.28.0 by @dependabot[bot] in #3333
fix(gateway): a refusal the gateway answers on its own is counted as unavailable, not as a backend error by @hello-alexmcc in #3181
fix(multimodal): the Qwen3-VL family's video prompt carries the reference processor's layout and frame stamps by @hello-alexmcc in #3272
feat(discovery): the file provider reads a manifest change as it happens by @XinyueZhang369 in #3318
fix(gateway): a mesh listener that cannot bind is a start-up error, not a node serving outside the mesh by @hello-alexmcc in #3328
fix(data-connector): a cached statement that went stale is prepared afresh and the call made again by @hello-alexmcc in #3327
perf(data-connector): a conversation's metadata update is one statement that returns the row by @hello-alexmcc in #3325
perf(multimodal): the CMYK to RGB conversion runs on the vector unit, in place as before by @hello-alexmcc in #3342
fix(config): the shared validation holds the stores' bounds and leaves the connection string to the driver; -1 is the one spelling of persistent Redis records by @hello-alexmcc in #3312
fix(data-connector): the unstored-record sweep index survives an object with a NUL escape, and the sweep reaches the store through the call bound by @hello-alexmcc in #3309
fix(gateway): an inbound request id is honoured only up to 128 visible-ASCII characters by @hello-alexmcc in #3308
perf(multimodal): WebP files decode through libwebp when it is present by @hello-alexmcc in #3313
fix(config): the priority scheduler refuses an unknown YAML key and an unknown default max class name at start-up by @hello-alexmcc in #3326
fix(metrics): a response body that fails after its head keeps its sample, told apart by error_code stream_error by @hello-alexmcc in #3335
fix(gateway): a prefill/decode leg start the worker's side cancelled is a backend failure to retry, not a 400 by @hello-alexmcc in #3338
perf(multimodal): the pixel-byte patch layout splits each source row into its channel runs with vector loads by @hello-alexmcc in #3316
fix(gateway): probe the default bootstrap port of a prefill leg registered without one by @hello-alexmcc in #3336
fix(gateway): a mesh server that cannot start is a start-up error, not a node serving outside the mesh by @hello-alexmcc in #3345
fix(gateway): the regular gRPC dispatch books its prompt on the worker before the call is polled by @hello-alexmcc in #3340
fix(multimodal): 16-bit truecolour PNGs narrow to the high byte of each sample, as the engine's own loader does by @hello-alexmcc in #3343
fix(multimodal): the Qwen-VL smart_resize rounds without a floor before the pixel-budget test, as the reference does by @hello-alexmcc in #3347
fix(responses): a background run is not dropped between its own terminal write and the conversation-item linking by @hello-alexmcc in #3349
fix(golang): a streamed tool-call delta keeps the text the parser released with it by @hello-alexmcc in #3324
style(multimodal): format the decoded_rgb fixture generator by @hello-alexmcc in #3353
fix(gateway): no cross-origin allowance without --cors-allowed-origins; '*' keeps the open policy by @hello-alexmcc in #3315
perf(multimodal): an image's orientation tag is applied by tile transposes and row reversals on the vector unit by @hello-alexmcc in #3321
perf(multimodal): the patch layout is written once, in the wire dtype, when every worker of the model reads the same one by @hello-alexmcc in #3329
fix(config): the shape checks of fields on RouterConfig run in the validator, on every construction path by @hello-alexmcc in #3339
fix(servicer): a request the engine left behind at shutdown ends with UNAVAILABLE, not an abort completion by @hello-alexmcc in #3346
feat(multimodal): the dump example and the opt-in test for bellwether's multimodal suite by @hello-alexmcc in #3348
fix(data-connector): the earlier sweep index is dropped concurrently, after the bootstrap's commit, so a start blocks no read or write on the responses table by @hello-alexmcc in #3350
fix(data-connector): the NUL replacement's literals are dollar-quoted, so the sweep index, the sweep and the lookup read the same under standard_conforming_strings = off by @hello-alexmcc in #3351
fix(golang): the streaming converter reads a forced tool choice the way the gateway does by @hello-alexmcc in #3352
fix(data-connector): the unstored-record sweep reports the server's message and a pool failure as the store being unavailable, as every other Postgres call does by @hello-alexmcc in #3355
fix(golang): a named tool choice is constrained with the named function's schema by @hello-alexmcc in #3357
fix(golang): the non-streaming chat path assembles a tool call streamed in pieces by @hello-alexmcc in #3359
fix(gateway): an HTTP/2 fault under a worker call is its transport failure, and an abandoned stream is released by the worker's end after the abort, not by a reset by @hello-alexmcc in #3360
perf(multimodal): a clip's frame count, rate and duration are read from the MP4 container when its tables are plain by @hello-alexmcc in #3322
fix(servicer): the SGLang servicer reports the running window the scheduler computed when --max-running-requests is unset by @hello-alexmcc in #3358
perf(tokenizer): add opt-in tokenizers v1 BPE encoding by @Moersity in #3356
fix(multimodal): the dump example reads a wire encoder input by @slin1237 in #3365
feat(multimodal): Nemotron-H Omni image, audio and video by @nperrin-fr in #3323
fix(gateway): a disaggregated pair missing a leg answers its family's 503, not the default router's 404 by @hello-alexmcc in #3361
perf(tokenizer): accelerate opt-in byte-level decoding by @Moersity in #3370
refactor(symphony): DSML is the attributed assembler with its own tags and string-attribute typing by @slin1237 in #3374
fix(gateway): a parallel pair's prefill silence is charged to a decode partner that stalled mid-stream by @hello-alexmcc in #3362
fix(multimodal): transparent images decode under the destination engine's own transparency rule; the vLLM family composites over white by @hello-alexmcc in #3341
refactor(symphony): one CallAssembler trait in place of the engine's enum over the syntaxes by @slin1237 in #3376
refactor(symphony): the three adapters fold a whole event sequence once by @slin1237 in #3377
docs(symphony): a table's module doc says what it reads; the notes move to docs/tables.md by @slin1237 in #3379
refactor(symphony): XTML is the attributed assembler with Kimi's tags, a marker tag end and type-attribute typing by @slin1237 in #3380
docs(symphony): the engine, grammar, adapter and assembler docs keep their lead; the notes move to docs/ by @slin1237 in #3381
fix(servicer): one servicer switch for the three launchers; only a flag decision reaches the environment by @hello-alexmcc in #3382
docs(symphony): the attributed assembler's doc keeps its spellings; the notes move to docs/assemblers.md by @slin1237 in #3384
fix(golang): the streaming converter keeps one tool parser per choice by @hello-alexmcc in #3364
refactor(symphony): the grammar prefix's inner tags take every attributed family by @slin1237 in #3389
fix(servicer): a launch that fails stops the bound Rust server in every launcher by @hello-alexmcc in #3383
fix(gateway): a decode leg earns its worker's breaker success at its first response, not at dispatch by @hello-alexmcc in #3368
refactor(symphony): the Gemma tag's node cap has one fallback by @slin1237 in #3391
refactor(gateway): one stream-failure answer and one SSE frame writer on the response paths by @hello-alexmcc in #3386
refactor(servicer): the vLLM engine config is read through one view by @hello-alexmcc in #3400
refactor(servicer): one lifecycle for the three Rust-servicer launchers by @hello-alexmcc in #3397
fix(gateway): a KV-store connector profile for disaggregated pairs by @hello-alexmcc in #3378
fix(gateway): an encode leg alone takes the EPD router, never the PD fallback by @hello-alexmcc in #3372
test(worker): the load-counter microbenchmark asserts no absolute floor unless asked to by @hello-alexmcc in #3390
refactor(gateway): the health loop's schedule decides a worker's probe slot in one place by @hello-alexmcc in #3403
fix: discover SGLang attention-DP routing ranks by @ekzhang in #3411
fix(multimodal): a frame's pixels follow its transparency policy when the policy is set after a decode by @hello-alexmcc in #3402
refactor(config): two validators split into named checks, four dead builder setters removed by @hello-alexmcc in #3392
docs(launcher): the flag mapping documented where it is read by @hello-alexmcc in #3394
fix(responses): every status transition of a background response is one conditional write of the store by @hello-alexmcc in #3387
fix(multimodal): a malformed temporal_patch_size is a configuration error on the video stamp path too by @hello-alexmcc in #3404
refactor(gateway): the router configuration built in named steps by @hello-alexmcc in #3401
refactor(gateway): the start-up sequence as named phases by @hello-alexmcc in #3399
fix(multimodal): a video stamp the tokenizer cannot encode fails the request in every registry spec, through one shared encoder by @hello-alexmcc in #3393
refactor(servicer): the vLLM launcher reads the engine config through the view by @hello-alexmcc in #3408
feat(rl): control endpoints for gRPC and ZMQ workers by @key4ng in #2667
fix(gateway): the transparency policy is resolved over the pipeline's routing pools, as the image wire dtype is by @hello-alexmcc in #3413
fix(golang): the constraint and the response parser read the tools declared on messages too by @hello-alexmcc in #3419
refactor(gateway): one reader of the chat stream's events for the Responses conversions by @hello-alexmcc in #3418
refactor(servicer): the SGLang servicers read the engine config through one view by @hello-alexmcc in #3414
perf(gateway): the completion stream serializes the bytes around a chunk's choice once per stream by @hello-alexmcc in #3395
refactor(gateway): the multimodal settings resolve through one layered rule and one switch spelling by @hello-alexmcc in #3398
perf(tokenizer): the L1 prefix cache finds its boundaries in one pass over the input by @hello-alexmcc in #3385
fix(gateway): a Failed mesh import under --remove-unhealthy-workers keeps its probe slot by @hello-alexmcc in #3423
perf(gateway): the HTTP layer's metrics keep their handles per worker instead of rebuilding the key on every record by @hello-alexmcc in #3417
ci(pre-commit): the bench syllable table's "ure" is no misspelling for codespell by @hello-alexmcc in #3430
refactor(servicer): the TokenSpeed servicers read the engine config through one view by @hello-alexmcc in #3412
refactor(gateway): the worker selection stage's leg table and post-admission refusal written once by @hello-alexmcc in #3407
perf(worker): a token within the stamped millisecond records no new progress by @hello-alexmcc in #3409
perf(tokenizer): streaming decode of sentencepiece-style vocabularies walks a piece table by @hello-alexmcc in #3416
fix(multimodal): one rule sizes an inline data URL for the router's plan check and the worker's item cap by @hello-alexmcc in #3415
refactor(servicer): the readiness probes share one stuck-scheduler rule by @hello-alexmcc in #3431
refactor(gateway): the PD pairing mismatch judged by one rule over named agreements by @hello-alexmcc in #3424
refactor(multimodal): a config or size the preprocessing refuses is the same error for the token count and the audio processor factory by @hello-alexmcc in #3421
refactor(tokenizer): each native renderer states its template answers in one table by @hello-alexmcc in #3425
refactor(servicer): the launchers share one KV-events default rule by @hello-alexmcc in #3436
refactor(multimodal): the tracker queues every media item through one enqueue by @hello-alexmcc in #3433
refactor(tokenizer): one place chooses the encode path and says why the others declined by @hello-alexmcc in #3427
fix(gateway): a sequential PD decode leg owes its first token within a bound, in the shape of the prefill leg's by @hello-alexmcc in #3369
fix(servicer): the SGLang Rust-path handshake reports the running window in the flag's units, on either engine release by @hello-alexmcc in #3366
fix(servicer): the KV-event stream reports a load record it cannot read, once per outage by @hello-alexmcc in #3437
refactor(gateway): the CLI fields grouped under their help headings by @hello-alexmcc in #3440
perf(gateway): the JSON access log is written without re-parsing the span's fields per record and per line by @hello-alexmcc in #3435
refactor(tokenizer): the DeepSeek renderers share their helpers through one module by @hello-alexmcc in #3426
refactor(discovery): the file provider's loop as a type with named steps by @hello-alexmcc in #3439
perf(gateway): a pixel tensor is written once on its way to the worker by @hello-alexmcc in #3429
fix(mesh): dead-owner worker keys are not re-imported after a router rollout by @hello-alexmcc in #3375
refactor(discovery): the Kubernetes provider's role table and a split discovery task by @hello-alexmcc in #3443
refactor(multimodal): a per-spec modality table drives the registry's per-modality methods; Gemma 4 is the first spec on it by @hello-alexmcc in https://github.com/smg-project/smg/pull/3432
refactor(gateway): the router families the weighted selection draws from are one table by @hello-alexmcc in https://github.com/smg-project/smg/pull/3438
fix(servicer): the SGLang servicer subscribes to one KV-event publisher per scheduler on either engine release by @hello-alexmcc in https://github.com/smg-project/smg/pull/3434
refactor(gateway): the KV-event snapshot tail nobody produces goes by @hello-alexmcc in https://github.com/smg-project/smg/pull/3441
feat(gateway): the gRPC chat driver reads with Symphony behind --parser symphony by @slin1237 in https://github.com/smg-project/smg/pull/3190
fix(tokenizer): the L1 prefix cache encodes each segment after the special token that ends its boundary by @hello-alexmcc in https://github.com/smg-project/smg/pull/3428
feat(gateway): explicit parser names choose Symphony's table, or the legacy parsers, per model by @slin1237 in

smg-project/smg v1.13.0 V1.13.0 on GitHub