Shepherd Model Gateway v1.15.0

SMG v1.15.0 makes the Rust engine servicer the default for vLLM, SGLang and TokenSpeed, streams every gRPC worker's load records to the router so that the load-based policies price a worker by what it is doing now, follows a hybrid model's KV cache groups in cache-aware routing, and spends the cycle on the cost of a request: the native tokenizer encodes 28 to 55 % faster per token on the GPT-4-template families and in a quarter to two fifths of the time on the DeepSeek V4 files, with the same ids, and a second series of changes took more off every vocabulary shape late in the cycle; an audio clip's preprocessing fell from 594 ms to about 20 ms; the image processors of the Qwen, Gemma 4, GLM, MiniMax, LLaVA, Phi-3, Pixtral and MoonViT families now produce the reference processors' bytes exactly, several times faster on 64-bit x86 processors; and a long series of small changes on the gateway's own request path (streaming frames packed per poll, one clock read per request, the request's preparation done once, no zero-filled buffers on the image path, the generate stream's text escaped once) each took a measured share off the instructions and allocations per request. The vision-language families gain video at the gateway under their engines' own loader rules, a Step3-VL image spec arrives, a tekken.json vocabulary loads, the control plane reloads its keys in place and answers its effective configuration with every secret redacted, and the Python package gains smg.tokenizer. Start with Before you upgrade and the Highlights; the Upgrade notes list every changed default and refusal.

This release includes 754 commits from 20 contributors, including 8 first-time contributors. Some defaults changed, and some deployments need a step before upgrading: see Before you upgrade.

The per-request figures come from the gateway's cost ledger, a CPU-only bed with a mock engine that counts the gateway's own instructions, allocations and bytes per request before and after each change; the tokenizer and media figures come from the crates' benchmarks on one pinned core. They measure different workloads and should not be combined into one overall speedup. Where a figure names a family or a vocabulary, it holds for that one.

Before you upgrade

• Engine pods start the Rust servicer by default for vLLM, SGLang and TokenSpeed. Roll gateways first, then install smg 1.15.0 together with smg-grpc-servicer 0.16.0. A launch the Rust servicer does not serve is refused by name, nothing falls back; --servicer-impl python (or SMG_<ENGINE>_SERVICER_IMPL=python) keeps the Python servicer, which is deprecated there and logs so whenever it is chosen. TensorRT-LLM keeps the Python servicer as its launcher's default. See Engine side.
• --least-load-max-waiting-requests is retired, with the least_load policy's max_waiting_requests key: a gateway or a policy configuration that still sets it does not start. Use --worker-overload-waiting-requests (per worker, under every policy) and --worker-overload-shed for the refusal. See Configuration that now refuses to start.
• Video loader rules on the engine command line are checked at worker start: a vLLM worker of the Qwen2-VL, Qwen3-VL or GLM-5 line given a launch-flag video rule its loader reads (fps, max_frames, min_frames, temporal_patch_size, video_backend) refuses to start, because the servicer's media pipeline cannot follow it; a loader named through the environment alone keeps letting the family start.
• Per-request media counts are no longer bounded by the gateway's own model specs for the Qwen2-VL, Qwen3-VL, Qwen3-Omni, Qwen3-ASR, Inkling, Muse Spark, Kimi K3, Gemma 4, Nemotron Omni and DeepSeek V4.1 families (none of them bounds the count itself). The bound is the engine's advertised limit, or the deployment's --mm-per-request-image-limit / SMG_IMAGE_MAX_COUNT, SMG_VIDEO_MAX_COUNT and SMG_AUDIO_MAX_COUNT. A deployment whose engine advertises no per-request limit sets those when it wants a count.
• Load records stream to the router from every gRPC worker, under every policy, whether or not KV events are on; a v1.14.0 servicer that does not stream keeps being polled. least_load and power_of_two now price a worker (and each rank of a data-parallel engine) by its current record, so their placements can differ from v1.14.0's under the same traffic.
• Prompts render as the engines render them on a few more paths (DeepSeek V3.2 and V4 developer messages, system-final conversations, the tools block, the reasoning-effort prompt; Kimi K2 tool declarations), so prompt token counts and prefix-cache keys change once for those requests.
• Dashboards and API clients: the in-flight connections gauge heals itself, the generate endpoint counts its prompt and cached tokens, a retried gRPC request is counted once, cross-field validation errors lose their internal field-set prefix, n on chat completions takes up to 128, and a Symphony tool call the parser could not read whole now reads incomplete instead of arriving as a well-formed call. See Upgrade notes.
• Out-of-tree routing policies: the policy trait's completion hook no longer takes a success flag; drop the parameter. See Bindings and Rust crates.
• Helm chart defaults: a gRPC worker's startup probe now asks the servicer's gRPC health service, and the engine's load and warm-up count against the startup budget (360 s by default), so a slower worker needs a larger startupProbe before the upgrade; the chart requires the native JPEG decoder (router.mmTurbojpeg: require), so a custom router image without libjpeg-turbo sets prefer; the chart refuses the regular Slurm job names and router.workerUrls under a disaggregation mode. See Changed defaults.
• Launcher and Python Router defaults: the launcher's --eviction-interval-secs defaults to 120 s, the CLI's value; smg_rs.Router(...) starts with the CLI's defaults (cache-aware routing, port 30000, a 1800 s wait for the first worker); a ZMQ worker is no longer pinned to the SGLang wire without --backend; an encode policy outside random, round_robin and consistent_hashing, and a discovery selector entry without key=value, are refused by name where they used to be dropped without a word.
• Rust library users: llm-tokenizer steps to 2.0.0: the three DeepSeek renderers raise one DsEncodingError type, re-exported under each renderer's path with every message unchanged, so a match over one renderer's error now sees the other dialects' variants on the same type. See Bindings and Rust crates.
• gpt-oss on the gRPC Harmony path puts a leading system or developer message into the developer message's instructions and writes no empty developer message, as vLLM renders it; a GLM-5.3 request's minimal, medium and xhigh reasoning efforts are read as low, high and max, as the provider reads them. Prompt token counts and prefix-cache keys change once for those requests.

Highlights

The Rust servicer is the default on every engine but TensorRT-LLM

The Rust gRPC servicer serves vLLM, SGLang and TokenSpeed by default; the Python servicer stays selectable, is deprecated on those engines and goes two minor releases after this one. The cycle closed the gaps a default needs: the SGLang and TokenSpeed servicers measure their prefill capacity and report queued token-work, hit rate and generation throughput like the vLLM servicer, probe the scheduler on HealthCheck and report a frozen one within 20 s, answer GetLoads with each rank's utilization and the aggregate's throughput instead of zeros, answer Abort for an unknown id as their Python counterparts do, and an encode worker pushes no zero-filled load record. The vLLM servicer's GetLoads answers only the rank a request names, advertises the engine's audio limit beside the image and video ones, and the measured prefill capacity of an engine without scheduling timestamps counts every request of an overlapping batch. The TensorRT-LLM servicer declares GetLoads and reports token-work, throughput and hit rate from the requests it forwards. The readiness hold admits only the warm-up requests of its own launcher process (a per-process nonce), a prefill side of a disaggregated pair warms up with one stand-in request, and the planned warm-up no longer runs the decode staircase where nothing decodes. vLLM encode/prefill/decode deployments route over gRPC with encode-side preprocessing and metadata-only prefill and decode inputs, and the SGLang worker takes mixed image and video references.

Load streaming to the router, and routing priced by measured work

Every gRPC worker streams its load records to the gateway: SubscribeKvEvents gains a loads_only request that a servicer answers for every data-parallel rank at the record's cadence, without KV events and without needing them enabled; older servicers ignore the field and keep being polled. least_load and power_of_two price a worker, and each rank of a data-parallel engine, by its current record, and prefill routing can compare workers by their measured input-token capacity and estimated drain time with the configured figure as the fallback. On the gateway, a pushed record updates its worker's report in place and its engine gauges through handles kept per worker and rank, so a gateway behind engines that push records allocates nothing per record; a least-load, power-of-two or cache-aware decision reads each candidate's report once instead of about six times and allocates nothing; the power-of-two, random and passthrough policies pick without listing the pool; and a consistent-hashing or prefix-hash placement no longer builds a map of the pool to translate the ring's answer. power_of_two can also place by the context a worker holds: --power-of-two-load-metric resident_tokens:<request_tokens> scores a worker's resident context tokens plus the request's, for engines that report resident tokens. A disaggregated request is no longer parked on a decode worker whose running window is full while a compatible sibling has room, so the load-scoring decode policies no longer herd under the admission gate; workers the routing vetoes exclude no longer count in the overload pool checks; from eight candidates up the least-load policy scores under a read guard instead of the dispatch tally's write lock, so concurrent decisions no longer wait on one another, and over a small pool it reads each candidate's tally once; a routing decision reads each worker's availability from its published state code, which halves the per-worker cost of a pool scan; and a keyed lookup on the consistent-hashing ring allocates nothing.

Cache-aware routing follows a hybrid model's cache groups, on a leaner index

Cache-aware routing over KV events follows a hybrid model's sliding-window and state cache groups, predicting what the engine will reuse instead of counting every block as full attention: the servicer forwards the cache-group mask of the events from an engine that reports each boundary once for every group, the TokenSpeed servicer keeps KV-event publishing on for a hybrid model when its engine publishes per cache group, and a Rust servicer no longer treats its engine as dead while the engine publishes KV events through a long prefill (the engine-silence bound counts from the engine's last sign of progress, an output or a KV-event batch its relay read live). The index behind it got cheaper: the string prefix index compares text eight bytes at a time when it looks up or stores a prompt, the token tree compares prefixes a page at a time on its insert, match and fused routing walks, a dead child run leaves its parent's table by a probe for its key instead of a scan of every slot, and the hash arrays take 4-word size classes below 32 words, so a chain of a dozen blocks grown through its decode costs 23 bytes a block in the index instead of 29, and the string index's prefix match counts the input's characters in the walk itself instead of a second pass over the text (about 47 % fewer instructions on a 4,096-character match), and its multi-byte fix-up steps back to the character boundary instead of re-walking the match. A store that would take one run's child table or hash array past an arena chunk is declined and counted (smg_kv_index_cap_refusals) instead of panicking in the event lane, and the arena's held bytes are exported beside the bytes it handed out (smg_kv_index_arena_chunk_bytes). A hybrid model's cache-group views are kept in a table by worker id, so a cache-aware decision over a fleet of hybrid workers costs about a fifth less per holder, and a worker whose group view is restarted over more than one main-signal group is counted and logged as a late view instead of being read as current.

A leaner request path

The streams are written by the response body itself instead of a task of their own, and the chat, Messages, completions and generate streams pack the engine events that are ready at one poll into one frame, so a streamed request costs fewer writes and context switches with the bytes, their order and the first token's timing unchanged. A chat stream's per-token chunk is written in one pass (8.6 % fewer instructions per 128-delta chat stream request on the cost ledger), a Messages stream's content_block_delta frames likewise (11.7 % fewer), and the chunk serialization appends into its buffer instead of going through a writer adapter (8.3 % fewer on the chat stream). A request's provider profile, parser names, table decision and the prompt's thought-state replay are derived once, in preparation, and every later stage reads the carried answer (between 0.3 and 1.2 % fewer instructions per chat request each, every decision unchanged); a request without media no longer looks its model up in the tokenizer and multimodal registries (3 to 4 % fewer instructions and about 19 fewer allocations per chat and Messages request). The gateway reads the clock once per request at its edge and shares the stamp with its metrics, access log, admission and timeout layers; a request's end no longer wakes the connection task already polling it; the HTTP edge runs its request-id, metrics and access-log layers without a boxed future per layer; the per-request entry stamp, the metrics path label, the request id and the tenant key are built once instead of copied. A prompt of up to sixteen kibibytes of rendered text is tokenized on the thread that holds the request, and the blocking pool takes only the encodes that outweigh the hand-off. The generate future each dispatch boxes per request fell from 9,632 to 4,784 bytes over two changes, and a Messages request without an output config or format is back in its earlier allocator size classes. Logprobs are read in place from the engine's frame, the top-logprob candidates are cut to the requested count before decoding, and the chat and completion streams keep one state entry per choice instead of a hash map per field. Later in the cycle: a collected (non-streaming) answer on the gRPC router reads the clock once, at its dispatch, instead of twice; the chat, Messages and completion stream readers borrow the engine message instead of taking it by value, which removes one copy of the message per streamed token; the generate stream escapes each new piece of text once instead of the whole answer so far on every event, so a long or slowly paced generate stream costs far fewer instructions per event with every byte unchanged; the per-request family draw reads a pool's size by borrowing the routing snapshot instead of cloning and dropping its Arc; the Symphony reader sizes its per-step and held buffers before they grow, and reserves its step vectors only for a step of more than one token, so a one-token streaming step pays one length compare instead of two reserves; the per-request metric records hash the gateway's own label keys with FxHash while keys a client controls stay on SipHash; a tenant reservation on the in-memory rate limiter allocates once instead of three or four times; and on the engine wire an add request is encoded into one buffer sized from its prompt, a decoded output reads its request id from the frame, and a finished output's id is shared with the step's finished list instead of copied into it. The serving authentication layer releases its key-map guard before the inner service runs, so many streams in flight on one thread no longer slow every later key lookup.

The tokenizer encodes faster on every family, with the same ids

The direct byte-level encode path pre-tokenizes the GPT-4 family's template (Qwen 2 and 3, Llama 3, GLM 4.6 and their kin) with one hand-written scan instead of a backtracking regex program, and the DeepSeek V4 family's three-stage chain with one scan instead of three: warm encode time per token falls by 28 to 55 % on prose, markdown, code, JSON and chat prompts of a 151k and a 128k byte-level vocabulary, and the DeepSeek V4.1 file encodes in 0.23 to 0.41 of its time per token. Sentencepiece-style vocabularies with byte fallback (the Gemma and Llama 2 families) and tiktoken vocabularies whose ranks and pattern fit it now encode on the direct path too, several times faster than through the generic pipeline, and a tokenizer.json with a TemplateProcessing BOS/EOS frame or text outside NFC no longer falls back to the tokenizers pipeline. Text the process has not seen before merges faster (one cache line per pair instead of three, merges on arrays of the word's own size, pieces merged from characters in multi-byte scripts, a thread's merge table read into the caches once at its first encode), the long piece tier keeps pieces up to 16 KiB so a repeated text in a script without word spaces no longer merges its long pieces on every encode (a 262k-token byte-fallback vocabulary reads 15.2 ns per token at 2k and 30.6 at 32k where it read 26.4 and 50.5), the pre-tokenizer runs as a scanner for every template shape in use, and the NFC quick check reads one bit per plain character (5 to 6 % faster encodes at 256 B and 7 to 9 % at 2 KB under the three vocabularies with the NFC normalizer). A second series of changes late in the cycle took more off every vocabulary shape, with token ids unchanged: the single-stage byte-level encoders probe each piece inside the scanner's loop instead of through two calls per piece; the three-stage chain scanner cuts a stretch as it reads it (8 to 14 % faster warm encodes from 2 KB up and 19 % at 256 B on the two vocabularies of that shape); the byte-fallback encoders scan the raw text and key their pieces by it instead of copying every text into its marker form first; an added-token set the start-byte scan declines is searched by the tokens' prefixes in one pass, so a vocabulary with an added token beyond ASCII no longer pays a byte-by-byte state-machine walk over CJK text or prose with typographic quotes (the two affected vocabularies encode a third faster on short warm text and more than twice as fast on Chinese prose), and a vocabulary whose added tokens include newline or tab runs keeps its anchored search instead of walking every byte; cold encodes (a thread's first sight of a text's pieces) take fewer instructions on the lean miss path, a long piece that cannot be a whole-vocabulary entry skips the lookup, and a tokenizer loaded from a tiktoken-style file pre-forms multi-byte characters as the conversion path does (non-ASCII text encodes 68 % faster cold through the tekken-file loader and 33 % through the tiktoken-file one); a chat template in the OpenAI content format no longer copies every message to wrap its string content. A checkpoint that ships its vocabulary as tekken.json now loads, alone or beside a tokenizer.json conversion, through the tiktoken backend, with the file's own BOS on request and every special id decoded by its spelling; and on a checkpoint that ships the family's own vocabulary file a control-token spelling typed into a request stays text, as the family's reference library reads it, while the template's own control tokens stay control tokens. A full decode of a byte-level BPE vocabulary goes through the byte table its streams already use, an order of magnitude faster than the piece-by-piece chain the default build used before; tiktoken-file checkpoints decode whole sequences through their per-id byte table (8.2 to 7.0 ns per token). The prefix cache evicts the least recently used of a uniform sample instead of the oldest of each shard's first entry; every prompt render costs fewer allocations; and Kimi-K3.1 chat requests render with the Kimi-K3 XTML encoder instead of failing for lack of a Jinja template. Token ids are unchanged on every vocabulary. The byte-level decode sizes its buffer from the pieces it joins instead of a four-byte guess per token, so a streamed token's piece no longer grows the buffer mid-decode.

The tokenizer, measured: 1.15.0 against 1.14.0 and two references

Setup for every number in this section: one x86-64 core per cell with its sibling idle, release builds, three rounds, the median of each round's p50; cold = every encode a different text (33 texts per size), warm = the same text every time; sixteen public vocabularies at five text sizes (about 256, 2k, 32k, 128k and 500k tokens of news text in eight languages and source code). The encode cells were read on commit 1e34e44 and the decode, streaming and batch cells on 173a216 (the byte-level decode's buffer sizing is the only change to the crate between the two). The two references ran in the same cells, interleaved with SMG: the fastokens path of vLLM's Rust tokenizer (fastokens 0.3.5; riptoken 0.3.0 on the tiktoken file, tekken 0.2.0 on tekken.json) and the tokenizers 1.0.0-rc.3 pipeline through this crate's own tokenizers-v1 feature. A margin is the reference's time per token over SMG's, minus one: positive means SMG is faster. Two builds of one source differ by up to about 20 % on a cell from code placement alone (the v1 build shares SMG's decode and streaming code, and those cells read -20 to +9 % between the two builds), so a margin inside that band, the placement band below, is not a difference between implementations. The full tables, per family and size, are in the Benchmarks section the crate README gains with #5580, which ships in 1.15.1.

• Against v1.14.0, every encode cell of the fourteen vocabularies whose encoder changed is faster (all 140 cells; BERT and T5, 20 cells, run the tokenizers crate in both builds and read within their round spreads). On the ten byte-level BPE vocabularies 1.15.0 spends 86 % less time per token cold (median over 50 cells; -97 % to -67 %) and 92 % less warm (-99 % to -65 %); by size, cold 256: 85 %; 2k: 81 %; 32k: 84 %; 128k: 88 %; 500k: 92 %; warm 256: 89 %; 2k: 86 %; 32k: 82 %; 128k: 94 %; 500k: 96 %. The large warm figures say what 1.14.0 did at 128k and 500k tokens: its warm encode ran at its cold rate because its piece cache did not hold a document that size, while 1.15.0's holds the whole document. Gemma 3, Gemma 4, Llama 2, Llama 3.2, Llama 4 ran through the generic tokenizers pipeline in 1.14.0 and take the direct path in 1.15.0: the byte-fallback vocabularies (Gemma 3, Gemma 4, Llama 2) read 92 % less time per token cold and 98 % less warm, Llama 3.2 95 % and 98 %, Llama 4 94 % and 98 %; the tiktoken file (Kimi K3) 60 % cold and 77 % warm. Both builds return the same token count on every cell.
• Against the fastokens path, cold. The byte-fallback vocabularies are ahead on 15 of 15 cells by +81 % to +232 %, Kimi K3 on 5 of 5 (+32 % to +137 %), and Ministral's tekken.json by +572 % and more against the tekken crate. On the other nine byte-level vocabularies SMG is still behind on fresh text: 0 of 45 cells ahead, margins -62 % to -21 % (median -40 %), every cell beyond the placement band; the first-sight merge of a piece the thread has not seen is the cost that remains, and it is the open item of the next cycle.
• Against the fastokens path, warm. At 256 and 2k tokens SMG is ahead beyond the placement band on 14 of 18 cells of the nine non-tekken byte-level families (+28 % to +127 %); the 256-token cells of Llama 3.2, gpt-oss, GLM-5.3 and Llama 4 read +17 % to +19 %, inside it; Ministral's tekken.json +572 % and more on every cell. At 32k and above the fastokens path is ahead beyond the band on 15 of 27 cells of the nine (-37 % to -21 %); the other 12 read -20 % to +18 %, inside it, and none is SMG's beyond it. The byte-fallback vocabularies are ahead on 15 of 15 warm cells by +179 % to +458 %, Kimi K3 on 5 of 5, every cell beyond the band.
• Against the tokenizers v1 pipeline. Cold, SMG is ahead beyond the placement band on all 24 byte-level cells at 32k tokens and above (+35 % to +149 %); of the 16 cells at 256 and 2k tokens, 15 read -10 % to +17 %, inside the band, and gpt-oss at 2k reads +21 %, just beyond it (median of the 40, +41 %); the byte-fallback vocabularies are ahead on 15 of 15 cold cells (+151 % to +1757 %), every one beyond the band. Warm, 38 of 40 byte-level cells are beyond the band (+21 % to +728 %; median of the 40, +289 %), DeepSeek V4.1 at 2k and Llama 3.2 at 256 inside it (+18 % to +19 %); the byte-fallback vocabularies' 6 of 6 cells at 128k and 500k are beyond it (+1107 % to +3535 %). Llama 2's warm cells at 256, 2k and 32k tokens are behind the pipeline's whole-text word cache beyond the band (-60 % to -38 %: SMG 8.1 to 14.7 ns per token against 4.1 to 5.9); Gemma 4's at 256, 2k and 32k tokens read -19 % to -9 %, inside it. That reference number is a cache hit on the whole text, not an encode; the crate's own exact-match cache, where a deployment enables it, answers a repeated text by one lookup in the same way. The pipeline declines Qwen 3.6 (an NFC normalizer), the tiktoken file, BERT and T5, so those cells have no v1 column.
• Decode and streaming, every family in the same cells. On the nine non-tekken byte-level vocabularies a full decode of 2,048 ids takes 6.9 to 9.9 ns per token against the vLLM crate's 24.1 to 32.8 (the v1 build shares the decode code: its 6.5 to 9.1 is a second reading of the same path); the byte-fallback vocabularies read 14.2 to 16.7 against 103.0 to 107.8, Kimi K3 7.5 against 9.0, and BERT, on the tokenizers crate in both, 687 against 554 (-19 %), inside the band; on the nine, incremental decode 88 to 95 ns per token against 286 to 344; the first delta of a single stream is out after 50 ns against 261 to 271 (50 on 1.14.0); the first delta of 64 interleaved streams arrives after 110 to 130 ns against 411 to 923 (the byte-fallback vocabularies 401 to 430 against 691 to 921), and a step of those streams costs 40 to 50 ns against 230 to 241 (Ministral's tekken.json is outside these ranges: the reference's single-sequence decode cell aborts in every round, its own defect, while its 64-stream cell ran to the end and reads 1,352 ns to the first delta against SMG's 120). Against 1.14.0 the full decode of the ten byte-level vocabularies reads -94 % to -93 % and of the byte-fallback vocabularies -2 % to -0 %; a one-thread batch of 1,000 texts runs at 8.2 to 42.6 times its 1.14.0 throughput (20.4 to 45.0 million tokens per second on the nine non-tekken byte-level vocabularies against fastokens' 24.0 to 60.6 and v1's 7.1 to 13.2 on the eight of them the v1 pipeline encodes (it declines the Qwen 3.6 file); 25.7 to 64.9 on the byte-fallback vocabularies against 7.0 to 27.2).
• Instructions per token on the headline cells (perf stat over fresh processes, the difference of a long and a short pass, so the load phase cancels; 1.14.0 -> 1.15.0, the fastokens path beside): DeepSeek V4.1 32k cold (wide corpus) 2978 -> 343 (fastokens 256); DeepSeek V4.1 32k warm 1382 -> 209 (fastokens 192); Llama 3.2 2k cold (wide corpus) 8895 -> 266 (fastokens 228); Llama 3.2 2k warm 6658 -> 128 (fastokens 210); Llama 3.2 256 cold (wide corpus) 9103 -> 295 (fastokens 254); Qwen 3.6 128k cold 9610 -> 473 (fastokens 449); Gemma 4 2k warm 5753 -> 129 (fastokens 399); Kimi K3 2k warm 1930 -> 702 (fastokens 1197).

Per text size (medians over the families of the group; cells = families; the count columns read: cells SMG ahead beyond the placement band / inside it / behind beyond it): the ten byte-level BPE vocabularies and the three byte-fallback vocabularies.

Multimodal: the reference processors' bytes, several times faster

The shared bicubic resize kernel of the Qwen, Gemma 4, GLM, MiniMax and LLaVA families runs its vector form on 64-bit x86 processors with AVX2 as it did on ARM, and the processors whose reference resizes with Pillow run both passes on the vector unit as byte dot products, each tap in exact chunks, with the same bytes as before; the MoonViT, glm53_flash, float-bicubic and Phi-3 processors now equal their reference processors bit for bit where each carried a last-bit rounding difference, a LLaVA-format checkpoint whose vision tower is Pixtral is preprocessed by the Pixtral processor with the Pixtral row layout, and MiniMax-M3 images are placed on the grid the model's own processor produces for tiny and extreme-aspect images too. A Step3-VL image spec arrives: the global view and its crops, with a token bound that counts the square pad of a thin image. The image path reuses its large per-request buffers (the decoded frame, the patch tensor, its wire encoding) and the allocator keeps the buffers that carry a pixel tensor to the engine, and the resamplers, destination images, pad canvases and the LLaVA and Gemma 4 tensors are written once instead of zero-filled first, which removes the page faults each image used to pay. The audio frontend builds its window, filter bank and FFT plan once and runs a clip through one scratch: a 30 s clip at 16 kHz preprocesses in about 20 ms instead of 594 ms, bit-identical; a clip whose header claims an impossible sample rate is refused as soon as the header is read instead of costing minutes of a core.

Video follows the engines' own loaders. Checkpoints of the qwen_vl family accept one clip per request, sampled by the family's loader rule (total / fps x 2 frames in whole temporal groups, at most 768) with the processor's grid and per-group timing passed to the engine; Qwen3-Omni video is sampled by its loader's rule and its video tokens carry the processor's time per temporal patch; the interval sampler hands on exactly the frames its loader picks for short and very long clips, a clip too long for the decode budget is refused rather than thinned, a container that claims more frames than its stream presents is recovered against its own claim, and a git-snapshot build of ffmpeg gets the current -fps_mode passthrough flag instead of the removed -vsync 0, so the frame selection each family's rule names is the one that runs. The multimodal configuration resolution fetches a checkpoint's configs inline through the Hub client's own HTTP client, so an elapsed bound ends the request with its connection, a Hub failure other than a 404 is a 502 for that request rather than a default pinned for the tokenizer's lifetime, and a failed fetch of a processor configuration file is logged with the file name. Icon files whose image is a PNG entry decode whatever the PNG's colour type, and on 64-bit x86 processors a CMYK JPEG is converted to RGB on the vector unit, which cuts its decode time. Images above the patch budget of the MoonViT-based processors are resized with the same bicubic kernel as the checkpoints' own processors and reach the engine byte-identical to the reference (they were one 8-bit level off on part of their pixels, and a transparent image further off through the premultiplication); the Pillow-exact resampler runs its two passes in cache-sized bands instead of through a whole intermediate image per request, which puts the preprocess stage of such an image back at the speed of the SIMD kernel it replaced for parity with no per-thread memory that grows with the image; and on 64-bit x86 processors the one-pass wire write runs its vector formula on AVX2, RGB patch rows split into channel runs with byte shuffles, and an image with an EXIF orientation is rotated and flipped on the vector unit. A gauge of the preprocessed media bytes in flight (smg_mm_inflight_bytes) and a pool_queue stage of the processing histogram (the time a task waited for a blocking-pool thread) make the pipeline's pressure readable; the image cache's fingerprint is built without cloning the model's JSON and a cached batch is no longer copied twice. The video families whose sampler reads its rate, frame budget or temporal group from the processor config now read the config their video processor is built from, and a sampler hands its frame picks to the decoder as one short expression per clip, so a long clip whose stride flips between two values is decoded with the loader's exact picks instead of being refused for the length of its frame list. On the engine side, a vLLM encode/prefill/decode fleet distributes the images of one request across its encoder workers by the existing encoder policy, and the Rust vLLM servicer advertises the engine's encoder-cache connector and carries the encoder-cache transfer parameters both ways, so such a fleet runs on the default servicer.

Multimodal, measured: 1.15.0 against 1.14.0 and the reference processors

• Faster per image on every class, the same bytes. Against v1.14.0 on one core, the Qwen3-VL pipeline takes 4 to 48 % less per image on every one of the 19 image classes (the 1920x1080 JPEG and PNG and the 8192x8192 JPEG about half the time: -48 %, -41 % and -46 %; the 4096x2304 JPEG -11 % and the 3840x2160 PNG -25 %; the 1024x768 and smaller classes -4 to -20 %), and the other families take 12 % (Kimi K3) to 57 % (DeepSeek V4.1) less at the median of the classes (MiniMax M3 -44 %, Gemma 4 -41 %, Qwen2-VL -38 %, LLaVA-NeXT -31 %, GLM-5.3-Flash -14 %), with CPU per image down by the same fractions (-11 to -58 %); of the 160 single-thread cells with a 1.14.0 arm one reads level (GLM-5.3-Flash JPEG 1920x1080, +0.8 %, inside its round spread) and none reads slower; under eight concurrent decodes (one per physical core): the time per image is 0.51x (DeepSeek V4.1) to 0.93x (Qwen3-VL) of 1.14.0 at the median of the classes, images per second 1.06-1.90x, CPU per image 0.52-0.93x; six of the 180 load cells read above 1.14.0 (GLM-5.3-Flash JPEG 1920x1080 +15 %, Qwen3-VL JPEG 1024x768 +12 %, JPEG greyscale +4 %, PNG 1024x768, PNG RGBA and BMP +1 to +2 %), inside those cells' round-to-round spread of up to 25 %, and are listed as read; the single-thread cells of the same classes read -8 to -17 % (GLM-5.3-Flash's +0.8 %). Against the engine's own transform (the checkpoint's processor at the engine image's library pins, one compute thread) the pipeline is 2.86x faster at the median of the Qwen3-VL classes (1.6x on the lossless WebP and the 8192x8192 JPEG to 6.1x on a 256x256 JPEG), 15.9x for DeepSeek V4.1, 4.9x Kimi K3, 3.1x Step3-VL, 2.2x MiniMax M3, 1.8x Qwen2-VL, 1.6x LLaVA-NeXT, 1.4x GLM-5.3-Flash and 1.15x Gemma 4; Gemma 4 is the one family where the engine's transform is as fast or faster on 8 of the 19 classes (0.74-0.98x: BMP, GIF, TIFF, the greyscale, EXIF-rotated and progressive JPEGs, both WebPs; its preprocess of the fixed 896x896 output costs a constant 6 ms per image, filed as a finding), and GLM-5.3-Flash's static GIF reads 0.97x. What moved: the Pillow-exact resample runs its two passes in cache-sized bands on AVX2 (and on the AVX-512 VNNI byte dot products where the core has them), the torch antialiased resize takes an AVX2 path, the wide resample taps are built for the vector rows only, the wire formula is written on AVX2, RGB8 images are oriented and CMYK JPEGs converted with vector lanes, the decoded frame, the patch tensor and its wire encoding are pooled across requests instead of zero-filled per request, a video frame moves to its patches without a copy, the Gemma 4 and LLaVA tensors are written once instead of zero-filled, and a vision processor borrows the batch's images instead of being handed copies.
• Video and audio. Measured on one core against v1.14.0: the 23 Qwen3-VL clip classes (H.264, H.265, AV1 and VP9; 1 to 60 fps and variable rate; 0.5 s to 300 s; 321x241 to 1920x1080; MP4, MOV, MKV, WebM and animated GIF) produce the same wire bytes in both releases, 20 of 23 are faster in 1.15.0 (0.69-0.94x of the 1.14.0 time: a 5 s 30 fps H.264 clip 69 ms against 85 ms, a 10 s 1080p clip 1.76 s against 2.12 s) and three are within round noise (animated GIF, MKV, AV1), at 0.5-0.6x of the 1.14.0 CPU per clip; the 26 Qwen3-ASR audio classes (WAV from 8 to 96 kHz, 1 to 6 channels, 16-, 24-, 32-bit, float and A-law; FLAC, MP3, AAC, Opus and Vorbis; 0.1 s to 600 s) produce the same wire bytes in both releases and every class is faster: a 30 s 16 kHz clip takes 14.6-22 ms where 1.14.0 took 450-461 ms, a 600 s clip 0.31-0.43 s instead of 9.1-9.2 s, inputs that need resampling (22.05 and 44.1 kHz, AAC, Vorbis) 0.47-0.49x, Opus 0.27x, 48 kHz 0.15x. Under load (8 concurrent decodes): the video clips at 0.82x of 1.14.0 at the median (0.67-0.98x, 23 of 23 the same bytes), the audio clips at 0.08x (0.03-0.49x, 26 of 26 the same bytes).
• Not yet bit-identical: LLaVA and LLaVA-NeXT (the decoded pixels are identical to the reference's, the resized tensors are not; the fix is in review). AVIF stays undecoded (the engine's own loader decodes it).

The numbers. Setup for every number below: one x86-64 core per decode thread, release builds, 2 warm-ups and 10 timed repeats per class (the clip classes: 1 warm-up and 5 repeats), three interleaved rounds per build for the image classes and two for the clip classes (medians); under load: 8 concurrent decodes on 8 physical cores. "The engine's own transform" is the checkpoint's processor at the engine image's library pins with the engine loader's rules, one compute thread. 1.15.0 = 1e34e4481, 1.14.0 = becc40c01.

Qwen3-VL, milliseconds per image:

Every image family the pipeline serves, medians over the 19 image classes:

Symphony: a call the table could not read whole says so

Tool calls in the GLM-style keyed, Qwen tagged, MiniMax M3 XML, attributed and pythonic syntaxes are no longer delivered as well-formed calls when the parser refused bytes inside them or the value nests past its reader's bound: such a call reaches a chat client with arguments a strict JSON reader refuses, reads incomplete on the Responses API and keeps the {} input a cut call has on the Messages API. A DeepSeek call the model writes without its calls block, in any DSML spelling the tables read, reaches the client as the call; with Symphony reading Kimi K3, a reply written into the prefilled thought and closed with the answer's closer no longer carries that closer into the reasoning text; the MiniMax M3 boundary closer keeps its meaning; a DeepSeek-V3.2-Exp deployment reads with the V3.2 table and deepseek-ai/DeepSeek-V4-Pro with the deepseek_v4 table under its own id; a marker kept as text past ignore_eos no longer counts as a malformed run. The reader borrows a tagged call's streamed pieces and the text a marker cuts out instead of copying them, keeps its counter handles instead of registering one per record, finds its turn opener with a finder built once with the table, and the formats registry finds a model's group in constant time; a table with no marker skips the marker scan, so the prompt replay and every output delta there cost nothing for it. A Symphony reading that is not clean marks the request span with its table, outcome, reason, bytes and finish, once per request, and the mark reaches the trace and the output's log line, with the malformed-output log lines bounded per table and reason. On the Harmony path a chat request returns only the top_logprobs candidates it asked for, none when the field is absent, and a gpt-oss request's leading system or developer message becomes the developer message's instructions, with no empty developer message written, as vLLM renders it.

OpenAI, Messages and Responses: more of each contract

/v1/completions accepts a prompt given as token ids, and a batch of them, routed and counted as the prompt, and returns the logprobs object on every choice, unary and streamed. logit_bias reaches the engine on the vLLM, SGLang and TokenSpeed wires; n on chat completions takes the public API's 1 to 128; the nested allowed_tools tool_choice shape is accepted and written back to relayed upstreams; an assistant refusal content part and a function message with content: null are accepted; under stream_options.include_usage every chunk before the usage chunk carries "usage": null; a tool whose function declares no parameters renders and forwards as each API's reference lists it. On the Messages API, output_config.format of type json_schema constrains the answer, thinking: {"type": "between_tools"} is thinking off, message_start reports the prompt's token count, an unreadable content block is refused by position and reason, the usage reports the engine's prefix-cache hit as cache_read_input_tokens, the response carries request-id beside x-request-id for the Anthropic SDKs, top_p and top_k are bounded as the engines bound them, and the Anthropic-shaped model entry carries lifecycle and the card's token limits with GET /v1/models paging by limit, after_id and before_id. On the Responses API, frequency_penalty, presence_penalty, top_k, min_p, repetition_penalty and stop reach the engine on every turn of the regular pipeline and stop on the Harmony pipeline too, a turn cut by anything but the token limit reads status: incomplete on every path and in the stored object, a Harmony stream's stored object keeps its text item, and every answer carries x-smg-parser. The generate endpoint's streamed answers carry the typed finish_reason object on a plain stream, a stream whose tokenizer cannot decode a chunk ends with an error event, and a pretokenized request whose ids run past the model's embedding table is refused before any engine is spoken to. A streamed response that stops inside a multi-byte character no longer ends with a replacement character for a character the model never finished, and otherwise ends with the characters a full decode produces. A GLM-5.3 request's minimal, medium and xhigh reasoning efforts are read as low, high and max, as the provider reads them, and the WASM and tokenizer management routes refuse in the control plane's {"error", "code"} envelope (a 409 names the id that holds the name, the 404s carry a code, a gateway without the module manager answers 503), while an unsupported encode policy is refused with a message that names the encode_policy field from the binary and the Python binding alike.

Mesh, discovery and operations

A mesh member keeps one connection per peer and direction for its probes, status broadcasts and state sync, reads its TLS material once, answers a heartbeat once and dials one every 15 s, and logs one line per membership change at the info level. A member another member still reaches is no longer marked Suspected and Down by one that cannot reach it directly; a member that comes back under its name after a crash tombstones its earlier life's worker keys, so its peers rebind their imports at once; an import follows the owner's in-place replacement of a worker; a gateway restarted under its mesh name publishes above the version its peers hold; every --mesh-peer-urls entry is dialed in turn and a malformed one refuses the start. A worker given at start-up is health-probed the moment the health loop starts; the liveness sweep and readiness checkpoint read the registry's membership snapshot instead of walking the worker map; the in-flight connections gauge follows the in-flight count on every release and heals within one sample interval; a gateway start validates its configuration once per path; and a tenant rate-limit reservation copies nothing per request while the limiter also holds requests through the disaggregated HTTP router and the external provider routers. On the engine side, a direct-ZMQ disconnect left over from a previous engine generation no longer closes the link admitted after a gateway restart, a bare IPv6 host without brackets is refused at registration, and a lockstep data-parallel group whose holder reads a request's add late is woken again instead of being declared dead. A membership change made by the Kubernetes router discovery logs the same Member <name> is <Status> line as gossip does, once per change; GET /engine_metrics scrapes HTTP workers only and answers 200 on a fleet without them, naming the workers it leaves out; GET /workers?format=manifest answers the registered fleet as a file discovery manifest with no secret in it, filtered by ?origin=local or ?origin=discovery; the CLI and the launcher refuse a discovery selector entry without key=value by name; and --admin-max-connections bounds the connections the admin listener holds open at once, as --max-connections bounds the data listener (no bound by default).

Control plane: keys reloaded in place, the configuration readable, every mutation audited

POST /secrets/reload reads the serving, tenant and control-plane key files again and puts their keys in force without a restart, for the admin role, and reports serving_changed: true when a tenant key moves to another tenant; with --secrets-auto-reload the gateway watches those files itself and puts a changed file's keys in force without a call, and GET /config names the keys in force by fingerprint after a reload, not the start's. GET /config answers the effective configuration with every secret redacted, for the admin role, when control-plane authentication is configured; the routes that exist only behind that authentication form one group with one layer, and the start-up record says whether they are mounted. --control-plane-api-keys-file and CONTROL_PLANE_API_KEYS_FILE read the control-plane API keys from a file such as a mounted Secret, with the source named in the start-up record (CONTROL_PLANE_API_KEYS now splits its entries at , and at line ends). Every control-plane mutation writes a second audit line after its handler, with the status, the resource and the request id of its admission line; the generated API document lists every control-plane route and marks each with the bearer scheme the control plane takes; and POST /wasm answers the uuid of the module it registered as soon as the load ends, two modules of one name each getting their own. The MCP client bounds the response bodies it reads from a server (rmcp 3.9), which closes a memory-exhaustion exposure reachable through a request's MCP server URL, and the dependency audit reads clean at the release.

Python package, bindings and chart

The smg Python package adds smg.tokenizer.Tokenizer, which loads a checkpoint's tokenizer with the Rust code SMG serves with, encodes and decodes as the gateway does, pickles for spawn-started DataLoader workers and datasets.map, renders a chat request through the gateway's own request path (render_chat) and takes transformers' apply_chat_template arguments. smg launch --backend and the Python Router accept gemini and trtllm, and both refuse a Prometheus bucket list that is not finite, positive and ascending and a JWT role mapping outside admin and user, as the binary does. The launcher passes its key files to the gateway, so POST /secrets/reload and --secrets-auto-reload work for a launcher-started gateway too. The Helm chart's router.parser picks the parser, router.policy accepts every policy the launcher takes, the worker takes servicerImpl, the dashboard gains a prefill-decode row of eight panels, the alerts gain SmgHighRouterErrorRatio, read the priority scheduler's families, fire SmgNoHealthyWorker on a gateway with no worker series, no longer count the kubelet's probe polls, and carry promtool unit tests. The chart also mounts the tenant rate-limit YAML and the priority scheduler's class table from ConfigMaps (router.tenantRateLimit, router.priorityScheduler) and renders their flags; renders the admin listener with its own Service (router.adminPort) and the control-plane API keys from a Secret mounted as a file under auth.controlPlane; takes the prefill, decode and encode roles of disaggregated serving as typed values under router.disaggregation and router.discovery.kubernetes; requires the native JPEG decoder by default (router.mmTurbojpeg: require); probes a gRPC worker's start-up through the servicer's gRPC health service, so the pod counts as started when the engine serves; and refuses the Slurm role job names without a disaggregation mode and the regular workers under one. The launcher and the Python Router start from the CLI's defaults (cache-aware routing, port 30000, a 1800 s worker start-up wait, a 120 s eviction interval), no longer pin a ZMQ worker to the SGLang wire without --backend, refuse an encode policy outside random, round_robin and consistent_hashing by name, keep the start-up worker URLs in IGW mode, and hand out the tokenizer thread pool only once every worker runs.

Measured at every family's shipped defaults, against 1.14.0

Each family ran v1.14.0 and the 1.15.0 candidate as shipped (each build's own defaults, identical engine and router arguments, no pinned engine configuration), one engine at tensor parallel 4 and one router per arm on separate hosts of the same type, both arms under every cell at the same time. The shape is the previous release table's 8k-token chat (a 4k-token shared prefix, 128 output tokens). Capacity is the open-loop ladder's last step that holds the objective (TTFT p99 under 1 s, goodput at least 0.9, no errors); the matched-load points are paced at 65 % and 85 % of the lower arm's capacity; router CPU per request is the router's CPU time per completed request over the paced points. v1.14.0 -> 1.15.0.

The previous table's other families (GLM 5.3 Flash, DeepSeek V4 Flash, Kimi K3, Gemma 4 26B, Qwen 3.6 35B-A3B, Inkling-Small and MiMo V2.6 Flash) were not measured in this release's window: no host of their type was free before the cut-off, so no figure is given for them.

In the simultaneous pairs the v1.14.0 arm's host ran its engine slower per step at light load (DeepSeek V4.1 Flash 9.9 vs 8.8 ms at 10-20 running requests, generated-token ceilings 1909 vs 1917 tokens/s per engine; MiniMax M3 14.1 vs 12.8 ms at 10-20 running requests, generated-token ceilings 1476 vs 1513 tokens/s per engine), so the TTFT margins at matched load include a host share in the candidate's favour. On the second DeepSeek V4.1 Flash host pair each build was read on each host in turn: on the host that ran either build's engine slowest (v1.14.0 first, then the candidate): v1.14.0 13.0 and the candidate 10.0 requests/s at the objective (first breaks 16.0 / 13.0); generated-token ceilings 1584 then 1581 tokens/s per engine, engine step at 10-20 running 15.1 then 12.4 ms; on its partner host (the candidate first, then v1.14.0): v1.14.0 18.0 and the candidate 18.0 requests/s at the objective (first breaks 20.0 / 20.0); generated-token ceilings 1880 then 1890 tokens/s per engine, engine step at 10-20 running 10.5 then 8.7 ms.

Closed-loop confirmation (16 clients per engine, no think time, equal-length requests): DeepSeek V4.1 Flash 11.02 -> 14.61 req/s, TTFT p50 232 -> 235 ms, TPOT p50 10.1 -> 6.7 ms, per-request inter-token p99 (median request) 178 -> 8 ms; MiniMax M3 13.59 -> 12.85 req/s, TTFT p50 98 -> 236 ms, TPOT p50 8.6 -> 8.0 ms, per-request inter-token p99 (median request) 42 -> 9 ms. With equal-length requests and no think time the clients run in convoy; the v1.14.0 arms' streams carry a per-request inter-token gap at the p99 that the candidate's streams do not, which spreads v1.14.0's clients apart (a shorter time to first token on MiniMax M3, a throughput loss on DeepSeek V4.1 Flash); the open-loop points above, with Poisson arrivals, are the comparison free of this effect.

Readings where 1.15.0 is behind 1.14.0, stated as measured:

• MiniMax M3: closed-loop at 16 clients per engine 13.59 -> 12.85 req/s (-5.4 %), a convoy-sensitive shape (see the closed-loop sentence)
• DeepSeek V4.1 Flash, same-host reading on the host that ran either build's engine slowest (v1.14.0 first, then the candidate): capacity at the objective 13.0 -> 10.0 req/s (the goodput criterion at the 13.0 step)

How the family figures were measured

Each family ran v1.14.0 and the candidate as shipped, one engine at tensor parallel 4 and one router per arm, each arm on its own host of the same type, both arms of a family under every cell at the same time (the same seed, per-arm prefix salts). The candidate arm ran the router and engine images built from main at 1e34e44 (2026-11-09 22:12 UTC) on the vLLM v0.31.0 base; the release record carries two later gateway-side changes that are not in the arms (#5556, the tokenizer's full-decode buffer; #5573, the gateway configuration view); the v1.14.0 arm ran the published smg:1.14.0 router image and an engine image of the same vLLM v0.31.0 base with the published smg 1.14.0 and smg-grpc-servicer 0.15.0 packages installed from PyPI (the published multi-engine image of that release carries no manifest for this host architecture, so the shipped packages were installed on the same base the candidate uses). Each arm kept its build's own defaults: the family recipe's engine arguments were identical on both arms (tensor parallel 4, the context length, a 16k-token admission step, the sequence cap, prefix caching, KV-cache events) with no CUDA-graph, compilation, eager or capture-cap setting on either side, and the router arguments were identical on both arms (cache-aware routing on the chain index with the family's block size, the family's parsers, media processing on the engine, Kubernetes service discovery, tracing and JSON logs).

Traffic was the 8k-token chat shape of the previous release table (8192 prompt tokens with a 4096-token shared prefix over 16 distinct bodies, 128 output tokens, greedy), generated by four shards per arm beside the arm's engine: an open-loop capacity ladder with Poisson arrivals over eight 150-second steps, with the capacity read at the last step that holds the latency objective (TTFT p99 under 1 s, goodput at least 0.9, no errors or dropped arrivals) before the first step that breaks it; paced open-loop points at 40, 65 and 85 % of the lower arm's capacity (180 s warm-up, 600 s measured, 60 s drain); a closed-loop confirmation point at 16 clients per engine (600 s); and the ladder once more where the window allowed. TPOT is the per-request mean inter-token latency; router CPU per request is the router container's CPU time over the measured window divided by the completed requests. DeepSeek V4.1 Flash ran on two host pairs, MiniMax M3 on one. On the second V4.1 pair the as-placed ladder showed one host running the identical engine image, with identical arguments, well below its partner (the same graph capture, KV configuration and clocks; no preemption), so both arms were torn down and re-applied with the builds exchanged on the same two hosts and the ladder run again: each build is read on every host it ran on (three ladder readings per build, the slower host once in each set), the same-host readings are stated on their own, and the exchanged pair's matched-load points stay out of the matched-load and router-CPU figures because its candidate sits on the slower host. The window of the release cut-off left no free host for the other families of the previous table, which are listed as not measured. Every figure comes from the shards' own summary files; the step length is shorter than the previous release table's 240-second steps because of the window.

How this release was validated

• Provider verification suites for four families (DeepSeek V4 and V4.1, Kimi K3, MiniMax M3), each through the legacy and the Symphony reader, were replayed against the routers of the candidate record (main at fc40559) on the standing engines, with the GLM 5.3 Flash rotation and the KV-event credit cells beside them: PASS on 8 of 8 family rows against the previous record's run (1e34e44) as the reference, 19 failed cases in all, every one but one in a documented draw class of its model (the reruns of the flaky-class cases passed); the exception, read as the model's draw: W04.chat.non_thinking.nonstream on the DeepSeek V4.1 legacy verification suite failed the kit's final-step exact-answer assertion on a draw that framed the authored answer with a preamble line (step 0's four parallel calls right); the same-router replays read 13 of 13 exact at step 1 and 13 of 13 with the four calls at step 0, the history-rendering class the DeepSeek V4 W01 and W04 ids carry, first seen on V4.1; the rotation inside its bands and the credit cells reporting the engine's cache hits in usage on every family. Between the verified record and the release's record (main at 06e1b29) seventeen merges landed: two features (#5588 gateway-perf, #5576 bindings); four fixes (#5592 gateway-perf, #5553 metrics, #5518 sglang, #5573 gateway); three performance changes (#5586 kv-index, #5552 kv-index, #5556 tokenizer); five test changes (#4794 e2e, #5589 gateway, #5587 zmq-client, #5578 zmq-client, #5579 policies); three CI fixes (#5582, #5581, #5577); one of them touches the engine-side package (#5518); the chain's next record verifies them. The table below is the record named.

• Shipped-defaults benchmark against v1.14.0 (the previous release table's 8k-token chat shape, each build as shipped, both arms of a pair at once): DeepSeek V4.1 Flash 18.0 -> 18.0 requests/s at the objective (level), TTFT p99 at matched load -21 % at 65 % and -10 % at 85 %, router CPU per request 3 % less; MiniMax M3 12.0 -> 12.0 requests/s at the objective (level), TTFT p99 at matched load level at 65 % and level at 85 %, router CPU per request 4 % less. Readings behind 1.14.0, stated as measured: MiniMax M3: closed-loop at 16 clients per engine 13.59 -> 12.85 req/s (-5.4 %), a convoy-sensitive shape (see the closed-loop sentence); DeepSeek V4.1 Flash, same-host reading on the host that ran either build's engine slowest (v1.14.0 first, then the candidate): capacity at the objective 13.0 -> 10.0 req/s (the goodput criterion at the 13.0 step).
• Media parity on the pinned reference processors: a 500-image tier of the 19,556-image fixture corpus (every one of its 51 derived encoding variants, photographs, scanned documents, charts, screenshots and diagrams) read through the release's pipeline for the three families whose engine-produced reference digests the corpus carries is bit-identical to those digests on the decoded pixels, the float32 tensor, the bf16 wire bytes and the image grid: Qwen3-VL 500 of 500, Qwen2-VL 500 of 500, MiniMax M3 500 of 500, 0 differences. On a 63-class synthetic corpus (JPEG baseline, progressive, CMYK, greyscale and EXIF orientations 1 to 8; PNG 8- and 16-bit, palette, transparency and RGBA; WebP, GIF, BMP, TIFF; 1x1 to 8192x8192 pixels) against each checkpoint's own processor at the engine image's library pins, every image family the pipeline serves is bit-identical on every decodable class except two: Qwen3-VL, Qwen2-VL, Qwen2.5-VL, Qwen3-Omni, GLM-5.3-Flash and MiniMax M3 58 of 58 each; Gemma 4, DeepSeek V4.1, Kimi K3, Kimi K2.5, Phi-3.5-vision, Nemotron Omni and Pixtral 61 of 61 each; Step3-VL 59 of 59; LLaVA and LLaVA-NeXT decode identical pixels but their resized tensors differ from the reference on 60 of 61 classes (the fix is in review).
• Parity censuses on the pinned reference trees: the render-parity harness (799 recorded models) and the tokenizer harness (822 models) read no unlisted difference, the Symphony parse replay reads every recorded family, and every image-processor change in this release carried its own bit-exactness proof against the reference processor on the family's oracle set.
• In CI: every pull request runs the chat and router suites on a CPU mock engine and the benchmark gates against its base; every change to the serving path carried a before/after table of instructions, allocations and bytes per request from the gateway's cost ledger, and the gateway cost benchmark can run its cells on a prefill-decode fleet, so changes to that path are measured before they merge; the audit of the dependency graph ran on every lockfile change and nightly, and reads clean at the release (the one advisory raised during the cycle, against the MCP client's rmcp dependency, was closed by the move to rmcp 3.9).
• The release branch's CI: every host row of the test matrix (the Rust and Python suites, lint, the wheel, the mock-engine chat and router suites, the benchmark gates) read green on the record, and so did the provider conformance suites whose keys reach the hosted runner pool through the key proxy of #5577 (the Anthropic Messages and OpenAI Responses rows among them); two provider conformance rows, the xAI Responses and the OpenAI Realtime suites, read red on that pool for a reason outside the code: the pool had not yet been rolled onto the runner values of #5577, so the proxy answered their calls with a 404 and refused the WebSocket upgrade with a 503; and one single-card chat leg of the matrix, bound to a two-runner pool that was offline from the evening before the cut, stayed queued with no runner, so the record's run could not reach its finish aggregate; the three were judged as environment rows for the tag.
• This release was cut from main at 06e1b29de on 2026-11-10; its release branch ran the full test matrix before the tag.

Notable fixes

These bugs shipped in v1.14.0. Problems introduced and fixed during this cycle are not listed.

Streaming, API output and tool calls

• A streamed response could end with a stray replacement character when the token limit fell inside an emoji or a request was cancelled; a stream now ends with the characters a full decode of the same tokens produces, on byte-level, tiktoken and byte-fallback vocabularies alike.
• logit_bias never reached the engine on the vLLM, SGLang and TokenSpeed wires; it does now, a bias or key outside the contract is refused with a 400, and TensorRT-LLM refuses the field.
• /v1/completions with logprobs returned no logprobs object on the gRPC router; every choice now carries the tokens, their logprobs, the requested alternatives and the text offsets, unary and streamed.
• A Symphony tool call with refused bytes inside it arrived as a well-formed call with the readable arguments only, or {}; it now reads incomplete, and a MiniMax M3 call nested past the parser's bound no longer comes back whole with a member silently dropped.
• A DeepSeek tool call written without its calls block streamed as assistant text with the call lost; it reaches the client as the call.
• Kimi K3 under Symphony carried the answer's closer into the reasoning text when the model wrote its reply into the prefilled thought; the closer ends the thought and the reply reaches the client once.
• Responses sampling knobs and stop sequences were dropped after the first turn of the regular pipeline, and stop never reached a Harmony model; both reach the engine on every turn now, and a turn cut by anything but the token limit reads status: incomplete on every path.
• A streamed Messages response reported 0 prompt tokens in message_start; a Messages request's output_config.format of type json_schema did not constrain the answer; top_p: 0.0 and a top_k of 0 were forwarded where the engines read them as unset. Each is now served or refused by name.
• A chat completion chunk before the usage chunk lacked "usage": null under stream_options.include_usage; a tool without parameters rendered differently from the engine's own server; the nested allowed_tools shape was refused; an assistant refusal part and a function message with null content were refused. All accepted now.
• A generate stream whose tokenizer could not decode a chunk sent the rest of the answer with empty text; it ends with an error event and a logged failure. The gRPC router's plain generate stream carried a bare finish_reason string where every other path carried the object.
• A character split across two upstream chunks reached the Responses stream broken through the external-provider router; it arrives intact.
• Chat requests on the Harmony path returned every top_logprobs candidate whatever the request asked, and a gpt-oss request's leading system or developer message was not rendered into the developer message's instructions while an empty developer message was; both render as vLLM does.
• A GLM-5.3 request's minimal, medium and xhigh reasoning efforts were forwarded as written where the provider reads them as low, high and max; they are read that way now.
• The gRPC router's time to first token read near zero and its generation duration short on workers that hold their response headers until the first chunk; the latency metrics are anchored at the RPC dispatch.
• The WASM and tokenizer management routes refused outside the control plane's error envelope, POST /wasm did not answer the uuid of the module it had registered, and an unsupported encode policy was refused without naming its field; each is served or refused by name now.
• The MCP client did not bound the response bodies it read from a server, a memory-exhaustion exposure reachable through a request's MCP server URL; rmcp 3.9 bounds them.

Routing, mesh and load

• A mesh member could be marked Suspected and Down by a peer that merely could not reach it directly while another member still did; a member that crashed and came back kept its old imports on its peers for good, deaf to its own removals; a worker replaced in place under the same address kept its first specification on its peers. All three follow the owner now.
• Every mesh probe opened a connection of its own and, under mTLS, read the CA twice and the certificate and key per dial; a member with a local ZMQ worker logged a refused tombstone for it every thirty seconds; a heartbeat was answered by both ends.
• Under the routing-key override, manual and consistent_hashing placed a header key by its full form, so turns of one conversation could land on different workers; they place by the lineage-stripped form, and consistent_hashing keys stay clear of the overload veto.
• The in-memory rate-limit backend kept one bucket per client IP for the life of the process; buckets nobody uses are dropped. The cache-aware mesh hash index was bounded by entry count alone; it is bounded by the text it holds.
• The SGLang prefill/decode rendezvous room on the HTTP path was minted without regard to the dp-aware prefill rank; it is congruent to it, and a prefill side of several ranks no longer logs a WARN per request when the room's residue overrides the pin.
• A retried gRPC request was counted once per attempt in the router request, error and duration families; it is counted once, with the duration from the request's start. The priority scheduler's clamp counters ignored priority_scheduler_tenant_metric_top_n; the generate endpoint counted no prompt or cached tokens.
• smg_http_connections_active could report a phantom request after a client closed before the answer, and stay above the in-flight count on an idle gateway; it follows the in-flight count and heals within one sample interval.
• A disaggregated request could be parked on a decode worker whose running window was full while a compatible sibling had room, so the load-scoring decode policies herded under the admission gate; the pick skips the full partner.
• Workers the routing vetoes exclude were counted in the overload pool checks; they are left out.
• The KV index could panic in the event lane, or truncate a hash array, when one run's child table or array outgrew an arena chunk; such a store is declined and counted, with the lookups under a run at the cap still bounded.
• GET /engine_metrics failed on a fleet without HTTP workers; it scrapes HTTP workers only and answers 200, naming the workers it leaves out.
• A discovery selector entry without key=value was dropped without a word by the CLI and the launcher; it is refused by name. A membership change made by the Kubernetes discovery was logged unlike gossip's; it logs the same Member <name> is <Status> line, once per change.

Prompts and tokenizer

• DeepSeek V3.2 and V4 prompts differed from the engine's own encoder for a developer message, a conversation ending in a system message, the tools block's blank lines and the reasoning-effort prompt; Kimi K2 tool declarations differed in key order and indentation. Each renders byte for byte as the checkpoint's encoder does.
• A tokenizer.json truncation or padding block cut the gateway's encodes at its max_length; a prompt encodes to its whole id list. A Llama-class checkpoint shipped without a config.json, or of a model type the reference leaves to the Llama class, carried a second word marker after a spelled special token.
• A tokenizer whose added tokens include a letter and its compatibility form under an NFKC normalizer picked an id per process; the ids are the same in every gateway process.
• A tiktoken-based checkpoint with named chat templates did not shape a tools request's content for the tool_use template it renders through.
• A failed fetch of a checkpoint's config.json, generation_config.json or processor configuration during a Hub download was silent; it is logged with the file name and the error, and a Hub failure other than a 404 fails the multimodal resolution for that request instead of pinning defaults.
• A control-token spelling typed into a request was encoded as the control token on a checkpoint that ships the family's own vocabulary file, where the family's reference library keeps it as text; it stays text, while the template's own control tokens stay control tokens.

Multimodal

• Several families were refused above a media count their checkpoints never set (ten images on the Qwen2-VL, Qwen3-VL, Inkling, Muse Spark and Kimi K3 families; one video or ten audio clips on those and Qwen3-Omni and Qwen3-ASR; counts on Gemma 4, Nemotron Omni and DeepSeek V4.1); the bound is the engine's or the deployment's.
• MiniMax-M3 tiny and extreme-aspect images were scaled up to a 112 px short side first and so carried a different token count; the MoonViT, glm53_flash, float-bicubic and Phi-3 processors carried last-bit rounding differences; the Phi-3 HD crop budget ignored the checkpoint's num_crops; a Pixtral-tower LLaVA checkpoint was preprocessed with the wrong processor. All equal their references now.
• Video frames were resampled by a generic rule instead of the engine loader's picks for short and very long clips, a container that overstated its stream decoded to different frames than the engine's loader, and a git-snapshot ffmpeg silently resampled every clip at the configured rate because of a removed flag.
• An audio clip with an impossible header rate could cost minutes of a core and tens of gigabytes; it is refused as soon as the header is read. An icon file whose PNG entry was not RGBA was refused.
• A Hub that answered a file's metadata and withheld its body left a download task and its connection behind on every elapsed bound; the fetch and its connection end together.
• Images above the patch budget of the MoonViT-based processors were one 8-bit level off the checkpoints' Pillow bicubic on part of their pixels, a transparent image further off through the premultiplication; they resize through the reference's kernel and reach the engine byte-identical.
• A video family's sampler read a processor config that was not the one its processor is built from, and a long clip sampled at a rate whose stride flips between two values was refused for the length of its frame list; the sampler reads the processor's own config and hands the decoder one short expression per clip.

Engine servicers and start-up

• The Rust SGLang and TokenSpeed servicers reported zeros for each rank's utilization and the aggregate's throughput, reported every abort as a success, and did not notice a frozen scheduler; the Rust TokenSpeed encode worker pushed a zero-filled load record per connected engine; the Python SGLang servicer did not complete a request when a scheduler abort arrived on the tokenizer socket. All fixed.
• A direct-ZMQ disconnect left over from a previous engine generation closed the link admitted after a gateway restart, so the worker failed until an operator restarted the engine. A tcp:// worker URL with a bare IPv6 host was accepted with an address no engine can dial. A lockstep data-parallel group that parked between a request's add and its wake was declared dead after the silence window.
• The planned warm-up ran the decode staircase on a prefill side that never decodes, where an engine release gave no stand-in's slots back and bring-up failed; a client's own request id could spell its way into the readiness hold's warm-up.
• SmgNoHealthyWorker stayed silent when a scraped gateway had no worker series; the chart's error-ratio alert counted the kubelet's probe polls; router.policy refused policies the launcher takes; the Python Router and smg launch started with a bad bucket list or role mapping the binary refuses.
• A Rust servicer treated its engine as dead during a long prefill that published KV events and no output; the engine-silence bound counts from the engine's last sign of progress, an output or a KV-event batch its relay read live.
• The TokenSpeed servicer turned KV-event publishing off for hybrid models, and its engine metrics flags never reached the engine; it keeps publishing on when the engine publishes per cache group, and --servicer-metrics-port / --servicer-metrics-host configure the servicer's own endpoint.
• The launcher pinned a ZMQ worker to the SGLang wire without --backend, defaulted --eviction-interval-secs to 60 s where the CLI, the Python Router and the chart use 120 s, and the Python Router started with defaults that differed from the CLI's; IGW mode lost the start-up worker URLs; the Python binding's tokenizer pool was handed out before every worker ran. All follow the CLI now.
• The chart's gRPC worker startup probe asked a port the servicer does not serve health on, so a pod counted as started before its engine served; it asks the servicer's gRPC health service. The chart accepted the regular Slurm job names and router.workerUrls under a disaggregation mode, where such workers cannot reach a disaggregated gateway; it refuses them.
• An SGLang engine's accepted speculative tokens were missing from usage on the gRPC path when the engine sent no proposal counts, and a continuous multi-choice stream reported its rejected counts before they were final; the accepted count is kept and the rejected count is reported once it is known.

Upgrade notes

Storage

No schema migration and no change to the stored shapes in this release.

Configuration that now refuses to start

Each of these used to start, or start wrongly, and now exits with the setting named, before any port binds:

• --least-load-max-waiting-requests, and a least_load policy configuration that still sets max_waiting_requests: the refusal names --worker-overload-waiting-requests and --worker-overload-shed.
• A malformed --mesh-peer-urls entry in any position (every entry is dialed in turn until one answers).
• On the Python Router and smg launch: a Prometheus duration bucket list that is not finite, positive and strictly ascending; a JWT role mapping to a gateway role other than admin or user, or from an empty IdP role.
• On the engine side: a vLLM worker of the Qwen2-VL or Qwen3-VL line, or the GLM-5 family, given a launch-flag video rule its own loader reads (fps, max_frames, min_frames, temporal_patch_size, video_backend), since the servicer's media pipeline cannot follow it; a loader named through the environment alone keeps letting the family start, and a family on the engine's default loader keeps starting under the kwargs that loader never reads. A launch the Rust servicer does not serve is refused by name now that it is the default; --servicer-impl python keeps it running.
• A discovery selector entry without key=value on the CLI and the launcher; an encode policy outside random, round_robin and consistent_hashing on the CLI, the launcher and the Python Router; in the chart, the Slurm role job names without router.disaggregation.mode, and router.workerUrls or the regular Slurm jobNames under a mode.

Changed defaults

• Engine servicer: --servicer-impl defaults to rust for vLLM, SGLang and TokenSpeed (SMG_<ENGINE>_SERVICER_IMPL=python keeps the Python servicer, deprecated there and logged); TensorRT-LLM keeps python; the chart's worker takes servicerImpl, and smg serve --servicer-impl follows the package's default.
• Load streaming: every gRPC worker streams its load records to the gateway under any policy; least_load and power_of_two price a worker (and each rank of a data-parallel engine) by its current record, and poll only an engine that does not stream.
• Per-request media counts: the gateway's own model specs no longer bound the images, videos or audio clips of the Qwen2-VL, Qwen3-VL, Qwen3-Omni, Qwen3-ASR, Inkling, Muse Spark, Kimi K3, Gemma 4, Nemotron Omni and DeepSeek V4.1 families; the engine's advertised limit, --mm-per-request-image-limit / SMG_IMAGE_MAX_COUNT, SMG_VIDEO_MAX_COUNT and SMG_AUDIO_MAX_COUNT bound them.
• Video: the qwen_vl family's checkpoints accept one clip per request when their config names a video token, sampled by the loader rule (total / fps x 2 frames in whole temporal groups, at most 768); a clip too long for the decode budget is refused rather than thinned; Qwen3-Omni video tokens carry the processor's two seconds a patch.
• Multimodal configuration: a Hub failure other than a 404 (a 5xx, a 429, a transport failure, an elapsed 30 s bound) answers the request 502 multimodal_config_unavailable and the next request resolves the checkpoint again.
• Encode placement: a prompt of up to sixteen kibibytes of rendered text is tokenized on the request's thread; larger prompts go to the blocking pool as before.
• Cache-aware trees: a node's tenant and children maps use two shards instead of four or eight; above the holders cap a node with more than eight tenants presents the pressure selection a different eight (the pick followed map order before as well).
• Mesh: one connection per peer and direction; a heartbeat is answered once, by the accepting end, and the dialer sends one every 15 s; the info level logs one line per membership change.
• Logs: the Rust vLLM servicer logs the engine's periodic KV transfer line per rank and, under RUST_LOG=info,engine_servicer::request=debug, the Python servicer's per-request line; the SGLang launcher's refusals name the servicer; a prefill side of several ranks warns once per connection about a room residue overriding the pin.
• Chart: router.mmTurbojpeg defaults to require (a custom router image without libjpeg-turbo sets prefer); a gRPC worker's startup probe asks the servicer's gRPC health service, and the engine's load and warm-up count against the startup budget (360 s by default: a slower worker needs a larger startupProbe).
• Launcher and Python Router: --eviction-interval-secs defaults to 120 s in the launcher (the cache-aware tree eviction and the idle-key sweeps of the manual policy and the routing-key override follow it; pass 60 to keep the old cadence); smg_rs.Router(...) starts cache-aware routing on port 30000 and waits 1800 s for a worker (pass policy, port or worker_startup_timeout_secs to keep the old values); a ZMQ worker is not pinned to the SGLang wire without --backend (the gateway detects the wire; SGLang schedulers need --backend sglang).
• Control plane: CONTROL_PLANE_API_KEYS splits its entries at , and at line ends (one entry reads as before); --admin-max-connections keeps no bound unless set; --secrets-auto-reload is off unless set (a changed key file is read on POST /secrets/reload or at the next start).
• Operations: GET /engine_metrics scrapes HTTP workers only; the Kubernetes discovery's earlier info lines about a membership change print at debug, replaced by the one Member <name> is <Status> line.

API behavior

• Prompts render as the engines render them on these paths, so prompt token counts and prefix-cache keys change once for the affected requests: DeepSeek V3.2 and V4 developer messages, a conversation ending in a system message, the V3.2 tools block, the V4 reasoning-effort prompt and chat_template_kwargs.enable_thinking; Kimi K2 tool declarations; Kimi-K3.1 through the K3 XTML encoder; a tool without parameters ("parameters": null in the prompt and on Responses, no key on Chat Completions); a tokenizer.json truncation or padding block no longer cuts a prompt.
• Completions: a prompt given as token ids, or a batch of them, bypasses the tokenizer and reaches the engine as sent (a mixed, negative or empty list is refused with the element at fault, an id past the vocabulary before any engine is spoken to); logprobs returns the object on every choice.
• Chat: n takes 1 to 128; logit_bias reaches the engine (a bias or key outside the contract is a 400; TensorRT-LLM refuses the field); the nested allowed_tools tool_choice is accepted and written back to relayed upstreams; an assistant refusal part and a function message with content: null are accepted; every chunk before the usage chunk carries "usage": null under stream_options.include_usage; cross-field validation refusals drop their internal field-set prefix and name their field in param.
• Messages: output_config.format of type json_schema constrains the answer and another type or a non-object schema is refused by name; thinking: {"type": "between_tools"} is thinking off and another field beside type is refused; message_start carries the prompt's token count; an unreadable content block is refused by position and reason; top_p: 0.0, a top_k of 0 and a top_k above 2147483647 are 400s; the usage carries cache_read_input_tokens; the response carries request-id; the model entry carries lifecycle and the card's token limits, and GET /v1/models pages by limit, after_id and before_id.
• Responses: frequency_penalty, presence_penalty, top_k, min_p, repetition_penalty and stop reach the engine on every turn; a turn whose last tool call was cut by anything but the token limit reads status: incomplete on every path and in the stored object, a cut custom tool call too, and a Harmony turn truncated by max_output_tokens reads incomplete_details.reason: max_output_tokens; the stored object of a Harmony stream keeps its text item; every answer carries x-smg-parser.
• Tool calls under Symphony: a call the parser could not read whole (refused bytes, a value nested past the reader's bound, a cut no strict reader takes) keeps an unclosed argument prefix on chat, reads incomplete on Responses and keeps {} on Messages; a DeepSeek call without its calls block is a call; a call written bare and again inside a block is two calls and the model's text.
• Generate: the gRPC router's plain stream carries the typed finish_reason object; a chunk the tokenizer cannot decode ends the stream with an error event; input_ids past the embedding table (or unknown to the tokenizer where no table size is reported) are refused with a 400 naming the id.
• Streams: a stream that stops inside a multi-byte character ends without a replacement character for the unfinished one, and otherwise releases the held replacement character and any complete character held with it, as a full decode would.
• GET /v1/models sends a caller's own provider credential to the external providers of the API family the request speaks, no longer to every configured provider.
• Multimodal refusals: multimodal_config_unavailable (502) for a Hub failure; an audio clip with an impossible header rate gets the undecodable-clip error at the header; a clip too long for the decode budget, or whose frame list the decoder cannot be given, is refused with the picks and the bound named; a Step3-VL image whose square pad would exceed the token bound is refused; .ico with an RGB or greyscale PNG entry decodes, AVIF stays refused by name.
• Tenancy: the tenant rate limiter also holds requests through the disaggregated HTTP router and the external provider routers, with the same 429 and Retry-After.
• Control plane: POST /secrets/reload (admin) re-reads the serving, tenant and control-plane key files and reports serving_changed; GET /config (admin, with control-plane authentication configured) answers the effective configuration with every secret redacted; GET /workers?format=manifest answers the fleet as a file discovery manifest, filtered by ?origin=local or ?origin=discovery; POST /wasm answers the registered module's uuid; the WASM and tokenizer management routes refuse in the {"error", "code"} envelope (a 409 names the holder's id, the 404s carry a code, 503 without the module manager); the control-plane routes appear in the generated API document with their bearer scheme.
• Harmony and efforts: gpt-oss on the gRPC Harmony path renders a leading system or developer message as the developer message's instructions and writes no empty developer message; the Harmony chat path returns only the requested top_logprobs, none when the field is absent; a GLM-5.3 request's minimal, medium and xhigh efforts are read as low, high and max.
• Tokenizer: a control-token spelling typed into a request stays text on a checkpoint that ships the family's own vocabulary file; a tekken.json vocabulary loads ahead of the tokenizer.json beside it, with the file's own BOS on request.
• Refusal messages: an unsupported encode policy names the encode_policy field, from the binary and the Python binding alike.

Metrics, logs and traces

• Changed: smg_http_connections_active follows the in-flight count on every release and is set from the tracker's count at the sampler's tick; smg_router_tokens_total counts the generate endpoint's prompt and cached tokens, streamed and collected; the gRPC pipeline counts a retried request once in the router request, error and duration families (the attempts stay in the retry families); the priority scheduler's clamp and unknown-priority counters keep at most priority_scheduler_tenant_metric_top_n tenants as tenant series and count the rest under other; the engine gauges of a pushed load record are set for the record's own rank.
• Chart: the Grafana dashboard gains a "Prefill-decode" row over the fifteen smg_pd_* families; SmgHighRouterErrorRatio (critical) is new; SmgNoHealthyWorker fires on a gateway with no worker series; the admission alert and panels read the priority scheduler's families; the error-ratio alert and the status panels exclude the kubelet's probe polls; the rules carry promtool unit tests under tests/rules.
• Logs: the mesh logs one line per membership change at the info level; a mesh member with a local ZMQ worker no longer logs a refused tombstone every thirty seconds; a failed Hub fetch of a configuration file is logged with the file name; the Rust vLLM servicer's KV transfer line and per-request debug line (above); a frame selection that fails is logged as a warning before the rate fallback runs; smg --help lists the three worker start-up bounds under Worker Configuration.
• New: smg_mm_inflight_bytes (the preprocessed media bytes in flight, with or without a budget); the pool_queue stage of the multimodal processing histogram (the wait for a blocking-pool thread); smg_kv_index_arena_chunk_bytes and smg_kv_index_cap_refusals; a Symphony reading that is not clean marks the request span (table, outcome, reason, bytes, finish) and its output's log line, bounded per table and reason; every control-plane mutation writes an outcome audit line after its admission line.
• Corrected: the gRPC router's latency metrics are anchored at the RPC dispatch (and the /metrics descriptions of the first-token and generation families say so: their clock starts at the answering attempt's RPC dispatch), so the time to first token and the generation duration read right on workers that hold their response headers until the first chunk; the HTTP routers count a request that failed after all its retries in smg_router_request_errors_total with the final answer's class and reason (a client's own upload fault on the streamed-body route does not count there).

Engine side

• Upgrade order: roll gateways first, then engine pods. A v1.14.0 servicer keeps working with the new gateway and is polled for its load; the streamed load records (loads_only), the measured prefill capacity on SGLang and TokenSpeed, the frozen-scheduler probe, the per-rank GetLoads fields, the audio limit in the advertised media limits and the TensorRT-LLM load figures need smg-grpc-servicer 0.16.0, which in Rust mode needs the smg 1.15.0 wheel beside it.
• The default: --servicer-impl rust on vLLM, SGLang and TokenSpeed; the Python servicer logs its deprecation when chosen and goes two minor releases after this one; TensorRT-LLM keeps the Python servicer as its launcher's default.
• Behaviour: SubscribeKvEvents takes loads_only; GetLoads on the Rust vLLM servicer answers the rank the request names; the Rust SGLang and TokenSpeed servicers report utilization, avg_throughput and avg_utilization, measure prefill_throughput, probe the scheduler on HealthCheck and answer Abort for an unknown id with success: false; the TensorRT-LLM servicer declares GetLoads, reports token-work and throughput, and its refusal of a full KV-event subscription names the missing publisher endpoint; the readiness hold's warm-up ids carry a per-process nonce; a prefill side warms up with one stand-in request; a TokenSpeed encode worker answers GetLoads with no scheduler rank and pushes no record; the ZMQ wire bounds queued and active input payload bytes; vLLM encode/prefill/decode placements and handoffs over gRPC are validated at start; the Python SGLang servicer completes a request on a scheduler abort from the tokenizer socket; the SGLang worker reuses media for mixed image and video references and restores its request metrics for gRPC serving. The servicer forwards the cache-group mask of KV events from an engine that reports each boundary once per group, and the TokenSpeed servicer keeps KV-event publishing on for a hybrid model whose engine publishes per cache group; the TokenSpeed engine metrics flags reach the engine, and --servicer-metrics-port / --servicer-metrics-host configure the Rust servicer's own endpoint; the engine-silence bound counts a KV-event batch published through a prefill as a sign of life; the Rust vLLM servicer advertises the engine's encoder-cache connector and carries the encoder-cache transfer parameters both ways, and a vLLM encode/prefill/decode fleet distributes one request's images across its encoder workers by the encoder policy.

Bindings and Rust crates

• Rust 1.98 is needed to build from source.
• Routing policies: the policy trait's completion hook no longer takes a success flag that was always true; a policy outside the tree drops the parameter (the request outcome stays on the worker's circuit breaker).
• Python: smg.tokenizer.Tokenizer (encode, decode, render_chat, apply_chat_template, picklable, fork-safe); Router and smg launch take --backend gemini and trtllm, and refuse a bad Prometheus bucket list or JWT role mapping as the binary does.
• Library crates with changed signatures: llm-tokenizer 2.0.0 (a major step): the three DeepSeek renderers raise one DsEncodingError, re-exported under each renderer's path with every message unchanged, so a match over one renderer's error sees the other dialects' variants on the same type; llm-multimodal 1.16.0 takes &[&DynamicImage] in VisionPreProcessor::preprocess, preprocess_with_context and preprocess_wire, and &[&DecodedAudio] in Qwen3AudioProcessor::preprocess_decoded_clips; smg-symphony 0.3.0, openai-protocol 1.18.0, engine-servicer 0.4.0, engine-zmq-client 0.5.0, engine-zmq-adapter 0.3.0, smg-grpc-client 1.17.0, kv-index 2.1.0, smg-auth 1.3.0, data-connector 2.5.0, reasoning-parser 1.11.0 and tool-parser 1.11.0 step a minor for the features above; smg-mcp 2.3.7 (rmcp 3.9), smg-mesh 1.4.6, smg-external-router 0.4.1 and smg-rl 0.3.1 step a patch.
• Go: not published as a module release this time; the Go bindings build from the tag.

Packages and engines

• smg 1.15.0 on PyPI and crates.io, with the container image ghcr.io/smg-project/smg:1.15.0 (also lightseekorg/smg:1.15.0 on Docker Hub, and latest) and Helm chart 1.15.0. Release engine images build on vLLM 0.31.0 (1.15.0-vllm-v0.31.0; also published on the 0.27.1 and 0.26.0 bases), SGLang 0.5.21 (1.15.0-sglang-v0.5.21), TensorRT-LLM 1.3.0rc24 (1.15.0-trtllm-1.3.0rc24; also rc23 and rc22) and the latest TokenSpeed nightly (tag 1.15.0-tokenspeed- followed by the nightly version the build resolved).
• smg-grpc-servicer 0.16.0 and smg-grpc-proto 0.5.0 (the proto files added since v1.14.0 ship in it, and the servicer's floor requires it). Roll gateways first, then engine pods: the Rust servicer default, the streamed load records and the servicer fixes above come from the new servicer, which in Rust mode needs the smg 1.15.0 wheel beside it.
• Library crates at the tag: openai-protocol 1.18.0, llm-tokenizer 2.0.0, llm-multimodal 1.16.0, smg-grpc-client 1.17.0, kv-index 2.1.0, smg-symphony 0.3.0, reasoning-parser 1.11.0, tool-parser 1.11.0, engine-servicer 0.4.0, engine-zmq-client 0.5.0, engine-zmq-adapter 0.3.0, smg-auth 1.3.0, data-connector 2.5.0, smg-mcp 2.3.7, smg-external-router 0.4.1, smg-rl 0.3.1, smg-mesh 1.4.6, smg-response-template 0.3.0 (it follows smg-symphony, whose types its public API carries); unchanged since v1.14.0 and not republished: smg-radix-tree 0.1.1, smg-wasm 1.1.5, wfaas 1.1.1, smg-client 0.1.4, smg-mm-rdma 0.1.1.
• The Go module is not released this time; the Go bindings build from the tag.
• Building from source needs Rust 1.98.

Known limits

• Twelve legacy parser names have no Symphony table and keep the older parsers under the default: step3, sarashina, cohere and cohere_cmd (no table), pythonic (the generic name; llama4_pythonic names the Llama 4 table exactly), kimi (the ◁think▷ markers), and the reasoning names base, glm45, nano_v3, qwen3_thinking and deepseek_r1, which alone select nothing and leave the legacy parsers in place.
• The Inkling table writes no forced-call tag for a prompt whose thought is open at the first token; such a forced call runs unconstrained.
• TensorRT-LLM keeps the Python servicer as its default. Its Rust servicer is opt-in per launch; Embed and SubscribeKvEvents (beyond loads_only) answer UNIMPLEMENTED there, as the engine's own servicer does, and beam search, the logits-returning flags, KV-cache retention, prompt tuning and lookahead configs are refused by name.
• SGLang Rust servicer: the encoder-only role of encode/prefill disaggregation runs the engine's own entrypoint, the encoders take images only (video and audio references are refused), and a prefill side of plain data-parallel replicas stays refused.
• TokenSpeed Rust servicer: multimodal items of MRoPE families other than the Qwen3.5 lineage are refused with a message naming the gap, and the encode role stays on the Python servicer. Greedy answers through the two TokenSpeed servicers can still differ late at near-ties, for a reason outside the request's sampling fields.
• The direct ZMQ lane serves a TokenSpeed engine's roles, control calls and media only when the engine declares them at its handshake; an engine that declares nothing reads as a regular text worker. Its tcp:// sockets carry no authentication.
• Hybrid models whose engine publishes no per-group KV-cache events are routed on the gateway's own approximate tree; cache-aware routing follows a hybrid model's cache groups only where the engine reports each boundary once per group (TokenSpeed does when it publishes per cache group).
• Video at the gateway covers the qwen_vl family's checkpoints, Qwen3-Omni, the interval-sampled families and the loaders named above, one clip per request; a family whose loader rule the pipeline cannot follow refuses the worker rather than guess.
• Slurm discovery needs a router image with the Slurm clients; the published image has none.
• The Go module is not published for this release.
• An engine core that freezes while holding a single pending request is not yet excluded from routing.

Thanks to everyone who contributed, and welcome @HyeonjeCho, @fparisio, @gty111, @jcvx93, @liuyiru-hz, @ibrahimois and @bhargavchintam!

Full changelog

What's Changed

754 commits since v1.14.0 on main at the cut (the merged pull requests, by type):

Features

• feat(gateway-perf): add a prefill-decode fleet to the cost benchmark (#5588)
• feat(bindings): pass the key files to the gateway for the reload (#5576)
• feat(gateway): watch the key files and reload the keys on a change (#5540)
• feat(review): the bots announce themselves when they start and request changes on Important findings (#5572)
• feat(gateway): bound the admin listener with --admin-max-connections (#5521)
• feat(cache-aware): follow a hybrid model's KV cache groups (#5404)
• feat(helm): mount the rate-limit and scheduler YAML from ConfigMaps (#5541)
• feat(multimodal): gauge the preprocessed media bytes in flight (#5526)
• feat(multimodal): record the preprocessing pool's queue time (#5509)
• feat(gateway): reload the keys from their files on POST /secrets/reload (#5489)
• feat(helm): render the admin listener and control-plane keys (#5499)
• feat(rate-limit): replace the tenant policy on the live buckets (#5418)
• feat(gateway): group the routes gated by control-plane authentication (#5465)
• feat(gateway): resident-token decode placement for power_of_two (#5467)
• feat(vllm): fan out images within EPD requests (#5447)
• feat(tokenizer): a tekken.json vocabulary loads, ahead of the tokenizer.json beside it (#5250)
• feat(helm): render the disaggregation roles from typed values (#5456)
• feat(gateway): add GET /config with every secret redacted (#5383)
• feat(helm): require the native JPEG decoder by default (#5453)
• feat(kv-index): the arena's held bytes join the index gauges beside the bytes it handed out (#5403)
• feat(mock-worker): add prefill and decode roles on the HTTP wire (#5420)
• feat(gateway): mark the request span with a Symphony reading's outcome (#5395)
• feat(openapi): list every control-plane route with its bearer scheme (#5378)
• feat(cli): read the control-plane API keys from a file (#5390)
• feat(servicer): decode the group mask of a per-boundary KV event feed (#5397)
• feat(gateway-perf): a paced generate path whose text passes the frame bound on every event (#5356)
• feat(servicer): fill the encoder-cache fields of the vLLM wire (#5387)
• feat(mock-worker): add an encode role on the vLLM wire (#5366)
• feat(gateway): add the manifest format to GET /workers (#5377)
• feat(auth): write an outcome audit line after a control-plane mutation (#5375)
• feat(multimodal): a Step3-VL image spec, the global view and its crops (#5302)
• feat(mock-worker): publish a hybrid engine's KV cache groups (#5338)
• feat(gateway-perf): a spaced-token generate path, the stream loops at an engine's pace (#5310)
• feat(vllm): support gRPC EPD routing (#5230)
• feat(mock-worker): answer the flush and profile fan-out on every wire (#5197)
• feat(multimodal): the qwen_vl registry spec serves video for a checkpoint whose config names a video token (#5226)
• feat(helm): add a prefill-decode row to the dashboard (#5204)
• feat(gateway-perf): a completion stream path and the load driver's completions mode (#5199)
• feat(tokenizer): render Kimi-K3.1 chat prompts with the K3 XTML encoder (#5123)
• feat(servicer): measured prefill capacity from the SGLang and TokenSpeed servicers (#5081)
• feat(grpc): the TensorRT-LLM service reports engine load (#5079)
• feat(bindings): [5/5] Tokenizer.apply_chat_template takes transformers' shape and refuses what SMG would not do (#4952)
• feat(bindings): [4/5] os.fork() waits for the Python tokenizer's calls in flight, and the tokenizer pickles with a fingerprint (#4951)
• feat(routing): use measured vLLM prefill capacity (#5055)
• feat(bindings): [3/5] Tokenizer.render_chat renders a chat request through the gateway's own request path (#4950)
• feat(servicer): the Rust gRPC servicer is the default for vLLM, SGLang and TokenSpeed; an explicit Python choice logs its deprecation (#4401)
• feat(release): the notes render the project's template, seeded by the pull requests' release notes (#4979)
• feat(bindings): [2/5] smg.tokenizer.Tokenizer encodes and decodes with the tokenizer SMG serves with (#4949)
• feat(release): release.sh ship cuts a candidate or a final in one resumable run, from the bump to the publish dispatches (#4977)
• feat(worker): every gRPC worker streams its load records to the worker monitor; the poll is the fallback (#4971)
• feat(grpc): a loads-only mode of the KV-event subscription streams every rank's load record (#4969)
• feat(sglang): support mixed image and video references (#4944)
• feat(symphony): DeepSeek-V4-Pro reads the V4-Flash group's table (#4906)
• feat(mock-worker): the gRPC workers take a prefill or decode role, the prefill side binding its bootstrap port (#4885)
• feat(mock-worker): a scripted answer can be filed for the first prompt that holds a text (#4886)
• feat(mock-worker): a gRPC worker's model info carries the model's sampling defaults (#4862)
• feat(mock-worker): a request that samples several choices takes one scripted answer per choice (#4863)
• feat(release): the trunk is next: the cut moves main to the next development version (#4853)
• feat(mock-worker): a scripted answer carries top-logprob alternatives (#4852)
• feat(bench): the gateway's cost per request is measured by a Rust crate (#4808)
• feat(helm): a router.parser value picks the gateway's parser; empty keeps the image's default (#4825)
• feat(http-router): the regular HTTP router holds its typed routes to the tenant rate limit (#4776)
• feat(mock-worker): a scripted answer stops at the request's token limit (#4802)
• feat(release): make release-plan reads the bump the next cut writes, quietly (#4755)
• feat(release): the gate reads crates.io and PyPI for versions published already (#4787)
• feat(sglang): add worker media reuse for GLM and MiniMax (#4721)

Fixes, performance, documentation and maintenance

The 208 fixes a v1.14.0 user could meet are told under Notable fixes and the 155 performance changes in the Highlights; they, 26 documentation changes and 294 maintenance changes (tests, refactors, dependency updates, CI and build) are listed in the full changelog.