smg-project/smg v1.8.0 on GitHub

🚀 Shepherd Model Gateway v1.8.0 Released

Our biggest multimodal release yet: EPD (Encode-Prefill-Decode) disaggregation, engine-agnostic tensor transport over SHM and RDMA, and audio support (Qwen3 ASR + Omni) — plus dedicated multimodal + audio support for Thinking Machines' Inkling model, per-tenant API keys, LoRA adapter management, and a second nightly correctness harness.

⚠️ Breaking change: The legacy smg server command has been removed. Use smg launch or smg serve instead. Update your deployment scripts before upgrading.

🧩 EPD: Encode-Prefill-Decode Disaggregation

Multimodal serving now disaggregates the encode stage from prefill and decode.

Just as PD disaggregation separates prefill from decode, EPD adds a third independently-scheduled stage — encode — for multimodal inputs. The vision/audio encoder runs on its own workers, so image, video, and audio preprocessing no longer competes with the prefill/decode path:

Dedicated encode routing — Multimodal inputs routed to encode workers, encoded tensors forwarded to prefill
Unified Mode-parameterized router — Regular, PD, and EPD collapsed into a single router with a shared EncodeStage, rather than three separate code paths
Python CLI + Helm exposure — Configure EPD directly from the CLI
EPD smoke CI — 4-GPU H100 multimodal coverage in the nightly matrix

Impact: Encode-heavy workloads (lots of images or video) scale independently. You can provision encode capacity separately from prefill/decode, and a burst of multimodal requests won't starve your text generation throughput.

🚚 Engine-Agnostic Tensor Transport: SHM + RDMA

Encoded multimodal tensors now move between workers over a configurable, engine-neutral transport layer.

Moving preprocessed tensors from encode workers to prefill workers efficiently is the hard part of EPD. This release generalizes that transport across engines and interconnects:

Configurable tensor transport — Engine-agnostic interface, not hardwired per backend
vLLM SHM transport — Shared-memory tensor handoff for same-node encode→prefill
RDMA pixel transport — New mm_rdma crate for cross-node tensor movement, routed through a shared payload resolver
Shared proto handles — ShmHandle and RemoteTensorHandle hoisted to common.proto for reuse across backends
Centralized backend×modality capability check — One place that knows which engine supports which modality over which transport

Impact: Multimodal disaggregation works the same way whether tensors move over shared memory on one node or RDMA across nodes, and the same transport layer serves vLLM, TokenSpeed, and future backends.

🎙️ Audio Support: Qwen3 ASR + Omni

SMG now handles audio — the modality lineup is text, image, video, and audio.

Qwen3 ASR — Automatic speech recognition
Qwen3 Omni — Full multimodal (text + vision + audio)
Audio-processor selection on ModelProcessorSpec — Per-model audio preprocessing
Verified correct — Full-tensor parity test for Qwen3 audio log-Mel features against HuggingFace transformers

Impact: Speech and omni-modal models run through the same gateway, the same routing, and the same APIs as everything else — with audio preprocessing validated bit-for-bit against the reference.

🧠 Dedicated Support: Thinking Machines' Inkling

SMG adds first-class, end-to-end support for Inkling — Thinking Machines' multimodal model — including its full modality set.

End-to-end integration — Complete serving-path support, from request handling through response assembly
Multimodality (image) — Image input support, with multimodal placeholders aligned to the Inkling checkpoint template
Audio — Audio input support, built on the same audio pipeline landing in this release

Impact: Inkling runs through SMG with dedicated handling across text, image, and audio — the same routing, caching, tokenization, and API surface as every other supported model, purpose-fit to Inkling's checkpoint. Thinking Machines models get a first-class home on SMG.

🎨 Multimodal Performance (contributed by NVIDIA)

Encoder and preprocessing optimizations from @yechank-nvidia:

Reduced TokenSpeed encoder input transport overhead
Optimized Qwen vision preprocessing
Optimized OpenCV video decoding and thread allocation

Impact: Lower latency across vision and video workloads. Thanks again to @yechank-nvidia for continued multimodal contributions.

✅ Correctness: Second Nightly Harness (tau2-bench)

Building on the v1.7.0 BFCL parity work, we've added a second correctness gate.

tau2-bench nightly A/B — Parser verification against pure vLLM, complementing BFCL
Alternating schedule — BFCL runs Mon/Wed/Fri, tau2-bench runs Tue/Thu/Sat
Hardened runs — Fail-fast timeouts and quieter logs

Impact: Two independent nightly harnesses now verify SMG's Rust-native tokenization, tool-call parsing, and reasoning parsing stay faithful to the reference. We're closing correctness gaps continuously, not once — and now double-checking with two different benchmarks.

🔐 Per-Tenant API Keys

Serving-path authentication now supports per-tenant API keys:

Issue distinct API keys per tenant
Authenticate serving requests against tenant-scoped keys

Impact: Multi-tenant deployments get proper key isolation on the serving path — each tenant authenticates with its own credentials.

🔌 LoRA Adapter Management

Dynamic LoRA adapter RPCs exposed in the SGLang scheduler proto:

Load, unload, and list LoRA adapters at runtime
No restart required to swap adapters

Impact: Serve and rotate fine-tuned LoRA adapters on the fly — load a new adapter, list what's active, unload what's stale, all through gRPC.

🔎 Local Model Discovery

Discover local HTTP model IDs from /v1/models — Auto-populate model IDs from HTTP-mode workers instead of manual configuration

🧠 Thinking / Reasoning Resolution

Cleaner, layer-unified control over reasoning activation:

reasoning_effort "none"/"minimal" → enable_thinking=false — Honor OpenAI-style effort hints on the gRPC chat path
Unified thinking resolution by layer — One consistent resolution path instead of scattered logic
Kimi K2.6 reasoning parser fix — Corrected kimi_k25 → kimi_thinking parser mapping

🤖 New Parser Support

Sarashina — New tool-call parser

🎧 Realtime API on HTTP Router

Realtime API support added to the HTTP router.

🐛 Notable Fixes

HTTP: Set application/json Content-Type for PD non-streaming responses
PD: Only treat line-anchored data: [DONE] as stream terminator, route chat using full message history
Worker: Query engine-specific HTTP load endpoint for vLLM/SGLang
Tool parser: Treat qwen_xml tool-call arg values literally
Data connector: Decode Postgres JSON columns as Value, not String
gRPC: GetTokenizer RPC implemented for TokenSpeed backend

📦 Dependencies

nixl-sys 0.10.1→1.3.1, str0m 0.20→0.21, lz4_flex 0.13→0.14, wasm-encoder 0.252→0.253

🏗️ Infrastructure

TokenSpeed engine image — New release workflow for TokenSpeed Docker images
Project governance docs — Added governance documentation
CI fixes for SGLang e2e (protobuf gencode), MLX (transformers 5.13), and TokenSpeed (cutlass-dsl pin)

🙏 Welcome New Contributors

5 first-time contributors landed in this release — thank you all!

Full Changelog: v1.7.0...v1.8.0

Upgrade now: pip install smg --upgrade

🐑 Any modality. Any engine. Production-proven.

What's Changed

fix(http): set application/json Content-Type for PD non-streaming responses by @chengxumiaodaren in #1857
chore(deps): update str0m requirement from 0.20 to 0.21 by @dependabot[bot] in #1859
feat(tau2): nightly tau2-bench A/B for parser verification by @key4ng in #1864
ci: alternate BFCL (Mon/Wed/Fri) and tau2 (Tue/Thu/Sat) nightlies by @key4ng in #1866
fix(ci): kimi-k2.6 SMG reasoning parser (kimi_k25 → kimi_thinking) by @key4ng in #1868
feat(realtime): support the Realtime API on the HTTP router by @slin1237 in #1870
fix(ci): pin nvidia-cutlass-dsl 4.5.2 for TokenSpeed (uv + build subprocess pip) by @key4ng in #1869
fix(tau2): harden nightly — fail-fast timeouts + silence cost-lookup log spam by @key4ng in #1872
fix(worker): query engine-specific HTTP load endpoint for vLLM/SGLang by @XinyueZhang369 in #1867
fix(pd): only treat line-anchored data: [DONE] as stream terminator by @junliu-mde in #1878
fix(ci): unbreak sglang/e2e (protobuf gencode) and MLX (transformers 5.13) jobs by @key4ng in #1880
fix(grpc/chat): honor reasoning_effort "none"/"minimal" as enable_thinking=false by @qywu in #1876
refactor(grpc/chat): unify thinking resolution by layer by @slin1237 in #1882
feat(discovery): discover local HTTP model ids from /v1/models by @SpencerGarnets in #1793
perf(multimodal): reduce TokenSpeed encoder input transport overhead by @yechank-nvidia in #1879
perf(multimodal): Optimize Qwen vision preprocessing by @yechank-nvidia in #1875
perf(multimdoal): Optimize OpenCV video decoding and thread allocation by @yechank-nvidia in #1865
docs: add project governance documentation by @slin1237 in #1883
feat(multimodal): add EPD encode routing by @chenht2022 in #1852
fix(import): Fix TokenSpeed KV events import by @lightseek-bot in #1884
chore(deps): update wasm-encoder requirement from 0.252 to 0.253 by @dependabot[bot] in #1886
fix(multimodal): expose EPD routing in Python CLI by @chenht2022 in #1889
refactor(multimodal): split grpc multimodal.rs into a module directory by @slin1237 in #1890
refactor(proto): hoist ShmHandle/RemoteTensorHandle to common.proto by @slin1237 in #1891
feat(multimodal): configurable engine-agnostic tensor transport by @slin1237 in #1892
feat(multimodal): vLLM SHM tensor transport by @slin1237 in #1893
feat(multimodal): optimize EPD encode routing by @chenht2022 in #1853
feat(multimodal): vLLM video via shared Modality enum by @slin1237 in #1895
chore(deps): Remove legacy smg server command by @lightseek-bot in #1897
fix(tool_parser): treat qwen_xml tool-call arg values literally by @key4ng in #1899
fix(k8s): qualify agentic service image references by @key4ng in #1900
feat(grpc): expose LoRA adapter load/unload/list RPCs in the SGLang scheduler proto by @yuanzh89 in #1901
feat(multimodal): support Qwen3 ASR and Omni by @chenht2022 in #1905
refactor(multimodal): extract engine-neutral RDMA pixel transport into crates/mm_rdma by @slin1237 in #1908
fix(multimodal): address Qwen3 review findings by @lightseek-bot in #1907
refactor(multimodal): move audio-processor selection onto ModelProcessorSpec by @slin1237 in #1913
refactor(grpc): centralize backend×modality capability check by @slin1237 in #1914
refactor(multimodal): route RDMA through the shared tensor payload resolver by @slin1237 in #1915
test(multimodal): full-tensor parity for Qwen3 audio log-Mel vs transformers by @slin1237 in #1910
refactor(grpc): unify regular/PD/EPD into one Mode-parameterized router + EncodeStage by @slin1237 in #1923
chore(deps): update nixl-sys requirement from 0.10.1 to 1.3.1 by @dependabot[bot] in #1922
feat(e2e): EPD multimodal smoke CI on 4-gpu-h100 by @key4ng in #1924
chore(deps): update lz4_flex requirement from 0.13 to 0.14 by @dependabot[bot] in #1919
ci(docker): add TokenSpeed engine image release workflow by @key4ng in #1927
feat(inkling): add end-to-end SMG support by @lightseek-bot in #1926
fix(pd): route chat using full message history by @Moersity in #1929
fix(inkling): align multimodal placeholders with checkpoint template by @chenht2022 in #1931
feat(grpc): implement GetTokenizer RPC for TokenSpeed backend by @key4ng in #1934
fix(e2e): wait for gateway readiness by @ChetanXpro in #1936
feat(auth): add per-tenant API keys for serving-path authentication by @XinyueZhang369 in #1932
fix(data_connector): decode Postgres JSON columns as Value, not String by @XinyueZhang369 in #1935
feat(tool_parser): add Sarashina tool-call parser by @Juno13340 in #1937
chore: bump versions for v1.8.0 release by @slin1237 in #1938

New Contributors

@chengxumiaodaren made their first contribution in #1857
@SpencerGarnets made their first contribution in #1793
@yuanzh89 made their first contribution in #1901
@ChetanXpro made their first contribution in #1936
@Juno13340 made their first contribution in #1937