llm-d/llm-d-router v1.0.0-rc.1

on GitHub

• fix(kvevents): drop the LoRA name from BlockStored extra keys by @wseaton in #2929
• deps(go): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.44.0 to 1.45.0 by @dependabot[bot] in #2916
• deps(go): bump the go-dependencies group across 1 directory with 5 updates by @dependabot[bot] in #2921
• feat(epp): complete W3C trace propagation for OpenTelemetry export by @dayakar349 in #2702
• fix: correct the Kubernetes development EPP config path by @yankay in #2940
• feat(sidecar): expose sglang /generate in sidecar proxy and coordinator by @sagearc in #2525
• ci: let dependabot Action PRs pass DCO and release-note checks by @elevran in #2903
• deps(go): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.44.0 to 1.45.0 by @dependabot[bot] in #2917
• deps(actions): bump the github-actions group with 2 updates by @dependabot[bot] in #2957
• cleanup: rename chat_completions.go to dispatch.go by @roytman in #2952
• Type EPP port options as uint16 by @magic-peach in #2879
• feat(apix): graduate EPPConfig API to v1 by @zdtsw in #2923
• fix(chart): use liveness/readiness service names in EPP health probes by @Iceber in #2901
• feat: add deployment config and guide for multimodal EPP direct routing by @DeanKelly751 in #2694
• metrics: remove deprecated kvcache_* metric aliases by @804533125 in #2942
• fix(scheduling): omit never-scraped endpoints from metric scorers by @apurv-1 in #2848
• Clone attribute values on Put to isolate stored state by @magic-peach in #2878
• fix(chart): use native preStop sleep for distroless EPP by @yankay in #2961
• fix: close TOCTOU port-bind race in EPP and coordinator tests by @elevran in #2861
• epp: register a built-in metric mapping for the tokenspeed engine by @0z5a in #2939
• refactor: send top-level transfer params on /inference/v1/generate by @804533125 in #2855
• fix: wait for EPP registry before scraping e2e metrics by @apurv-1 in #2904
• fix: inject listeners so sidecar proxy tests do not rebind by @apurv-1 in #2905
• Add audio content block support to tokenizer for multimodal routing by @RishabhSaini in #2823
• Add tool_choice, continue_final_message and max_output_tokens signals to outlen-bucket classifier by @DolevAdas in #2727
• fix API types conversation by @roytman in #2935
• chore(tls): align the TLS flags and messages across all three components by @zdtsw in #2908
• Document the measured work-router composition by @nilig in #2891
• fix(deploy): use liveness/readiness service names in EPP health probes by @Iceber in #2900
• feat(request): drop stateful Responses fields the router cannot honor by @roytman in #2950
• fix(security): harden pr-rebase GitHub Actions workflow against untrusted checkouts by @elevran in #2986
• fix(security): harden dataplane plugins (URLs, paths, log fields, PRNG) by @elevran in #2985
• fix(security): set ReadHeaderTimeout on EPP pprof metrics server by @elevran in #2984
• fix(server): validate --pool-group and correct its help text by @ptimizeroracle in #2969
• fix(predictedlatency): correct transposed model name labels by @malamsyah in #2947
• perf(sidecar): skip usage rewrite parsing for frames without usage by @sudoalok in #2481
• Fix: Grant lease RBAC for explicitly enabled single-replica EPP leader election by @weizhoublue in #2970
• Record prefix-cache hit predicted for the selected endpoint by @asharkhan3101 in #2897
• feat(epp): add session-state-producer plugin by @paulzhn in #2531
• Honor per-endpoint rankIndex in file discovery by @hengguangcui11-cyber in #2953
• bench: exercise 220 KiB prompts through EPP by @0z5a in #2699
• test(epp): assert endpoint address in TestStart_RankIndex by @hengguangcui11-cyber in #3004
• fix(workflows): close untrusted-checkout gap in pr-rebase by @elevran in #3011
• chore(security): harden TLS config defaults in coordinator, sidecar proxy, and EPP server by @elevran in #2983
• fix(security): kvcache/prefix_match.go integer conversions by @elevran in #3013
• fix(security): single-issue prod cleanups (bounds, unsafe audit, false-positive rename) by @elevran in #3015
• test: real-fix security annotations (perm literals, env, listeners) by @elevran in #3016
• chore: drop wrong-linter annotation on type assertion in proxy.go by @elevran in #3017
• test: narrow nolint:all to specific linters in proxy test handlers by @elevran in #3018
• chore: convert false-positive nolint:gosec / remove now-stale directives by @elevran in #3020
• fix(security): coordinator request-handling hardening (SSRF / XSS / request-forgery) by @elevran in #3021
• feat(chart): add securityContext values for the EPP pod and containers by @olgsm in #2998
• fix: create the load-test marker without following a symlink by @apurv-1 in #3010
• fix(datalayer): record extract errors on the runtime endpoint and k8s notification dispatch paths by @ptimizeroracle in #3007
• feat: pod watch peer discovery plugin by @evacchi in #2501
• chore: clean defer Close()/Write() suppressions in proxy and connectors by @elevran in #3019
• fix(security): flowcontrol registry integer conversions by @elevran in #3014
• fix(ci): don't cancel in-progress lint runs on main pushes by @elevran in #3026
• fix(requestcontrol): stop writing default priority into stored objective by @Frapschen in #3023
• test: freeze the stable plugin config surface by @sudoalok in #2384
• Publish the builder image from a content hash tag by @hickeyma in #2799
• Annotate verified false-positive (mostly) G101 security scan alerts by @elevran in #3032
• Attach per platform SBOMs to release assets by @hickeyma in #2802
• ci: ignore HTML comments when parsing PR and issue bodies by @himanshu1573 in #3036
• Make multimodal cache affinity use optional token lengths by @guygir in #1397
• fix(datalayer): recover panicking extractors on the endpoint and notification dispatch loops by @ptimizeroracle in #3027
• fix(coordinator): bind listen address before TLS setup by @elevran in #3050
• feat(coordinator): add OpenTelemetry tracing by @PrateekKumar1709 in #2975
• fix: warn when refresh-metrics-interval exceeds metricsStalenessThreshold by @varad-ahirwadkar in #2645
• fix(epp): pass refresh interval in the stable config test call by @roytman in #3067
• deps(go): bump the kubernetes group across 1 directory with 6 updates by @dependabot[bot] in #3071
• deps(actions): bump the github-actions group across 1 directory with 3 updates by @dependabot[bot] in #3072
• feat(sidecar): support the /v1/responses API in the proxy by @roytman in #2987
• fix :preserve signed 64-bit hash representations in KV event adapters by @bermanamit in #3079
• Fix failing make presubmit issues by @anishasthana in #3074
• feat(sidecar): expose Prometheus /metrics on the P/D sidecar by @gyliu513 in #1677
• fix(scheduling): log ProcessResults success only when it returns no error by @satyamg1620 in #3095
• ci: separate Check Typos run groups by triggering event by @elevran in #3103
• chore: standardize gosec suppressions and close G104/codeql false-positives by @elevran in #3048
• routing: extract HasPreference from IsConditionalDecode by @revit13 in #3099
• Bump Dockerfile.builder tool versions to close CVEs by @elevran in #3047
• ci: exempt release-notes bot PRs from release-note check by @elevran in #3044
• feat(metrics): link metrics to traces with Prometheus exemplars by @sudoalok in #2774
• fix(scheduling): validate plugins in schedulingProfile by @zdtsw in #3102
• fix(epp): record encode-stage endpoint in multimodal encoder-cache by @acardace in #2627
• refactor: declare shared wire constants once by @roytman in #3082
• epp: tokenize /v1/responses requests via chat-completions render by @hengguangcui11-cyber in #2963
• deps: bump Gateway API/GAIE to v1.6.2, drop legacy CRD group by @elevran in #3086
• epp: hold flow-control queue under saturation via in-flight credit by @ruocco in #2066
• epp: keep malformed subset hints off the default candidate cache key by @LukeAVanDrie in #3128
• fix(epp): strip client-supplied routing headers before forwarding by @satyamg1620 in #3090
• feat(epp): remove deprecated GAIE Pod metadata fallbacks by @liu-cong in #3127
• feat(datalayer): confine per-request attribute access to declarations by @satyamg1620 in #2580
• docs: Update AGENTS.md: require value check before constant declaration by @roytman in #3107
• build: bump ENVTEST_K8S_VERSION to 1.36.2 by @elevran in #3114
• chore: fix lint and test build tags by @zdtsw in #3147
• feat(epp): estimate audio token footprint in token-producer by @sudoalok in #2886
• feat(coordinator): add pipeline support for the /v1/responses API by @roytman in #3130
• fix(sidecar): keep discarded decode-first headers out of the response by @hmdsefi in #3121
• fix(kvevents): clear learned group metadata when a pod retires by @Iceber in #2898
• Count the request's own tokens in the concurrency filter by @vMaroon in #3066
• fix(kvcache): keep evict activity from promoting LRU recency by @ptimizeroracle in #3005
• fix(epp): clean up mm-embeddings-cache-producer state on pod removal by @LukeAVanDrie in #3125
• ci: tolerate noreply author email on merge_group DCO check by @elevran in #3158
• fix(kvevent): adapt SGLang decoding for map payloads by @bermanamit in #2966
• kvevents: clear pod state on sequence reset without replay by @rishabhsai in #2854
• refactor(metrics): share disaggregation path label values by @satyamg1620 in #3113
• fix(epp): Add check for unpopulated load attributes for TTFT load gate by @d0w in #2652
• coordinator: share the connector and decode proxy code of the steps by @revit13 in #3101
• fix(sidecar): recover a streamed decode-first abort in shared storage by @hmdsefi in #3120
• Fix: Serial MoRI-IO 2P2D Requests Fail on Pod-Local Rank Validation by @weizhoublue in #2922
• Fix sidecar proxy test build after shared wire constants move by @ahg-g in #3162
• Record prefix-cache prediction for the prefill endpoint under P/D by @satyamg1620 in #3070
• feat(sidecar): warn at startup when SSRF protection is disabled by @satyamg1620 in #3100
• Fix KV event hash integer conversion by @Alex-ai-future in #2601
• fix(sidecar): record stage metrics in the P2P connector by @hmdsefi in #3093
• fix(epp): attribute MM prefix matches across multi-prompt requests by @RishabhSaini in #3163
• fix(test): fix flaky parallel test in stage-aware saturation gating by @shwetha-s-poojary in #3148
• Add render metrics and failure log to token-producer by @huhongyu20040201-personal in #3161
• coordinator: share the gateway send code of the prefill and encode steps by @revit13 in #3055
• test(coordinator): parallelize the e2e suite by @revit13 in #2302
• chore: Bump Go to 1.27.1 and golangci-lint to v2.13.2 by @asharkhan3101 in #3169
• refactor(flowcontrol): add BandSelectionPolicy extension point by @loicmarchal in #2920
• feat: Add selective KV transfer policy by @albertoperdomo2 in #2833
• fix(kvcache): count admissions/evictions only on successful index writes by @ptimizeroracle in #3062
• fix(disagg): do not count rejected prefill requests as decode-only by @satyamg1620 in #3092
• fix(epp): log a failed request at error level once by @satyamg1620 in #3177
• perf(openai-parser): detect chat content format before decoding by @asharkhan3101 in #3167
• fix(sidecar): send error responses as application/json by @satyamg1620 in #3172
• [Bug][Charts] preserve explicit false for metricsDataSource.insecureSkipVerify by @vmanvs in #3182
• fix(epp): default approx-prefix blockSizeTokens to the 64-token floor by @satyamg1620 in #3175
• feat(epp): promote core plugins to stable by @liu-cong in #3160
• Add EPP leader-election lease flags and drain on lease loss by @vMaroon in #3064
• epp: report multimodal features for /v1/responses requests by @hengguangcui11-cyber in #3009
• Add missing strings import in tokenizer estimate tests by @ahg-g in #3191
• sidecar: correlate SGLang P/D requests with traces and logs by @zetxqx in #3189
• Parse only listed metric families in the metrics data source by @vMaroon in #3035
• Count dispatched requests in saturation before PreRequest by @vMaroon in #3065
• feat(kvevents): warm subscribers from vLLM KV-event snapshots by @vMaroon in #2946
• chore: exclude test paths from the gosec security gate by @marzmesas in #3117
• Track multimodal encoder cache capacity and active references by @Levius-Fubuki in #3033
• fix(flowcontrol): label flow_control_stale_endpoints by pipeline stage by @hengguangcui11-cyber in #3045
• perf: Add EPP tracing overhead performance study by @albertoperdomo2 in #3068
• feat(epp): Graduate InferenceObjective to v1 and allow targeting multiple pools by @RishabhSaini in #2925
• feat(observability): add fairness identity trace attribution by @madhugoutham in #2843
• feat(coordinator) rename coordinator headers to the x-llm-d- prefix by @roytman in #3200
• feat(api): serve llm-d.ai/v1 InferenceObjective, deprecate v1alpha2 by @RishabhSaini in #3206
• feat(sidecar): record stage metrics on the shared-storage path by @satyamg1620 in #3094
• perf(kvevents): decode vLLM events with typed MessagePack fields by @nt591 in #3131
• eviction: fix cleanup race in TestRequestEvictor_CtxCancellationTriggersCleanup by @ahg-g in #3212
• datalayer/extractor: reject non-finite scraped metric values by @ahg-g in #3201
• Remove support for the inference.networking.x-k8s.io pool group by @ahg-g in #3195
• Feature/issue 1349 epp autoscaling by @ishwar170695 in #3133
• kvcache: remove deprecated KVBlockScorer surface by @ahg-g in #3198
• Fix flaky TestUnmarshalEnvelopeStringAllocations by @ahg-g in #3197
• fix(epp): serialize rejections as JSON error envelopes by @blacksoxx in #3207
• feat(predictedlatency): include selected prediction in training entries by @madhugoutham in #2383
• tokenizer: record render start time before arming timeout context by @ahg-g in #3209
• fix(chart): gate metrics auth RBAC on prometheus.auth.enabled by @cne27 in #3211
• chore: use the shared content-type constants across EPP and coordinator by @blacksoxx in #3225
• feat(charts): add Horizontal Pod Autoscaler support for standalone proxy service by @liu-cong in #3216
• Add issue triage automation by @kaushikmitr in #3138
• fix(sidecar): enforce target ports in the SSRF allowlist by @LukeAVanDrie in #3126
• predictedlatency: add prediction failure counter and rate-limited error log by @ahg-g in #3199
• ci: add workflow to label and notify PRs with merge conflicts by @liu-cong in #3226
• scheduling: add SchedulingResult.PrimaryEndpoint and use it by @revit13 in #3232
• test(epp): opt hermetic configs out of default metrics-source injection by @elevran in #3115
• chore(epp): enable OpenMetrics through controller-runtime HandlerOpts by @madhugoutham in #3228
• feat(kvcache): batch Index.Evict to replace per-key eviction loops by @Iceber in #2673
• perf(epp): share per-invocation state across scoped endpoints by @cne27 in #3237
• feat(sidecar) deprecate disaggregation headers to use the x-llm-d- prefix by @roytman in #3208
• fix(sidecar): recover encoder fanout response aborts by @Iceber in #3244
• Disable GC during TestUnmarshalEnvelopeStringAllocations by @ahg-g in #3248
• fix(datalayer): read queue, running requests and KV use across a pod's engines by @mtparet in #3085
• deps(go): bump the go-dependencies group across 1 directory with 8 updates by @dependabot[bot] in #3250
• feat(metrics): scrape vLLM NIXL failure counters per endpoint by @satyamg1620 in #3108
• perf(approximateprefix): index blocks per pod and match by galloping search by @LukeAVanDrie in #3241
• perf(prefixhash): hash prompt blocks without a streaming digest per block by @LukeAVanDrie in #3243
• fix(sidecar): bind EC handlers to each data-parallel rank by @Iceber in #3249
• cleanup(semconv): remove orphaned KVBlockScorer attributes by @ahg-g in #3253
• fix(security): resolve CodeQL alerts on main (unsafe-quoting, insecure-tls) by @elevran in #3259
• Report the best prefix-cache hit the router could have chosen by @asharkhan3101 in #3057
• Add coordinator support for topology-aware P/D routing by @elevran in #2319
• fix(sidecar): record streaming chunked decode stage errors by @ahg-g in #3251
• [ci] Clear production gosec findings and turn on at the gate by @hickeyma in #3261
• Add EPP HPA and envoy service autoscaling benchmark and verification suite by @jtechapps in #3186
• feat(sglanghttp): accept text prompts in SGLang /generate requests by @Madhumasa84 in #3262
• fix(epp): correct /v1/messages usage counts and model rewrite by @roytman in #3252
• feat(metrics): add multimodal prefix-cache prediction metrics by @RishabhSaini in #3171
• docs(sglanghttp): update parser doc comment for text prompt support by @Madhumasa84 in #3263
• feat(coordinator): add pipeline amplification and media metrics by @zhouyou9505 in #2616
• security: bump Go to 1.27.2, golangci-lint to v2.14.0, and x/net to v0.60.0 by @ahg-g in #3277
• feat(epp): add a session manager used by thunder agent by @zetxqx in #2968
• feat(metrics): label prefix-cache prediction metrics by modality by @RishabhSaini in #3164
• feat(epp): emit mm.matched_blocks/mm.total_blocks span attributes by @RishabhSaini in #3272
• Add vllm.omitMMKwargs to drop multimodal tensors from render responses by @vMaroon in #3273
• epp: add endpoint-pin-screener to pin a request to one endpoint by @revit13 in #3145
• epp: promote priority-holdback-policy to beta by @LukeAVanDrie in #3284
• docs: link EPP autoscaling to fail-open and service mode by @canvascloudai in #3282
• test: add Mooncake trace replay benchmarks for the prefix indexers by @LukeAVanDrie in #3291
• perf(kvblock): hash CBOR-FNV blocks without marshaling by @LukeAVanDrie in #3286
• docs: fix MD links the upstream llm-d router docs by @roytman in #3321
• vllmhttp-parser returns HTTP 400 for generate requests with trailing slashes by @weizhoublue in #3300
• epp: key per-request state by internal request ID by @cjackal in #3309
• fix(coordinator): count asyncbroker error responses in error metrics by @shreyaabaranwal in #3320
• docs: assemble release notes for v0.11.0 by @llm-d-router-release-notes[bot] in #3043
• feat(chart): expose command and args for the latency predictor containers by @olgsm in #3001
• fix(kvevents): reject conflicting HMA group block sizes by @0z5a in #2693
• Fix(helm): standalone proxy sidecar preStop delay by @weizhoublue in #3331
• Keep IPv6 addresses intact in endpoint subsets by @bojiang-li in #3305
• Add Code of Conduct document by @davidgs in #3260
• feature(CrossReplicaSyncer): add Redis syncer by @Lucas-Fernandes-Martins in #2837
• fix(coordinator): forward untouched request fields as raw JSON by @himanshu1573 in #3141
• Fix(helm): misindented standalone proxy lifecycle YAML by @weizhoublue in #3303
• perf(kvblock): peek walk keys in batches under one read lock by @LukeAVanDrie in #3288
• perf(kvcache): fold keys that repeat the previous key's entries in one step by @LukeAVanDrie in #3318
• Remove deprecated pd-sidecar vllm-port configuration by @0z5a in #3112
• Honor SGLang's cache_salt field in KV events by @bermanamit in #3219
• docs: document P/D sidecar metrics and tracing by @ahg-g in #3356
• feat(epp):p2p cost scorer by @kaushikmitr in #3008
• fix(kvcache): do not fail an entire Redis Lookup on one key's error by @hengguangcui11-cyber in #2977
• feat(epp): thunder-agent scorer, keep a session on the pod that holds its KV by @zetxqx in #3052
• fix(kvevents): skip sub-canonical stores with an unresolved parent by @ahg-g in #3255
• fix(epp/preciseprefixcache): reject estimate-backed token-producer by @gkneighb in #1529
• fix(kvblock): early-stop lookup on missing key or empty filtered pods by @quangh33 in #2713
• perf(sglanghttp): keep input_ids raw when decoding the payload map by @cne27 in #3333
• fix: add render sidecar to simulator deployment by @Phil-OSophy-42 in #2888
• fix(sidecar): continue one assistant message across decode chunks by @Iceber in #3302
• test: bump the e2e simulator pin to v0.12.0 by @roytman in #3317
• test(flowcontrol): deflake TestZombieCapacityStarvation by @iit2009046 in #3367
• fix(epp): isolate endpoint subset cache keys by @bojiang-li in #3335
• fix(sidecar): forward echo requests without chunked decode by @Iceber in #3307
• [Bug][async-broker] Keep async-broker error results when delivery fails by @vmanvs in #3270
• Fix: Keep Flow Control running until EPP finishes draining on SIGTERM by @weizhoublue in #3311
• fix(sidecar): treat cache_threshold on any choice as a prefill fallback by @hmdsefi in #3267
• epp: count model server errors reported in the :status header by @tadler10 in #3330
• fix(sidecar): handle tombstones in allowlist informer deletion handlers by @Iceber in #3280
• fix(kvevents): accept retained history during cold replay by @bojiang-li in #3338
• fix(sidecar): canonicalize IPv6 P2P endpoint identities by @bojiang-li in #3337
• Add /hold and /hold cancel comment automation by @elevran in #3381
• ci: harden markdown link check by @zdtsw in #3377
• Read tools and reasoning effort for responses in outlenbucket, fix(epp) by @Madhumasa84 in #3315
• fix(epp): count Anthropic stream events by @Cindia-blue in #3295
• epp: remove deprecated data-parallel-profile-handler plugin by @ahg-g in #3395
• feat(epp): simplified thunder-agent per-pod admission gate in fairnessPolicy by @zetxqx in #3076
• coordinator: extend audio/video media support alongside images by @dmitripikus in #2659
• fix(sidecar): abort decode when prefill fails by @sagearc in #3364
• fix(sidecar): fix shared allowlist validator lifecycle by @Iceber in #3279
• add toolcalling telemetry to EPP by @Gregory-Pereira in #1949
• fix(sidecar): prime Anthropic image blocks on the encoder by @hmdsefi in #3266
• fix(tokenizer): count chat-completions tool_calls in the estimate backend by @ptimizeroracle in #3257
• feat(sidecar): treat an Anthropic cache_threshold stop reason as a prefill fallback by @hmdsefi in #3268
• fix(sidecar): forward requests with n > 1 without chunked decode by @hmdsefi in #3388
• fix(epp): use the pod's node name as the topology hostname fallback by @hmdsefi in #3386
• fix(epp): reject file-discovery entries without a unique name by @hmdsefi in #3390
• feat(epp): serve aggregated /v1/models from router by @zdtsw in #2308
• fix(kvevents): index live events when no replay base is available by @vMaroon in #3402
• fix(coordinator): answer every inference rejection with the OpenAI error envelope by @blacksoxx in #3393
• test(e2e): split helpers into scope-specific packages by @delete-cloud in #3042
• feat(metrics): add trace exemplars to the other EPP latency histograms by @sudoalok in #3350
• fix(kvevents): load the snapshot before replaying on connect by @vMaroon in #3401
• deps(go): bump github.com/fxamacker/cbor/v2 from 2.9.4 to 2.9.6 in the go-dependencies group by @dependabot[bot] in #3403
• deps(go): bump the kubernetes group with 7 updates by @dependabot[bot] in #3404
• deps(actions): bump the github-actions group with 2 updates by @dependabot[bot] in #3405
• docs(operations): clarify HA switchover trade-offs and add Helm snippets by @liu-cong in #3283
• docs(release): check the embedded llm-d-async producer version at release time by @shimib in #3408
• deps: bump llm-d-async api and producer to v0.11.0 by @shimib in #3406
• fix(sidecar): forward upstream headers on shared-storage error responses by @Iceber in #3342
• feat(metrics): add trace exemplar to flow control queue duration by @sudoalok in #3420
• fix(epp): avoid nil endpoint and datastore panic by @zdtsw in #3416
• fix: Stop rejecting Anthropic content blocks whose source is not an image by @albertoperdomo2 in #3419
• test(e2e): cover a model version rollout with InferenceModelRewrite by @vjsairam in #3414
• Fix: sidecar requests bypassing PD disaggregation with trailing slashes by @weizhoublue in #3410
• fix(latency): preserve disabled weights during score normalization by @LysanderPhong in #3413
• fix(coordinator): add Responses and Messages metric route labels by @Phil-OSophy-42 in #3424
• fix(epp): preserve a typed Admitter denial's status code and drop reason by @damodaryekkuluri in #3369
• fix(sglanghttp): read and rewrite the model name by @saylee21 in #3375
• charts: add service mode for the latency predictor by @kaushikmitr in #3339
• fix(sidecar): cancel concurrent prefill when decode fails or finishes by @Iceber in #3412
• flowcontrol: add explicit domain to priority-holdback-policy by @ishwar170695 in #2026
• fix(prefixhash): separate model and cache salt in the block hash seed by @saylee21 in #3378
• tokenizer: cap max_output_tokens on the prefillOnly render copy by @satyamg1620 in #3238
• feat(renderer): experiment support for sglang generate requests rendereing by @sagearc in #3397
• docs: correct standalone fail-open note in EPP autoscaling section by @canvascloudai in #3427
• test: bump the simulator pin to v1.0.0-rc.2 by @mayabar in #3429
• standalone: route fail-open traffic to model servers when EPP is unreachable by @ahg-g in #3433
• ci: automatically enqueue approved PRs with passing checks by @ahg-g in #3275
• fix(coordinator): build every encode sub-request before sending any by @ahg-g in #3428
• Support active-active EPP load sharing in Helm charts by @Lucas-Fernandes-Martins in #3431
• perf(preciseprefixcache): reuse candidate predictions in PreRequest by @yankay in #3281
• test: bump the simulator pin to v1.0.0 by @mayabar in #3447
• fix ci error by @roytman in #3448
• feat(coordinator): add pipeline support for /v1/messages api. by @Mohammad-nassar10 in #3296
• Keep synthetic completion out of predicted latency token counts by @bojiang-li in #3299
• Reject nonfinite predicted latency SLO headers by @bojiang-li in #3436
• [ci] Run gosec in presubmit and after earlier lint failures by @hickeyma in #3355
• Send mm_payload with return_mm_kwargs on vLLM render copies by @vMaroon in #3455
• deps: bump llm-d-async api and producer to v1.0.0 by @shimib in #3460

New Contributors

• @wseaton made their first contribution in #2929
• @dayakar349 made their first contribution in #2702
• @DeanKelly751 made their first contribution in #2694
• @804533125 made their first contribution in #2942
• @malamsyah made their first contribution in #2947
• @paulzhn made their first contribution in #2531
• @hengguangcui11-cyber made their first contribution in #2953
• @Frapschen made their first contribution in #3023
• @PrateekKumar1709 made their first contribution in #2975
• @bermanamit made their first contribution in #3079
• @anishasthana made their first contribution in #3074
• @hmdsefi made their first contribution in #3121
• @rishabhsai made their first contribution in #2854
• @shwetha-s-poojary made their first contribution in #3148
• @huhongyu20040201-personal made their first contribution in #3161
• @vmanvs made their first contribution in #3182
• @Levius-Fubuki made their first contribution in #3033
• @blacksoxx made their first contribution in #3207
• @cne27 made their first contribution in #3211
• @canvascloudai made their first contribution in #3282
• @cjackal made their first contribution in #3309
• @shreyaabaranwal made their first contribution in #3320
• @bojiang-li made their first contribution in #3305
• @davidgs made their first contribution in #3260
• @iit2009046 made their first contribution in #3367
• @tadler10 made their first contribution in #3330
• @Cindia-blue made their first contribution in #3295
• @vjsairam made their first contribution in #3414
• @LysanderPhong made their first contribution in #3413
• @damodaryekkuluri made their first contribution in #3369
• @saylee21 made their first contribution in #3375