github llm-d/llm-d-router v0.11.0
on GitHub

latest releases: v1.0.0, v1.0.0-rc.2, v1.0.0-rc.1...
one month ago

Release v0.11.0

Highlights

  • Flow control for P/D: Saturation gating is stage-aware. Prefill and decode are measured separately and admission gates on the busier stage (#2221). The beta max-saturation-detector combines several detectors and gates on the highest one, and each child can be limited to prefill or decode (#2827, #2926). Queue TTLs can be set per priority band, clients can ask for a shorter one with x-llm-d-inference-ttl, and noEndpointRequestTTL bounds the wait while the pool is empty (#2649, #2284). New capacity utilization gauges and an x-llm-d-flow-queue-duration-ms response header (#2184, #2490).
  • Scheduling and routing: The new latency-observation-scorer ranks endpoints by TTFT measured from real responses (#2371). In P/D, load is split by role: input tokens count against prefill and estimated output tokens against decode (#2396). The context-length-aware scorer can route cached follow-ups by the prefill work that remains (#2559). Endpoints tied for the top score take turns instead of one pod getting the whole burst (#2515, #2709). New turn-priority fairness strategy and endpoint attribute weight scorer (#2116, #2204).
  • Tokenization and render: token-producer renders Anthropic Messages natively through vLLM /v1/messages/render, and falls back to the old conversion when the endpoint is missing (#2707). It can balance render calls across discovered endpoints (#2701), and supports HTTPS, mTLS and render endpoints started with --api-key (#2664, #2678, #2763). Opt-in xxhash block hashing makes precise prefix hashing about 8x cheaper (#2237).
  • API coverage: SGLang /generate (#2391), OpenAI text-to-speech (#2484), /v1/images/edits (#2562), and pre-tokenized batch prompts on /v1/completions (#2523). Paths no parser claims are forwarded to the model server by default through passthrough-parser (#2507). ATOM is a built-in engine type (#2476).
  • Coordinator and DisaggregatedSet: The coordinator exposes Prometheus metrics, optionally over HTTPS, forwards unregistered paths such as /v1/models to a decode pod, and adds an optional async-broker step (#2435, #2752, #2357, #2325). EPP replicas agree on DisaggregatedSet revision decisions, and the coordinator keeps the same decision across encode, prefill and decode (#2570, #2571, #2572, #2576).
  • Operations: A --feature-gates flag enables gates without restating the whole config (#2388). Helm adds Envoy priority routing for EPP failover in standalone mode, router.epp.deploymentStrategy, structured router.epp.pluginsConfig and router.imagePullSecrets, and restarts the EPP pod when its ConfigMaps change (#2228, #2603, #2620, #2766, #2460).
  • Observability: Logs carry trace_id and span_id. New spans cover tokenization and the KV-event pipeline, and trace context is read from ext_proc gRPC metadata so traces connect end to end (#2271, #2272, #2830). Spans carry service.name and the pod resource attributes (#2298).
  • Fixes: golang.org/x/crypto v0.55.0 for CVE-2026-56854 (#2739). Stale precise prefix-cache state after endpoints are removed (#2824). The PD Multi Tier decode leg starting before prefill stored its KV (#2451). min_tokens breaking the prefill leg (#2593). MoRI-IO concurrent dispatch (#2816). Published standalone charts pointing agentgateway at a missing image tag (#2732).
  • Precise prefix cache: LoRA requests keep BlockStored extra keys aligned with request-side keys, restoring precise cache matches (#2929).

Upgrade Steps & Deprecations

This release removes many items deprecated in v0.7.0 and v0.9.0. Configurations that still use them fail at startup.

  • Removed plugins: pd-profile-handler, use disagg-profile-handler with the nested profiles and deciders format (#2509). precise-prefix-cache-scorer, use precise-prefix-cache-producer plus prefix-cache-scorer with prefixMatchInfoProducerName (#2844). by-label and by-label-selector, use label-selector-filter or the role filters (#2865). tokenizer, use token-producer (#2869).
  • Removed config: The top-level saturationDetector and parser fields, use flowControl.saturationDetector and requestHandler.parsers (#2838). The UDS tokenizer: udsTokenizerConfig and indexerConfig.tokenizersPoolConfig are rejected, set vllm.url on token-producer instead (#2512). lasDecayFactor on program-aware-fairness, decay uses lasHalfLifeSeconds only (#2120).
  • Pod label: llm-d.ai/role: both no longer matches. Use prefill-decode (#2509).
  • Removed metrics: The legacy inference_extension_*, llm_d_inference_scheduler_*, core request and inference-pool, and inference_objective predicted-latency families. Use the llm_d_epp_* series. Three inference_extension_* series stay for the workload-variant autoscaler (#2826, #2675, #2821, #2509).
  • Changed metrics: KV-event metrics use the llm_d_epp prefix instead of llm_d_router_epp (#2385). kv_cache_index_lookup_hits_total and kv_cache_index_max_pod_hit_count_total count only the longest contiguous prefix on one pod, so they read lower. Re-baseline dashboards and alerts (#2400, #2680). An empty-pool expiry is labelled EvictedNoEndpoints instead of EvictedTTL (#2284).
  • Coordinator TLS on by default: The inference listener serves HTTPS. Use https, or set --secure-coordinator=false to keep plaintext (#2805).
  • JSON logs: Field names follow OpenTelemetry. Collectors that match on msg need to match body (#2643).
  • Tracing: With tracing on and OTEL_TRACES_EXPORTER unset, spans go to OTLP instead of stdout (#2566).
  • Helm: Charts default to Prometheus Operator monitoring for every gateway provider. GKE users who relied on implicit GMP monitoring must configure it explicitly (#2374). The Prometheus ClusterRole and binding are created only when metrics authentication is on (#2798).
  • Endpoint attribute plugins: endpoint-attribute-filter and endpoint-attribute-scorer take a separate producer parameter, and the combined Name/Producer form is rejected. Out-of-tree plugins that use AttributeMap need updating (#2190).
  • Prefix scoring: Device tiers without a configured weight score 0 instead of 1.0 (#2809).
  • Stricter startup checks: Optional dependency cycles and invalid plugin ordering fail startup (#2721). Non-positive --metrics-staleness-threshold and --refresh-prometheus-metrics-interval are rejected (#2914).
  • Conditional decode: The Prefer: if-available 412 gate moved into prefix-based-pd-decider. Without that plugin, such requests get 412. Set nonCachedTokens: 1 to keep the old behavior (#2335).
  • P/D failures: When a required prefill stage finds no endpoint, the request fails instead of running prefill on the decode pod (#2521). No eligible decode endpoint returns 503 rejected-no-endpoints (#2631).
  • Deprecated: dataLayer.discovery.pluginRef, use dataLayer.discovery.endpoints.pluginRef; old configs are migrated automatically (#2434). kvcache.KVBlockScorer and kvcache.NewTracedScorer (#2669).
  • Latency predictor: Charts from this release use the v0.8.0 latency predictor images, the latest published predictor release.

Known Issues

  • Endpoint score emission (--emit-endpoint-scores) emits nothing under the deprecated data-parallel-profile-handler.
  • With KV-event replay disabled, an inference engine that restarts at the same endpoint can retain stale precise prefix-cache state until eviction. Follow #2853 and #2854.

What's Changed

Click to expand
  • fix(kvevents): drop the LoRA name from BlockStored extra keys by @wseaton in #2929
  • test(flowcontrol): eviction dynamics benchmark by @LukeAVanDrie in #2107
  • refactor(datalayer): key AttributeMap by DataKey instead of string by @satyamg1620 in #2190
  • Filter prefix cache events by cache group kind by @vMaroon in #2363
  • docs: add release-note fragment for PR #2190 by @llm-d-router-release-notes[bot] in #2367
  • fix(charts): emit leading document separators in monitoring partials by @LukeAVanDrie in #2368
  • Bumped Go version directive from 1.26.5 to 1.26.6. by @ahg-g in #2386
  • Unify the metrics subsystem on llm_d_epp by @ahg-g in #2385
  • fix(flowcontrol): prevent aggregate stats underflow on flow GC by @LukeAVanDrie in #2382
  • docs: add release-note fragment for PR #2385 by @llm-d-router-release-notes[bot] in #2387
  • feat: EPP peer discovery infrastructure by @evacchi in #2341
  • make metricsLoggingInterval values match the comment by @yamt in #2405
  • fix(tokenizer): translate Anthropic tool calls and thinking for render by @mschulist in #2389
  • Bound and resume KV-event replay by @vMaroon in #2410
  • fix(options): validate refresh-metrics-interval 50ms floor at startup by @noalimoy in #2394
  • deps(actions): bump github/codeql-action/upload-sarif from 4.37.6 to 4.37.7 in the github-actions group by @dependabot[bot] in #2418
  • deps(go): bump google.golang.org/protobuf from 1.36.12-0.20260120151049-f2248ac996af to 1.36.12 in the go-dependencies group by @dependabot[bot] in #2417
  • docs: add release-note fragment for PR #2389 by @llm-d-router-release-notes[bot] in #2413
  • test(tokenizer): cover Anthropic tool render request by @vMaroon in #2412
  • perf(flowcontrol): index active queues so dispatch scans O(active) flows by @LukeAVanDrie in #2120
  • feat(flowcontrol): split queue-wait budget by unavailability regime by @satyamg1620 in #2284
  • docs: add release-note fragment for PR #2120 by @llm-d-router-release-notes[bot] in #2436
  • perf(epp): reduce streaming parser allocations by @muchengl in #2425
  • fix(epp): merge streaming usage per field instead of replacing it by @himanshu1573 in #2437
  • feat(epp): record a termination cause on end-of-stream response records by @LukeAVanDrie in #2444
  • docs: move Shmuel Kallner to emeritus, drop from CODEOWNERS by @elevran in #2449
  • [ci]: Add path based PR labeling by @hickeyma in #2281
  • feat(epp): expose a streamed-event count on the response record by @LukeAVanDrie in #2446
  • docs: assemble release notes for v0.10.0 by @llm-d-router-release-notes[bot] in #2424
  • docs: add release-note fragment for PR #2437 by @llm-d-router-release-notes[bot] in #2452
  • docs: add release-note fragment for PR #2444 by @llm-d-router-release-notes[bot] in #2453
  • docs: add release-note fragment for PR #2446 by @llm-d-router-release-notes[bot] in #2461
  • feat(flowcontrol): add per-band and global capacity utilization gauges by @sudoalok in #2184
  • feat(telemetry): align kvcache OTel scope and span names with router standard by @Alex-ai-future in #2164
  • feat(kvcache): add opt-in xxhash hashing to the KV-block token processor by @vMaroon in #2237
  • docs: add prose rules for model-drafted text to AGENTS.md by @LukeAVanDrie in #2420
  • Skip body re-marshal/replace when no model-name rewrite occurred by @elevran in #2318
  • feat(epp): add sglanghttp parser for /generate by @NaomiEisen in #2391
  • docs: add release-note fragment for PR #2391 by @llm-d-router-release-notes[bot] in #2467
  • docs: add release-note fragment for PR #2237 by @llm-d-router-release-notes[bot] in #2466
  • fix: Make TokenizedPrompt.MultiModalFeatures per-prompt by @albertoperdomo2 in #1633
  • refactor(epp): unexport RequestContext protocol and response state by @LukeAVanDrie in #2457
  • feat(datalayer): record more OpenAI fields on the models attribute by @zdtsw in #2479
  • fix(kvcache): report index hit metrics as contiguous prefix chains by @nilig in #2400
  • docs(epp): fix stale links in pkg/epp/README.md by @varad-ahirwadkar in #2454
  • [Flow Control] Stage-Aware Saturation Gating for disaggregated P/D deployments by @loicmarchal in #2221
  • perf(epp): remove per-scrape allocations in the metrics extractor by @LukeAVanDrie in #2473
  • cleanup(sidecar,logging): share the level encoder and use named verbosity constants by @LukeAVanDrie in #2474
  • docs: add release-note fragment for PR #2221 by @llm-d-router-release-notes[bot] in #2497
  • docs: add release-note fragment for PR #2400 by @llm-d-router-release-notes[bot] in #2496
  • fix(chart): avoid inferring GMP from GKE gateway provider by @Alex-ai-future in #2374
  • tracing: configure SDK explicitly instead of mutating process env by @satyamg1620 in #2298
  • fix(prefix-scorer): aggregate missing match logs by @nilig in #2404
  • docs: add release-note fragment for PR #2404 by @llm-d-router-release-notes[bot] in #2502
  • docs: add release-note fragment for PR #2298 by @llm-d-router-release-notes[bot] in #2499
  • docs: add release-note fragment for PR #2374 by @llm-d-router-release-notes[bot] in #2498
  • feat(scorer): migrate five legacy Consumes scorers to ConsumerPlugin by @elevran in #2355
  • fix(scheduling): return typed capacity rejection when filters drain candidates by @LukeAVanDrie in #2462
  • docs: add release-note fragment for PR #2184 by @llm-d-router-release-notes[bot] in #2464
  • docs: add release-note fragment for PR #2462 by @llm-d-router-release-notes[bot] in #2504
  • perf(epp): derive Scope allowed-key sets once at datalayer init by @LukeAVanDrie in #2471
  • perf(epp): guard hot-path log and span construction in scheduler, director, and handlers by @LukeAVanDrie in #2472
  • Add passthrough-parser to the default parser list by @zetxqx in #2507
  • docs: add release-note fragment for PR #2507 by @llm-d-router-release-notes[bot] in #2508
  • Add an outcome metric for the affinity filter plugin by @mathias-kende in #2494
  • feat(epp): move conditional-decode gate into prefix-based-pd-decider by @dmitripikus in #2335
  • Output length aware TokenLoadScorer by @DolevAdas in #2309
  • docs: add release-note fragment for PR #2494 by @llm-d-router-release-notes[bot] in #2526
  • feat(epp): add x-llm-d-flow-queue-duration-ms response header by @jtechapps in #2490
  • deps(actions): bump github/codeql-action/upload-sarif from 4.37.7 to 4.37.8 in the github-actions group by @dependabot[bot] in #2535
  • docs: add release-note fragment for PR #2490 by @llm-d-router-release-notes[bot] in #2538
  • docs: add release-note fragment for PR #2335 by @llm-d-router-release-notes[bot] in #2527
  • docs: add release-note fragment for PR #2309 by @llm-d-router-release-notes[bot] in #2528
  • deps(go): bump the go-dependencies group with 2 updates by @dependabot[bot] in #2532
  • deps(docker): bump golang from 1.26.6 to 1.26.7 by @dependabot[bot] in #2533
  • deps(go): bump the kubernetes group with 6 updates by @dependabot[bot] in #2534
  • epp: remove code deprecated since v0.7.0 by @roytman in #2509
  • docs: add release-note fragment for PR #2509 by @llm-d-router-release-notes[bot] in #2542
  • fix(epp): fail P/D requests when a required prefill stage finds no endpoint by @elinacse in #2521
  • fix(epp): classify non-OK gRPC trailers as errors by @yuanhuizang-cloud in #2517
  • Preserve JSON number precision during request repackaging by @yankay in #2519
  • feat(epp): add --feature-gates flag by @abhishekkarki in #2388
  • feat: Support array<array> prompt in /v1/completions by @albertoperdomo2 in #2523
  • docs: add release-note fragment for PR #2388 by @llm-d-router-release-notes[bot] in #2545
  • docs: add release-note fragment for PR #2164 by @llm-d-router-release-notes[bot] in #2465
  • docs: add release-note fragment for PR #2523 by @llm-d-router-release-notes[bot] in #2546
  • config: consolidate discovery plugins under a single discovery section (follow up #2341) by @pazqo in #2434
  • tracing: correlate logs with traces at request entry points by @satyamg1620 in #2271
  • feat: auto-tune approx prefix cache block size from prefix_match_unit by @RishabhSaini in #2455
  • refactor(flowcontrol): read capacity counters where they live by @LukeAVanDrie in #2456
  • docs: add release-note fragment for PR #2434 by @llm-d-router-release-notes[bot] in #2551
  • docs: add release-note fragment for PR #2271 by @llm-d-router-release-notes[bot] in #2552
  • feat(plugin-state): Allow customizing staleness threshold by @asharkhan3101 in #2138
  • test(e2e): warm up before disruption in streaming-request spec by @elevran in #2378
  • docs: add release-note fragment for PR #2284 by @llm-d-router-release-notes[bot] in #2439
  • tracing: support standard sampler values and validate the argument by @satyamg1620 in #2550
  • fix(dev): remove stale external tokenizer config by @yuanhuizang-cloud in #2491
  • Add endpoint-attribute parity coverage and a benchmark config (#2059 item 1) by @MicheleCampi in #2393
  • tracing: default the trace exporter to otlp instead of console by @satyamg1620 in #2566
  • feat(telemetry): finish dropping component prefixes from EPP scoring and P/D proxy spans by @gyliu513 in #2558
  • feat(telemetry): drop misleading gateway. prefix from EPP root spans by @gyliu513 in #2557
  • docs: add release-note fragment for PR #2455 by @llm-d-router-release-notes[bot] in #2553
  • feat: parse multipart /v1/images/edits requests in OpenAI parser by @zetxqx in #2562
  • docs: add release-note fragment for PR #2562 by @llm-d-router-release-notes[bot] in #2569
  • [KVEvents] add dp rank to KV events by @sagearc in #2379
  • docs: add release-note fragment for PR #2138 by @llm-d-router-release-notes[bot] in #2555
  • Raise e2e EXISTS_TIMEOUT to 90s in CI to avoid namespace teardown flakes by @zetxqx in #2582
  • feat(coordinator): proxy unregistered paths to the decode profile by @dmitripikus in #2357
  • feat(epp): make the fairness_id metric label cap configurable by @mathias-kende in #2495
  • pdd: Allow PD decider to select cache producer by @wez470 in #2560
  • docs: add release-note fragment for PR #2560 by @llm-d-router-release-notes[bot] in #2592
  • docs: add release-note fragment for PR #2357 by @llm-d-router-release-notes[bot] in #2583
  • docs: add release-note fragment for PR #2495 by @llm-d-router-release-notes[bot] in #2584
  • Fix Data race in TestFlowControlAdmissionController_Admit by @ahg-g in #2579
  • fix: prune inflight metric series on endpoint deletion by @MisterCool in #2577
  • refactor(flowcontrol): derive QueueOutcome from the finalization error by @LukeAVanDrie in #2458
  • Add prefixMatchInfoProducerName to inflightload README.md by @nt591 in #2599
  • feat(coordinator): expose Prometheus metrics by @dmitripikus in #2435
  • docs: add release-note fragment for PR #2435 by @llm-d-router-release-notes[bot] in #2615
  • [disagg-sidecar] Fix cached token usage for non-streaming P/D responses by @learner0810 in #2372
  • Add Bob as request lifecycle code owner by @vMaroon in #2613
  • Add Nili as P2P routing code owner by @vMaroon in #2614
  • check pluginType for defaults metrics source by @zetxqx in #2618
  • fix(epp): preserve endpoint attributes in data-parallel profile by @yuanhuizang-cloud in #2563
  • perf(epp): optimize HTTP token ID request parsing by @yankay in #2522
  • docs: add release-note fragment for PR #2372 by @llm-d-router-release-notes[bot] in #2624
  • Fix: Scope Endpoint Metric Cleanup by Namespace and Producer by @weizhoublue in #2602
  • fix(metrics): read SGLang cache capacity from dedicated gauges by @satyamg1620 in #2619
  • llm-d observability: align OpenTelemetry GenAI semconv and refactor telemetry usages by @PlateauGao in #2503
  • docs: add release-note fragment for PR #2458 by @llm-d-router-release-notes[bot] in #2597
  • deps(go): bump google.golang.org/grpc from 1.82.1 to 1.82.2 in the go-dependencies group by @dependabot[bot] in #2632
  • docs: add release-note fragment for PR #2619 by @llm-d-router-release-notes[bot] in #2635
  • feat(epp): add atomic cross-replica coordination API [Safe DisaggSet 1/4] by @hasB4K in #2570
  • fix(epp): use net.JoinHostPort for fallback random endpoints by @liuyuan10 in #2607
  • docs: add release-note fragment for PR #2570 by @llm-d-router-release-notes[bot] in #2646
  • docs: add release-note fragment for PR #2607 by @llm-d-router-release-notes[bot] in #2647
  • docs: document that endpoint discovery honours custom pod readinessGates by @almogtavor in #2604
  • deps(go): bump google.golang.org/grpc from 1.82.2 to 1.83.1 by @dependabot[bot] in #2660
  • fix: IPv6 address in remaining endpoint by @zdtsw in #2658
  • fix(epp): reject negative maxPrefixBlocksToMatch config by @Iceber in #2656
  • feat: add OpenTelemetry spans for tokenization and KV-cache event processing by @satyamg1620 in #2272
  • feat(epp): support OpenAI text-to-speech requests by @xiaojun-zhang in #2484
  • tracing: support OTLP/HTTP as an alternative to the gRPC exporter by @satyamg1620 in #2573
  • docs: add release-note fragment for PR #2272 by @llm-d-router-release-notes[bot] in #2662
  • Fix routing imbalance: deterministic tie-breaking for equal-score endpoints by @bbenshab in #2515
  • chore(epp): remove dead nil check and fix comment typo in approx-prefix by @Iceber in #2655
  • Enable Envoy priorities for high availability for standalone mode by @rlakhtakia in #2228
  • fix(chart): allow overriding the EPP Deployment strategy so multi-replica EPP can satisfy helm --wait by @almogtavor in #2603
  • docs: add release-note fragment for PR #2228 by @llm-d-router-release-notes[bot] in #2681
  • approximateprefix: change block hash insertion order to tail-first by @Iceber in #2661
  • docs: add release-note fragment for PR #2603 by @llm-d-router-release-notes[bot] in #2683
  • fix(kind): forward Prometheus host port by @Alex-ai-future in #2654
  • feat: add custom TLS support for vLLM render endpoint by @zdtsw in #2664
  • fix(epp): support vLLM render endpoints started with --api-key by @Iceber in #2678
  • docs: add release-note fragment for PR #2664 by @llm-d-router-release-notes[bot] in #2688
  • deps(actions): bump the github-actions group with 2 updates by @dependabot[bot] in #2634
  • fix(CrossReplicaSyncer): disentangle CrossReplicaSyncing interval from main ticket in PoolingDispatcher by @Lucas-Fernandes-Martins in #2310
  • docs(epp): add affinity filter composition requirements by @LukeAVanDrie in #2651
  • docs: add release-note fragment for PR #2678 by @llm-d-router-release-notes[bot] in #2689
  • docs: add a repository contributing guide covering issue claiming by @marzmesas in #2666
  • Fix(helm): Enforce Boolean Types for Priority Routing and Health Checking by @weizhoublue in #2686
  • feat(epp): order all requestcontrol hooks by data-dependency DAG by @chethanuk in #2359
  • refactor(kvcache): fold prefix matching into one accumulator by @nilig in #2669
  • Remove deprecated UDS/legacy tokenizer code and drop llm-d-kv-cache by @satyamg1620 in #2512
  • docs: use blockSizeTokens in tokenProcessorConfig samples (#641) by @chethanuk in #2705
  • fix(sidecar): await prefill before dispatching PD Multi Tier decode leg by @nilig in #2451
  • docs: add release-note fragment for PR #2451 by @llm-d-router-release-notes[bot] in #2711
  • docs: add release-note fragment for PR #2512 by @llm-d-router-release-notes[bot] in #2703
  • docs: add release-note fragment for PR #2669 by @llm-d-router-release-notes[bot] in #2700
  • docs: add release-note fragment for PR #2359 by @llm-d-router-release-notes[bot] in #2691
  • [ci]: Run required checks on merge_group events by @hickeyma in #2682
  • [ci]: Sign published container images with actions/attest by @hickeyma in #2285
  • Validate prefix cache affinity threshold bounds by @googs1025 in #2720
  • fix(epp): register default producer for UncachedRequestTokens by @ambermmxChen in #2407
  • docs: drop links to deleted IGW guide pages (#1863) by @chethanuk in #2704
  • Point sim-epp token-producer at the render Service by @tonibofarull in #2426
  • docs: add release-note fragment for PR #2407 by @llm-d-router-release-notes[bot] in #2728
  • ci(trivy): bump scanner to v0.74.0 by @elevran in #2718
  • Enforce ordering for configured optional dependencies by @vMaroon in #2721
  • Update Kubernetes libraries to 1.36 by @hasB4K in #2733
  • fix: omit unexecuted pipeline step timings by @learner0810 in #2726
  • bump golang.org/x/crypto@v0.53.0 to 0.55.0 due to a critical vulnerability by @roytman in #2739
  • test(sidecar): add audio and video multimodal test coverage by @dmitripikus in #2639
  • docs: add coordinator metrics specification by @revit13 in #2277
  • fix(request): strip min_tokens in single-token request priming by @roytman in #2593
  • deps(go): bump sigs.k8s.io/structured-merge-diff/v6 from 6.4.0 to 6.4.2 in the kubernetes group by @dependabot[bot] in #2736
  • deps(actions): bump crate-ci/typos from 1.50.0 to 1.50.1 in the github-actions group by @dependabot[bot] in #2737
  • deps(docker): bump golang from 1.26.7 to 1.26.8 by @dependabot[bot] in #2734
  • deps(go): bump the go-dependencies group across 1 directory with 3 updates by @dependabot[bot] in #2735
  • docs: add release-note fragment for PR #2593 by @llm-d-router-release-notes[bot] in #2744
  • docs: add release-note fragment for PR #2739 by @llm-d-router-release-notes[bot] in #2740
  • feat(epp): gate DisaggregatedSet readiness on initial Pod snapshot by @hasB4K in #2576
  • docs: add release-note fragment for PR #2576 by @llm-d-router-release-notes[bot] in #2745
  • ci: skip gosec SARIF upload on merge_group events by @elevran in #2751
  • feat(epp): coordinate DisaggregatedSet revision decisions [Safe DisaggSet 2/4] by @hasB4K in #2571
  • docs: add release-note fragment for PR #2571 by @llm-d-router-release-notes[bot] in #2756
  • feat(epp): add latency-observation scorer and producer by @Mohammad-nassar10 in #2371
  • ci: add DCO check that runs on merge_group by @elevran in #2750
  • docs: add release-note fragment for PR #2721 by @llm-d-router-release-notes[bot] in #2730
  • role-aware load split for P/D disaggregation by @DolevAdas in #2396
  • fix(config): auto-inject saturation detector as scheduling filter by @RishabhSaini in #1741
  • docs: add release-note fragment for PR #2396 by @llm-d-router-release-notes[bot] in #2759
  • docs: add release-note fragment for PR #2371 by @llm-d-router-release-notes[bot] in #2758
  • fix: quote lychee glob and exclude auth-walled URLs by @chethanuk in #2754
  • Enable redundant-import-alias lint rule, remove existing hits by @roytman in #2760
  • [ci]: Attach SBOM to published container images by @hickeyma in #2729
  • docs: describe standalone service proxy mode by @seanwestfall in #2748
  • fix(charts): roll the EPP pod when its ConfigMaps change by @shimib in #2460
  • feat(flowcontrol): support scoped request queue TTLs by @nt591 in #2649
  • feat(flowcontrol): make stale-metrics scoring configurable by @todayim in #2492
  • perf(kvcache): match prefixes over an ordered index walk by @nilig in #2680
  • fix(docs): repair broken links surfaced by quoted lychee glob by @Iceber in #2764
  • docs: add release-note fragment for PR #2460 by @llm-d-router-release-notes[bot] in #2768
  • docs: add release-note fragment for PR #2649 by @llm-d-router-release-notes[bot] in #2769
  • docs: add release-note fragment for PR #2492 by @llm-d-router-release-notes[bot] in #2770
  • docs: add release-note fragment for PR #2748 by @llm-d-router-release-notes[bot] in #2767
  • Add session affinity perf configurations and summary by @jtechapps in #2217
  • test(kvcache): release CostAware test resources by @Alex-ai-future in #2350
  • feat(epp): add OTel span to prefix cache affinity filter decisions by @sudoalok in #2594
  • fix(sidecar): forward untouched request fields as raw JSON to keep key order by @rebel-jinhwan in #2591
  • fix(lora-affinity): correct ActiveModels docs and remove wrong capacity by @zdtsw in #2625
  • feat: emit OpenTelemetry structured JSON logs to stdout by @den-rgb in #2643
  • docs: add release-note fragment for PR #2591 by @llm-d-router-release-notes[bot] in #2772
  • fix(cicd): move all lychee configuration to config file by @elevran in #2776
  • fix(epp): rotate only equal top-scoring endpoints in maxscore picker by @quangh33 in #2709
  • refactor(sidecar): extract shared concurrent prefill/decode dispatch helper by @roytman in #2589
  • docs: add release-note fragment for PR #2643 by @llm-d-router-release-notes[bot] in #2773
  • Fix agentgateway image tag defaulting to the release tag by @magic-peach in #2732
  • docs(metrics): clarify metric sources and Prometheus series by @zdtsw in #2747
  • test(epp): add coverage for InferencePoolToEndpointPool by @magic-peach in #2710
  • fix(kvevents): wait for subscriber goroutines to exit on shutdown by @quangh33 in #2717
  • Validate KV cache utilization threshold bounds by @googs1025 in #2753
  • chore(chart): introduce imagePullSecrets to pull from private registries by @PiePra in #2766
  • Rename completionRequest to body in sidecar proxy by @cyclinder in #2775
  • docs: add release-note fragment for PR #2732 by @llm-d-router-release-notes[bot] in #2781
  • docs: add release-note fragment for PR #2766 by @llm-d-router-release-notes[bot] in #2785
  • fix(docs): repair session affinity performance links by @zdtsw in #2782
  • test: validate CPU vLLM metrics for EPP by @pazqo in #2784
  • metrics: remove core legacy metric families by @JaredTan95 in #2675
  • fix: preserve API type through encoder disaggregation by @JaredTan95 in #2749
  • docs: add release-note fragment for PR #2675 by @llm-d-router-release-notes[bot] in #2791
  • feat(sidecar): add TLS security profile flags by @andresllh in #2676
  • docs(epp): update maxscore picker doc for round-robin tie-breaking by @quangh33 in #2789
  • feat(chart): Gate Prometheus RBAC on authentication by @learner0810 in #2798
  • test(epp): wait for metrics delivery in endpoint scores test by @Iceber in #2797
  • docs: add release-note fragment for PR #2676 by @llm-d-router-release-notes[bot] in #2795
  • feat(coordinator): async-broker step serving OpenAI compatible enqueue, wait, and passthrough modes by @BenjaminBraunDev in #2325
  • fix(scheduling): classify unavailable endpoints by @nicole-lihui in #2631
  • fix(kvcache): score unconfigured tiers zero and add storage tier weights by @Iceber in #2809
  • llm-d observability: finish remaining OpenTelemetry GenAI semconv and telemetry refactoring by @PlateauGao in #2516
  • docs: add release-note fragment for PR #2325 by @llm-d-router-release-notes[bot] in #2803
  • docs: add release-note fragment for PR #2631 by @llm-d-router-release-notes[bot] in #2804
  • docs: add release-note fragment for PR #2809 by @llm-d-router-release-notes[bot] in #2810
  • fix(perf): honor tracing configuration and report scheduler p99 by @0z5a in #2725
  • feat(datalayer): typed attribute slots and value-safety registry by @elevran in #2352
  • feat(chart): support vLLM Rust renderer flavor for tokenizer sidecar by @Madhumasa84 in #2657
  • feat(metrics): add ATOM as a built-in engine by @Jasen2201 in #2476
  • perf(picker): guard debug logging to avoid hot-path reflection by @mauriciopoppe in #2806
  • feature(epp): authenticate token-producer warmup probe via VLLM_API_KEY by @Iceber in #2763
  • ci: assemble release notes once per tag instead of per PR merge by @elevran in #2778
  • feat(helm): support structured plugin config by @hugosmoreira in #2620
  • docs: add release-note fragment for PR #2725 by @llm-d-router-release-notes[bot] in #2811
  • perf(tokenload): guard debug logging in TokenLoadScorer by @ishwar170695 in #2819
  • feat(sidecar+coordinator): add support for HTTPS /metrics by @zdtsw in #2752
  • docs: add release-note fragment for PR #2763 by @llm-d-router-release-notes[bot] in #2814
  • chore: remove deprecated disagg header handlers by @amirialy in #2815
  • metrics: remove predicted latency legacy metrics by @JaredTan95 in #2821
  • refactor(epp): add NewExtProcServerRunner and route every caller through it (#1370) by @chethanuk in #2706
  • fix(agentidentity): trim whitespace from extracted session headers by @ishwar170695 in #2818
  • epp: add turn-priority strategy to program-aware fairness by @ruocco in #2116
  • refactor: unify API type handling across sidecar and coordinator by @revit13 in #2743
  • Honor podLabelSelector for KV-event subscriptions by @vMaroon in #2445
  • apix: remove deprecated EPPConfig top-level saturationDetector and parser by @zdtsw in #2838
  • chore(epp): remove deprecated precise-prefix-cache-scorer plugin by @zdtsw in #2844
  • metrics: remove deprecated legacy metric series by @JaredTan95 in #2826
  • Broaden the link check exclusion pattern to cover the whole make documentation tree by @roytman in #2863
  • chore(epp): remove deprecated by-label filter plugins by @zdtsw in #2865
  • Pass native vLLM render requests through without content conversion by @vMaroon in #2707
  • test(flowcontrol): stop background GC racing hand-driven GC in band test by @heliubj18 in #2845
  • chore(epp): remove deprecated tokenizer plugin by @zdtsw in #2869
  • fix(kvevents): wait for old subscriber to exit on endpoint change by @quangh33 in #2719
  • feat(flowcontrol): fail open in concurrency detector filter when all endpoints are filtered out by @jtechapps in #2883
  • Add discovery-backed vLLM render balancing by @vMaroon in #2701
  • coordinator: remove unused "tokens" field from chat-completions requests by @roytman in #2842
  • fix(coordinator): log pipeline step names with their durations by @etai2404 in #2882
  • fix(kvevents): stop the replay test buffer from dying on a client hangup by @elevran in #2860
  • ci: retire release-notes.d fragment pipeline remnants by @elevran in #2868
  • fix(kvevents): validate kvEventsConfig to prevent panics by @Iceber in #2885
  • Document memory sizing for the in-memory KV index cap by @magic-peach in #2880
  • RHOAIENG-83718 | standardize llm-d OTel stdout logs by @den-rgb in #2813
  • chore: replace body-part terms in docs, comments, and code by @zdtsw in #2712
  • test(e2e): deploy router through the standalone Helm chart by @delete-cloud in #2832
  • fix(epp): extract trace context from gRPC metadata by @Phil-OSophy-42 in #2830
  • fix(kvevents): Fix memory leak in dedup map for precise KV cache index by @Lucas-Fernandes-Martins in #2824
  • feat(coordinator): share revision decisions across phases [Safe DisaggSet 4/4] by @hasB4K in #2572
  • Route cached follow-ups by remaining prefill work by @nilig in #2559
  • [doc] Add release verification guide for published images by @hickeyma in #2787
  • feat(coordinator): serve the inference listener over TLS by @zdtsw in #2805
  • chore: correct copyright attribution across migrated and llm-d-original files by @elevran in #2450
  • feat(scheduling): add endpoint attribute weight scorer by @nicole-lihui in #2204
  • ci: skip DCO and release-note checks for dependabot PRs by @elevran in #2856
  • Propagate EPP-resolved priority to native backend schedulers (SGLang/… by @sywangyi in #2265
  • fix(sidecar): send a pod-local DP rank on the MoRI-IO concurrent-dispatch path by @shikamd123 in #2816
  • fix: run the release pipeline once per tag by @apurv-1 in #2847
  • deps(actions): bump github/codeql-action/upload-sarif from 4.37.9 to 4.38.0 in the github-actions group across 1 directory by @dependabot[bot] in #2851
  • docs(dataproducer): explain ProducerPlugin vs DataProducer by @chethanuk in #2894
  • deps(go): bump go.opentelemetry.io/otel/sdk from 1.44.0 to 1.45.0 by @dependabot[bot] in #2915
  • fix(server): reject non-positive metrics timing flags by @ptimizeroracle in #2914
  • deps(go): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.44.0 to 1.45.0 by @dependabot[bot] in #2918
  • [Flow Control] Support Multiple Detectors by @robertgshaw2-redhat in #2827
  • feat(flowcontrol): scope max-saturation-detector children to stages by @vMaroon in #2926

New Contributors

  • @yamt made their first contribution in #2405
  • @muchengl made their first contribution in #2425
  • @himanshu1573 made their first contribution in #2437
  • @Alex-ai-future made their first contribution in #2164
  • @NaomiEisen made their first contribution in #2391
  • @DolevAdas made their first contribution in #2309
  • @yuanhuizang-cloud made their first contribution in #2517
  • @abhishekkarki made their first contribution in #2388
  • @pazqo made their first contribution in #2434
  • @wez470 made their first contribution in #2560
  • @MisterCool made their first contribution in #2577
  • @nt591 made their first contribution in #2599
  • @PlateauGao made their first contribution in #2503
  • @liuyuan10 made their first contribution in #2607
  • @almogtavor made their first contribution in #2604
  • @xiaojun-zhang made their first contribution in #2484
  • @bbenshab made their first contribution in #2515
  • @ambermmxChen made their first contribution in #2407
  • @tonibofarull made their first contribution in #2426
  • @Mohammad-nassar10 made their first contribution in #2371
  • @seanwestfall made their first contribution in #2748
  • @shimib made their first contribution in #2460
  • @todayim made their first contribution in #2492
  • @rebel-jinhwan made their first contribution in #2591
  • @den-rgb made their first contribution in #2643
  • @quangh33 made their first contribution in #2709
  • @magic-peach made their first contribution in #2732
  • @PiePra made their first contribution in #2766
  • @cyclinder made their first contribution in #2775
  • @JaredTan95 made their first contribution in #2675
  • @0z5a made their first contribution in #2725
  • @Jasen2201 made their first contribution in #2476
  • @mauriciopoppe made their first contribution in #2806
  • @hugosmoreira made their first contribution in #2620
  • @amirialy made their first contribution in #2815
  • @ruocco made their first contribution in #2116
  • @heliubj18 made their first contribution in #2845
  • @etai2404 made their first contribution in #2882
  • @delete-cloud made their first contribution in #2832
  • @sywangyi made their first contribution in #2265
  • @apurv-1 made their first contribution in #2847
  • @ptimizeroracle made their first contribution in #2914

Full Changelog: v0.10.0...v0.11.0

Check out latest releases or
releases around llm-d/llm-d-router v0.11.0

Don't miss a new llm-d-router release

NewReleases is sending notifications on new releases.

Get notifications