github llm-d/llm-d-router v0.7.0

latest releases: v1.0.0, v1.0.0-rc.2, v1.0.0-rc.1...
7 months ago

llm-d-inference-scheduler v0.7.0 Release

Docker images are available:

# EPP
docker pull ghcr.io/llm-d/llm-d-inference-scheduler:v0.7.0
# sidecar
docker pull ghcr.io/llm-d/llm-d-routing-sidecar:v0.7.0

Highlights

  • Gateway API Inference Extension v1.4 (and Gateway API v1.5.1): standalone EPP, InferencePool / Helm / gRPC-related improvements, significant work in flow control and datalayer internals.
  • Initial EPD support (disaggregation profile, KV cache enablement).
  • Sidecar command line option handling has been cleaned up and simplified.
  • Numerous testing and developer experience improvements (e.g., new Makefile targets. elimination of CGO requirements, containerized builds, improved Podman support, automatic dev image built on every PR to main, etc.)

Note

Trivy reports a CVE-2026-4046 on ubi9-micro. As both inference-scheduler and sidecar executables are built with CGO_ENABLED=0, the binaries are fully statically linked and do not link against glibc at all. This is listed for awareness.

What's Changed

  • fix(ci): update Trivy to v0.69.2 by @pierDipi in #675
  • Allow sidecar server to reload TLS certificates by @pierDipi in #607
  • Use trivy action for image scanning by @elevran in #688
  • deps(go): bump the go-dependencies group with 7 updates by @dependabot[bot] in #692
  • feat(sidecar): simplify TLS command line options with StringSlice flags by @gyliu513 in #682
  • Fix terminolgy and add links to docs/archiecture.md by @elevran in #695
  • replace map[string]bool with map[string]struct{} by @roytman in #696
  • Add presubmit makefile target for local verification before pushing a PR by @elevran in #687
  • Update Trivy scan to run newer version with explicit auth tokens by @elevran in #698
  • Remove extra trivy params by @elevran in #702
  • fix: simplify InferencePool flag to namespace/name format by @gyliu513 in #685
  • Trivy complains of user-password mismatch by @elevran in #704
  • fix(test): Add unit test for pd_prerequest.go by @gyliu513 in #706
  • Remove trivy cache and enable manual workflow dispatch by @elevran in #713
  • initial E/PD extension of the sidecar by @roytman in #643
  • Check for uniqueness of media URLs by @roytman in #717
  • Move typo checking from tools makefile to main, under lint by @elevran in #719
  • Rename EncoderPodsHeader according to other constants by @roytman in #721
  • Implement Options pattern for sidecar proxy by @Mohamedma96 in #697
  • Rename common constants by @roytman in #722
  • deps(actions): bump dorny/paths-filter from 3 to 4 by @dependabot[bot] in #723
  • Enable major version updates to gh actions by @elevran in #714
  • NonCachedTokens defines the minimum number of non-cached tokens required by @modassarrana89-new in #691
  • Add external tokenizer PrepareData plugin and TokenizedPrompt scorer support by @acardace in #694
  • Deprecate the workaround used to support vLLM Data Parallel on Istio 1.28 by @shmuelk in #727
  • build: remove CGO dependency by migrating to pure-Go ZMQ by @elevran in #728
  • deps(go): bump google.golang.org/grpc from 1.79.2 to 1.79.3 by @dependabot[bot] in #737
  • Use kubectl kustomize instead of standalone kustomize by @elevran in #741
  • Add idle pod config to active-request-scorer by @dagrayvid in #669
  • [build] Optimize docker build (local and CICD) by @elevran in #740
  • feat: speculative indexing for PrecisePrefixCacheScorer by @bongwoobak in #659
  • Remove obsolete hashBlockSize by @roytman in #748
  • feat: add optional Prometheus monitoring to Kind dev environment by @hexfusion in #742
  • fix: support podman-docker in e2e tests and Makefile by @hexfusion in #730
  • Remove UDS tokenizer image build from inference scheduler repo by @elevran in #739
  • deps(go): bump the kubernetes group with 5 updates by @dependabot[bot] in #754
  • test: add disruption e2e tests for scheduler failure scenarios by @hexfusion in #735
  • Unified Disaggregate Handler by @roytman in #732
  • [build] Add test coverage reporting by @elevran in #749
  • [refactor] Unify sidecar Config and Options by @elevran in #751
  • [cicd[ Make coverage comparison optional by @elevran in https://github.com//pull/762
  • A temporary fix to use the previous simulator image by @roytman in #766
  • Combine EncodeHeaderHandler and PrefillHeaderHandler into a single DisaggHeadersHandler by @roytman in #758
  • Prevent mismatch between new and deprecated APIs by @roytman in #756
  • Revert IGW import to 1.4.0 and update tokenizer plugin accordingly by @vMaroon in #770
  • fix(test): Increase test coverage for prefix_based_pd_decider.go by @gyliu513 in #715
  • Run unit, integration tests and builds in a container by @acardace in #521
  • update Makefile to run e2e tests locally with Podman by @roytman in #775
  • implement context-length-aware plugin (scorer/filter) by @vMaroon in #550
  • Build images use Go builtin cross compilation by @shmuelk in #776
  • feat: bump llm-d-kv-cache for MM-aware prefix-cache routing by @vMaroon in #772
  • change simulator version to v0.8.1 by @mayabar in #773
  • [fix] [cicd] Add missing newline at end of OWNERS, make script more robust for missing newlines by @elevran in #779
  • [fix] Added custom LevelEncoder for sidecar by @elevran in #778
  • Update GW API to v1.5.1 by @elevran in #780
  • Add E2E_KEEP_CLUSTER_ON_FAILURE env variable by @roytman in #782
  • bump kvc import to v0.7.0 by @vMaroon #789

New Contributors

Full Changelog: v0.6.0...v0.7.0

Don't miss a new llm-d-router release

NewReleases is sending notifications on new releases.