github llm-d/llm-d-router v0.5.0

latest releases: v1.0.0, v1.0.0-rc.2, v1.0.0-rc.1...
9 months ago

Docker image is available at:

docker pull ghcr.io/llm-d/llm-d-inference-scheduler:v0.5.0

Notable

  • Prefill/Decode disaggregation awareness in filters and scorers
  • Support for data parallel serving validated with vLLM and inference-sim
  • Various CICD enhancements
  • IGW: Flow control scale from/to zero support
  • IGW: Standalone EPP

What's Changed

  • Fix for flaky sites during lychee md link checker by @pierDipi in #485
  • deps(actions): bump actions/checkout from 5 to 6 by @dependabot[bot] in #487
  • deps(go): bump google.golang.org/grpc from 1.76.0 to 1.77.0 in the go-dependencies group by @dependabot[bot] in #486
  • Use kv-cache-manager based on Go mod version instead of hardcoded by @pierDipi in #484
  • update llm-d-kv-cache version to v0.4.0 by @vMaroon in #492
  • fix: github action missing Trivy scan on sidecar image by @zdtsw in #481
  • [Fix] Enhance macOS Makefile to Support Non-Homebrew Python Installations by @hyeongyun0916 in #489
  • feat(allowlist): support both v1 and v1alpha2 InferencePool APIs with flag by @googs1025 in #474
  • fix: make 'install-dependencies' and 'build' target by @zdtsw in #493
  • skip lint and test when only docs change by @setsunakute in #494
  • fix: Fixes for Data Parallel support when also running with Prefix Disaggregation by @shmuelk in #498
  • sync gie to v1.2.0 by @nirrozenbaum in #499
  • Error if PYTHON_CONFIG is empty by @elevran in #497
  • Add GH action to check for signed and verified commits in PR by @elevran in #500
  • deps(actions): bump crate-ci/typos from 1.39.2 to 1.40.0 by @dependabot[bot] in #501
  • build: make build should use CGO_CFLAGS, CGO_LDFLAGS by @evacchi in #503
  • chore: bump gie to v1.2.1 by @nirrozenbaum in #504
  • deps(go): bump sigs.k8s.io/gateway-api from 1.4.0 to 1.4.1 in the kubernetes group by @dependabot[bot] in #508
  • deps(go): bump the go-dependencies group with 3 updates by @dependabot[bot] in #507
  • Miscellaneous dependency updates by @shmuelk in #510
  • deps(go): bump the kubernetes group with 5 updates by @dependabot[bot] in #513
  • Fix running make env-dev-kind by @acardace in #512
  • test: add precise_prefix_cache_test by @evacchi in #505
  • test: reuse upstream data store and enable logr in unit tests by @MregXN in #518
  • feat: allow pd_profile_handler to handle diverse plugin types by @hyeongyun0916 in #516
  • deps(actions): bump crate-ci/typos from 1.40.0 to 1.40.1 by @dependabot[bot] in #526
  • deps(go): bump google.golang.org/grpc from 1.77.0 to 1.78.0 in the go-dependencies group by @dependabot[bot] in #527
  • feat(metrics): add model_name label to PD decision metric by @googs1025 in #528
  • deps(actions): bump crate-ci/typos from 1.40.1 to 1.41.0 by @dependabot[bot] in #532
  • Configure dependabot ignores Go version updates by @elevran in #533
  • Updates the architecture description by @davidbreitgand in #525
  • Dependabot: exert finer control over package updates by @elevran in #542
  • port auto-assign action from llm-d-kv-cache by @vMaroon in #551
  • refactor: set python version and pin docker image with tag by @zdtsw in #543
  • chore(test): update API version for nixl test by @zdtsw in #555
  • deps(go): bump the go-dependencies group with 2 updates by @dependabot[bot] in #558
  • deps(actions): bump crate-ci/typos from 1.41.0 to 1.42.0 by @dependabot[bot] in #557
  • deps(actions): bump actions/checkout from 4 to 6 by @dependabot[bot] in #556
  • Update auto-assign logic by @elevran in #560
  • Remove newline in unsigned commit message by @elevran in #561
  • bump gie to v1.3.0 rc2 by @nirrozenbaum in #562
  • Update OWNERS by @elevran in #559
  • refactor: Makefile, update docs by @zdtsw in #463
  • feat: add metrics validation in e2e test by @googs1025 in #529
  • feat: make no-hit-lru P/D-aware by @evacchi in #522
  • Update disaggregated Prefill/Decode inference serving documentation by @mayabar in #571
  • deps(actions): bump crate-ci/typos from 1.42.0 to 1.42.1 by @dependabot[bot] in #572
  • deps(go): bump github.com/onsi/ginkgo/v2 from 2.27.4 to 2.27.5 in the go-dependencies group by @dependabot[bot] in #573
  • fix reviewers auto assign minor bug by @nirrozenbaum in #575
  • fix(scorer): make active request pd aware by @kyanokashi in #569
  • test(e2e): cleanup kind cluster by @zdtsw in #563
  • refactor: add early validation in DP profile handler by @zdtsw in #554
  • deps(go): bump the kubernetes group with 2 updates by @dependabot[bot] in #574
  • refactor: kv cache manager repo by @sagearc in #570
  • bumping IGW version to the full released version by @kfswain in #583
  • Enable prefix-cache awareness in active-active multi-replica scheduler deployments by @vMaroon in #578
  • Switch to pre-built vLLM wheels for CPU builds by @sagearc in #582
  • update llm-d-kv-cache import to v0.5.0-RC1 by @vMaroon in #584
  • Use 1.3.0 CRDs by @shmuelk in #586

Updates in Inference Gateway Extension v1.3.0

llm-d-inference-scheduler v0.5.0 has been updated to use the latest version of the Inference Gateway Extension which is 1.3.0.
You can see those changes here

New Contributors

Full Changelog: v0.4.0...v0.5.0

Don't miss a new llm-d-router release

NewReleases is sending notifications on new releases.