Docker image is available at:
docker pull ghcr.io/llm-d/llm-d-inference-scheduler:v0.4.0What's Changed
- Use a production version of Istio by @shmuelk in #334
- add vMaroon as code owner by @elevran in #342
- Upgrade
github.com/llm-d/llm-d-kv-cache-managerimport tov0.3.0by @vMaroon in #344 - add a hold label when PRs are pushed to branch other than main by @nirrozenbaum in #345
- sync gic to latest v1.0.0 release by @nirrozenbaum in #353
- deps(actions): bump actions/stale from 9 to 10 by @dependabot[bot] in #350
- deps(actions): bump actions/setup-go from 5 to 6 by @dependabot[bot] in #351
- deps(actions): bump crate-ci/typos from 1.35.7 to 1.36.2 by @dependabot[bot] in #348
- deps(go): bump the go-dependencies group with 7 updates by @dependabot[bot] in #349
- bump llm-d-kv-cache-manager version by @vMaroon in #359
- fix: Rename config to
kv-cache-utilization-scorerfromkv-cache-scorerby @yankay in #358 - updating release issue-template by @kfswain in #361
- bump llm-d-kv-cache-manager version (v0.3.2) by @vMaroon in #365
- deps(go): bump github.com/onsi/ginkgo/v2 from 2.25.3 to 2.26.0 in the go-dependencies group by @dependabot[bot] in #368
- feat: Add a scoring plugin to distribute new groups evenly by @usize in #357
- implement PreRequest and PostResponse interface checks by @learner0810 in #372
- deps(go): bump the kubernetes group with 2 updates by @dependabot[bot] in #369
- deps(go): bump google.golang.org/grpc from 1.75.1 to 1.76.0 in the go-dependencies group by @dependabot[bot] in #374
- Supports the ResponseComplete plugin by @learner0810 in #378
- deps(actions): bump crate-ci/typos from 1.36.2 to 1.38.1 by @dependabot[bot] in #373
- Fix multi-architecture image issues with Kind by @shmuelk in #362
- feat: Moved the Routing Sidecar from its own repo to the inference-scheduler repo by @shmuelk in #379
- Upgrade to use Gateway Inference Extension 1.1.0 rc.1 by @shmuelk in #384
- deps(go): bump github.com/onsi/ginkgo/v2 from 2.26.0 to 2.27.1 in the go-dependencies group by @dependabot[bot] in #389
- Ensure that max_completion_tokens=1 in Prefill by @shmuelk in #403
- Add explanation of inference-scheduler relation to IGW/GIE by @elevran in #393
- Add test coverage to test-unit Makefile target by @carlory in #391
- Add regression tests for max_completion_tokens by @pierDipi in #411
- Makefile refactoring to minimize the number of targets by @shmuelk in #397
- feat: Add vLLM Data Parallel support to llm-d-inference-scheduler by @shmuelk in #392
- fix(scorer): prevent potential division by zero in ActiveRequest.Score by @googs1025 in #413
- Fixed wildcard targets by @shmuelk in #416
- deps(actions): bump crate-ci/typos from 1.38.1 to 1.39.0 by @dependabot[bot] in #419
- deps(go): bump github.com/onsi/ginkgo/v2 from 2.27.1 to 2.27.2 in the go-dependencies group by @dependabot[bot] in #417
- Missed change to the Go code coverage output file names in the Makefile refactoring by @shmuelk in #422
- Fix: Remove reference to the missing make target by @andreyod in #423
- deps(actions): bump actions/upload-artifact from 4 to 5 by @dependabot[bot] in #420
- deps(go): bump sigs.k8s.io/controller-runtime from 0.22.3 to 0.22.4 in the kubernetes group by @dependabot[bot] in #418
- Enhancement: return 503 instead of 502 when decode node is not ready by @Phil-OSophy-42 in #412
- Remove endpointslices from RBAC by @elevran in #424
- Fix Image Loading for Podman in E2E Tests by @hdefazio in #406
- readme meetings update by @nirrozenbaum in #427
- Fix references to the SideCar's tag by @shmuelk in #428
- Remove duplicate error logs by @hyeongyun0916 in #429
- Upgrade to istio-1.28 by @irar2 in #431
- Complete upgrade to Istio 1.28.0 by @shmuelk in #433
- Upgrade GIE dependency to 1.1.0 by @shmuelk in #435
- Remove dev from branch list in PR actions by @elevran in #434
- Added support for Data Parallel in a Disagregated Prefil/Decode setup by @shmuelk in #432
- Remove code coverage from CI workflow by @carlory in #437
- test: Scale up and down the model server during an end to end test by @shmuelk in #354
- fix: add validation in ByLabelFactory to prevent invalid configurations by @googs1025 in #440
- deps(actions): bump golangci/golangci-lint-action from 8 to 9 by @dependabot[bot] in #444
- change lmcache connector to nixlv2 by @googs1025 in #446
- fix: Roll back automatic updates to Dockerfiles by @shmuelk in #447
- deps(go): bump golang.org/x/sync from 0.17.0 to 0.18.0 in the go-dependencies group by @dependabot[bot] in #443
- fix(profile): validate handler parameters to prevent invalid config by @googs1025 in #449
- Added chat completions preprocessing support by @guygir in #426
- docs: add integration guide for external prefill/decode workloads by @googs1025 in #451
- Define and manage PR lifecycle by @elevran in #450
- test: End to End test for Data Parallel support by @shmuelk in #442
- docs: add PD-aware examples for by-label and by-label-selector plugins by @googs1025 in #454
- deps(actions): bump crate-ci/typos from 1.39.0 to 1.39.2 by @dependabot[bot] in #459
- Add SGLang Connector for Prefill/Decode Disaggregation (migrated from llm-d-routing-sidecar#64) by @bongwoobak in #456
- deps(go): bump the kubernetes group with 4 updates by @dependabot[bot] in #460
- add unit test in scheduler plugin part(by-label, data-parallel-profile-handler, pd-profile-handler) by @googs1025 in #461
- test: Enable running the end to end tests on K8S clusters other than Kind by @shmuelk in #453
- Allow the sidecar to sample from a list of prefill host ports by @smarterclayton in #404
- fix: Fixed issues running locally 'make lint' and 'make test-unit' by @shmuelk in #464
- cleanup: Followup to Python paths fix by @shmuelk in #468
- Replace tab with spaces to avoid treating as make target by @elevran in #469
- minor refactoring of
precise-prefix-cachescorer plugin by @vMaroon in #473 - feat: Add initial metrics and update dependencies by @JeffLuoo in #414
- sync with gie v1.2.0 rc1 by @nirrozenbaum in #478
- Remove use of klog package from sidecar by @elevran in #471
- fix: github action runner not exist by @zdtsw in #477
- Bump kv-cache-manager to v0.4.0-rc2 by @pierDipi in #467
precise-prefix-cachescorer plugin minor refactoring by @vMaroon in #482
New Contributors
- @usize made their first contribution in #357
- @learner0810 made their first contribution in #372
- @andreyod made their first contribution in #423
- @Phil-OSophy-42 made their first contribution in #412
- @hdefazio made their first contribution in #406
- @guygir made their first contribution in #426
- @bongwoobak made their first contribution in #456
- @smarterclayton made their first contribution in #404
- @JeffLuoo made their first contribution in #414
Full Changelog: v0.3.2...v0.4.0