kubernetes-sigs/gateway-api-inference-extension v1.0.0-rc.1 on GitHub

What's Changed

chore(deps): bump golang.org/x/sync from 0.15.0 to 0.16.0 by @dependabot[bot] in #1160
feat: Introduce pluggable queue framework by @LukeAVanDrie in #1138
removed USE_STREAMING env var from conformance + tests by @nirrozenbaum in #1157
Conformance: Fixes the EPP ConfigMap Namespace by @danehans in #1166
feat: Introduce pluggable intra-flow dispatch policy framework by @LukeAVanDrie in #1139
Add support for plugin configuration in the InferencePool helm chart by @ahg-g in #1168
feat(epp): use kebab-cased flags for epp by @Xunzhuo in #1177
chore: remove duplicated import for code polish by @Xunzhuo in #1179
Add documentation for the new Configuration via text feature by @shmuelk in #1110
fix: set epp image tag when releasing by @Xunzhuo in #1182
feat: Introduce pluggable inter-flow dispatch policy framework by @LukeAVanDrie in #1167
Update istio release by @LiorLieberman in #1186
test: kubectl-validate manifests in presubmit by @chewong in #1083
Delete the unnecessary Marshal of processRequestBody by @whzghb in #1127
feat(flowcontrol): Introduce ManagedQueue and Service Contracts by @LukeAVanDrie in #1174
(feat) initial types and interfaces for pluggable data layer by @elevran in #1154
Fix a regression in prefix plugin which can cause data race by @liu-cong in #1188
feat: generate crd with version annotation. by @zetxqx in #1134
chore: update vllm deployment tag to latest by @Xunzhuo in #1184
moved build details to version package by @nirrozenbaum in #1185
Add an "Implementing a Compatible Data Plane" section to the implementers guide by @AndresGuedez in #1143
feat(flowcontrol): Implement registry shard by @LukeAVanDrie in #1187
feat(flowcontrol): refine types and consolidate docs by @LukeAVanDrie in #1191
docs: update to use kebab-cased flags changed at #1177 by @nekomeowww in #1193
added graceful shutdown when scheduler config is not initialized by @nirrozenbaum in #1198
feat: move x-k8s to apix and add v1 InferencePool to api/v1 by @capri-xiyue in #1116
feat: Change epp and conformance to use v1 type InferencePool by @capri-xiyue in #1118
chore(deps): bump the kubernetes group with 6 updates by @dependabot[bot] in #1200
Enhanced InferencePool Chart Configurability by @vMaroon in #1211
refactor(flowcontrol): Enable behavioral mocking by @LukeAVanDrie in #1202
random endpoint pick on tie break in max score picker by @nirrozenbaum in #1205
removed cmd/registry file by @nirrozenbaum in #1206
Support scraping metrics from target running with TLS by @pierDipi in #1190
gke-gateway v0.5.0 conformance test report 9/9 by @zetxqx in #1005
added join slack badge to readme by @nirrozenbaum in #1218
chore: 🔨 Use the v0.3.0 llm-d-inference-sim image tag. by @yafengio in #1140
style: ✨ optimize import order and more readable. by @yafengio in #1220
Remove TODO stubs from website by @sats-23 in #1221
docs: update whole repo to v1 inferencepool by @capri-xiyue in #1213
release issue template: updated the tag command to include the -s for signing the tag by @nirrozenbaum in #1196
fix try it out section in quickstart by @nirrozenbaum in #1197
Do not log potentially sensitive data below DEBUG log level by @pierDipi in #1192
Update index.md with gateway-inference-extension slack by @LiorLieberman in #1225
Add fallback logic to support multiple endpoints by @rlakhtakia in #1122
chore: 🔨 add fmt-imports tool for import order. by @yafengio in #1228
fix: missing permission to list inference.networking.k8s.io/v1/inferencepool by @nekomeowww in #1230
fix: Make test iter deterministic to fix flake by @LukeAVanDrie in #1231
feat(flowcontrol): Implement ShardProcessor engine by @LukeAVanDrie in #1203
Add a set of configuration defaults by @shmuelk in #1223
Proposing the successor to the InferenceModel API by @kfswain in #1199
cleanup of unused fields and functions by @nirrozenbaum in #1233
chore: update CRD BundleVersion to main-dev by @zetxqx in #1216
Change String() to accept a value reciever. by @elevran in #1239
renamed kvcache-scorer to kvcache-utilization-scorer by @nirrozenbaum in #1238
Add unit tests by @elevran in #1195
test-report: istio 1.28-alpha v0.4.0 & v0.5.0 report 9/9 by @aslakknutsen in #1102
added scheduler config logging on bootstrap by @nirrozenbaum in #1247
fix: updated to v1 inferencepool in manifests by @capri-xiyue in #1248
chore(deps): bump github.com/onsi/gomega from 1.37.0 to 1.38.0 by @dependabot[bot] in #1253
chore(deps): bump sigs.k8s.io/yaml from 1.5.0 to 1.6.0 by @dependabot[bot] in #1251
chore(deps): bump google.golang.org/grpc from 1.73.0 to 1.74.2 by @dependabot[bot] in #1252
Update the Endpoint Picker Protocol with a new metadata field that communicates status associated with picked endpoints by @AndresGuedez in #1226
chore(deps): bump sigs.k8s.io/controller-tools from 0.17.3 to 0.18.0 by @dependabot[bot] in #1254
Update golangci lint to v2.x by @elevran in #1256
Add nightly benchmarking documentation by @kaushikmitr in #1234
normalize score to make sure it is always in the range of [0,1] by @nirrozenbaum in #1236
updated metrics and logging for plugins by @nirrozenbaum in #1235
fix(flowcontrol): Prevent panic on nil item during shard shutdown by @LukeAVanDrie in #1257
chore(deps): bump github.com/elastic/crd-ref-docs from 0.1.0 to 0.2.0 by @dependabot[bot] in #1250
cleanup of config from scheduling package by @nirrozenbaum in #1263
Add support for multi platform image by @adarshagrawal38 in #1010
epp: add more error integration test cases by @zhengkezhou1 in #1074
fix: split EPP RBAC into cluster and namespaced scoped permission by @chewong in #1071
Renaming InferenceModel to InferenceObjectives by @kfswain in #1255
cleanup: refactor PodList calls to prepare for making pod metrics staleness configurable by @nayihz in #1046
refactor(conformance): restructure tests and resources by @zetxqx in #1232
Refactor(conformance): merge similar helper functions and make the condition check on inferencePool stricter by @zetxqx in #1261
Update lora affinity to be a scorer. by @rlakhtakia in #1121
fix image not build issue by @zetxqx in #1286
Fixes for make fmt-imports by @elevran in #1287
Docs: fixed InferenceObjective in docs by @capri-xiyue in #1284
adding fairness-id header to be used in flow control by @kfswain in #1282
update release template to include patch release by @nirrozenbaum in #1270
Switch to the new default scheduler plugins in integration test by @liu-cong in #1291
Revert "fix image not build issue" by @danehans in #1295
Rename lora affinity plugin to lora-affinity-scorer to be consistent with others by @liu-cong in #1297
revert #1010 to resume new main EPP image build by @zetxqx in #1300
chore(deps): bump github.com/prometheus/client_golang from 1.22.0 to 1.23.0 by @dependabot[bot] in #1303
fix(conformance): Reduce flakiness by using service selector modification in EppUnAvailableFailOpen by @zetxqx in #1265
Promote plugin v2 config to be the default by @liu-cong in #1290
feat: changed to support both v1 and v1a2 ip in EPP by @capri-xiyue in #1277
Deprecate legacy filters by @liu-cong in #1305
Updating tabs to spaces by @kfswain in #1311
Pluggable metrics collection by @elevran in #1237
Removing concurrency issues from Random Picker by @kfswain in #1314
Docs: Bumps kgateway Version in Quickstart by @danehans in #1318
Conformance: Adds Report for kgateway by @danehans in #1317
Refactor the configuration defaults handling code by @shmuelk in #1294
Select InferenceObjective by header by @kfswain in #1307
refactor: 👷 clean unused randomGenerator parameter. by @yafengio in #1322
Docs: Updates kgateway in Implementations Guide by @danehans in #1325
remove protocol specifics from cmd-line flags by @nirrozenbaum in #1296
Adds envoy-ai-gateway conformance report by @Xunzhuo in #1320
Filter inference objectives based on inference pool group by @nicolexin in #1306
Correcting title name by @kfswain in #1334
added plugin state that can be used to share data between different extension point of a plugin by @nirrozenbaum in #1299
Updating model name rewrite to be done by header key by @kfswain in #1331
docs: Enable GIE in Istio installation command by @zhengkezhou1 in #1345
InferenceObjective: Updates PoolRef Group Version by @danehans in #1346
Modifying Criticality; from string, to int by @kfswain in #1348
explicit return from test after nil check by @elevran in #1350
fix: change the inferenceobjective to use v1 inferencepool by @capri-xiyue in #1338
[bug] Fix datalayer Collector test flake by @elevran in #1342
cleanup: fix typo and delete useless parameter by @nayihz in #1310
refactor(flowcontrol): Adopt Composite FlowKey as Primary Identifier by @LukeAVanDrie in #1340
Conformance: Adds InferenceObjective Request Header by @danehans in #1353
test: enhance plugin state test coverage and readability by @yankay in #1349
refactor(conformance): Relocate constants to minimize package dependencies by @zetxqx in #1355
[BBR] perf: optimize model name extraction with selective JSON unmarshaling using struct tags by @pierDipi in #1359
chore(deps): bump google.golang.org/protobuf from 1.36.6 to 1.36.7 by @dependabot[bot] in #1358
Rename criticality to priority by @ahg-g in #1363
Affiliate ready state with leader election with a flag ha-enable-leader-election by @yangligt2 in #1337
cleanup: simplify endpointpickerconfig by @capri-xiyue in #1324
Apply shedding upon saturation for priority below 0 by @ahg-g in #1361
Add agentgateway as implementation by @howardjohn in #1321
Upgrade the inferencePool selector to a struct from a map. by @zetxqx in #1330
fix: make extensionRef to be optional by @capri-xiyue in #1365
fix: make v1a2 remove inline and change conversion logic to match the convention by @capri-xiyue in #1368
Pluggable data layer: transition backend/metrics to use type aliases from datalayer package by @elevran in #1351
docs: match Inference Extension CRDs by @zhengkezhou1 in #1377
refactor: prevent double logging of NamespacedName across reconcilers by @chewong in #1379
Enable kubeapilinter for GIE APIs by @rikatz in #1366
feat: TargetPortNumber int32 to become TargetPorts []Port by @capri-xiyue in #1354
Update doc on sglang models support. by @ReneeZhuGG in #1369
feat: added shortname as alias by @capri-xiyue in #1375
Tooling: Adds PR Template by @danehans in #1385
docs: add prometheus + grafana deployment guide by @EyalPazz in #1019
docs: how to debug integration tests by @zhengkezhou1 in #1067
doc: update the release-quickstart.sh to include the image tag for lora-syncer by @Ruoyu-y in #1080
Fixes Quickstart Script by @danehans in #1388
gitignore macOS generated files. by @bexxmodd in #1378
feat: added env var for pool group by @capri-xiyue in #1328
updated env var example in helm chart by @nirrozenbaum in #1390
remove InferenceModel section from EPP protocol by @nirrozenbaum in #1389
fix: make port number become pointer by @capri-xiyue in #1400
Fix: Handle empty string healthcheck as readiness healthcheck by @yangligt2 in #1402
fix: updated listtype for targetports by @capri-xiyue in #1401
chore(deps): bump the kubernetes group with 6 updates by @dependabot[bot] in #1403
chore(deps): bump github.com/onsi/ginkgo/v2 from 2.23.4 to 2.24.0 by @dependabot[bot] in #1404
depreacte post cycle from scheduling framework by @nirrozenbaum in #1392
Conformance: Updates InferencePoolInvalidEPPService Test Manifest by @danehans in #1417
Conformance: Fixes Secondary InferencePool Selector by @danehans in #1419
Fix Makefile syntax and version consistency by @ErikJiang in #1413
Enable pluggable datalayer as experimental feature by @elevran in #1391
fix(cleanup): change the naming to be endpointpickerref by @capri-xiyue in #1420
fix api-ref-docs make target and generate result by @kfswain in #1422
trace logging for scores per pod by @nirrozenbaum in #1395
remove env vars for cmd-lind args by @nirrozenbaum in #1397
Ensure EPP flags are configurable via Helm chart by @rahulgurnani in #1302
feat(flowcontrol): Implement the FlowRegistry by @LukeAVanDrie in #1319
fix(cleanup): change spec doc by @capri-xiyue in #1434
Cleanup helm flags which have default values from values yaml by @rahulgurnani in #1429
Fix epp startup error due to missing plugin config file flag by @liu-cong in #1439
Update inference gateway public docs to use helm charts by @rahulgurnani in #1370
Fix typo. by @zetxqx in #1437
Final bits of v1.0 API cleanup by @robscott in #1441
remove wg serving leads from owners file by @nirrozenbaum in #1445
fix: first hash of prefix cache with same model name by @livelxw in #1341
cleanup: final clean up of pointer by @capri-xiyue in #1444
adding mutex and contention profile gathering by @kfswain in #1448
remove empty condition list when doing v1 and v1alpha2 conversion. by @zetxqx in #1447
Updates InferencePool API Conversion Code by @danehans in #1451

New Contributors

@whzghb made their first contribution in #1127
@AndresGuedez made their first contribution in #1143
@nekomeowww made their first contribution in #1193
@vMaroon made their first contribution in #1211
@pierDipi made their first contribution in #1190
@sats-23 made their first contribution in #1221
@yangligt2 made their first contribution in #1337
@rikatz made their first contribution in #1366
@ReneeZhuGG made their first contribution in #1369
@Ruoyu-y made their first contribution in #1080
@bexxmodd made their first contribution in #1378
@ErikJiang made their first contribution in #1413
@rahulgurnani made their first contribution in #1302
@livelxw made their first contribution in #1341

Full Changelog: v0.5.1...v1.0.0-rc.1