kubeflow/trainer v2.4.0

on GitHub

This is Kubeflow Trainer v2.4.0 release.

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 2.4.0 \
    --set runtimes.defaultEnabled=true

Breaking Changes

• [release-2.4] feat(api): BREAKING CHANGE: limit ReplicatedJob name length to 20 characters in runtimes (#4394 by @andreyvelich)
• fix(framework): BREAKING CHANGE: Set the TrainJob image on the MPI launcher and nodes (#4301 by @matbun)
• fix(manifests): BREAKING CHANGE honour dataCache.lws.install in the Helm chart (#4120 by @reckless-sherixx)
• fix(manifests): BREAKING CHANGE keep the CRDs on helm uninstall (#4122 by @reckless-sherixx)
• chore(api): BREAKING CHANGE: rename CertManagement configuration fields to generic names (#3386 by @tariq-hasan)
• fix(runtimes): emit PRTE env vars so MPI plugin works with Open MPI 5 (#4029 by @matbun)
  • IMPORTANT: suspended TrainJobs that use the MPI plugin (e.g. DeepSpeed, MLX) and were created before the upgrade
    fail to resume after upgrading to v2.4, because the new PRTE_MCA_* env vars cannot be applied to the existing immutable JobSet.
    Wait for MPI-based TrainJobs to complete before upgrading, or recreate the affected TrainJobs after the upgrade.

🚀 Features

Workload-Aware Scheduling

• feat: Workload-Aware Scheduling for TrainJob – Phase 1 (KEP-3015) (#4254 by @andreyvelich)
• feat(api): KEP-3015: Workload Aware Scheduling for TrainJob (#3219 by @andreyvelich)

OptimizationJob API

• feat(controller): Add cache expectations to the OptimizationJob controller to prevent duplicate trial creation (#4150 by @aniket2405)
• feat(docs): add OptimizationJob user guide and runnable example (#4268 by @aniket2405)
• feat(controller): KEP-3562: [OptimizationJob] Add OptimizationJob alpha feature gate (#4331 by @akshaychitneni)
• feat(controller): KEP-3562: [OptimizationJob] Add OptimizationJob controller (#3828 by @aniket2405)
• feat(api): introduce OptimizationJob CRD under Trainer (#3552 by @aniket2405)

Dynamic Resource Allocation

• feat(api): implement DRA support for TrainJob (KEP-2782) (#4027 by @Sridhar1030)
• feat(api): KEP-2782: Dynamic Resource Allocation support for TrainJob (#3540 by @Sridhar1030)

Scheduling & Integrations

• feat(api): allow per-TrainJob schedulerName overrides (#4265 by @sfeng1996)
• feat(controller): Support Job Replicas for multi-slice TPU (#4250 by @andreyvelich)
• feat(framework): clean up Volcano PodGroup on TrainJob terminal or suspended state (#4059 by @daehyun-kim-94)
• feat(ci): add Prometheus monitoring overlay (#4102 by @abhijeet-dhumal)
• feat: Add slurm-bridge scheduling documentation (#3951 by @killianmuldoon)

🐛 Bug Fixes

• [release-2.4] fix(examples): correct OptimizationJob SDK and memory requirements (#4403 by @danvau7)
• [release-2.4] fix(controller): validate status server tokens with TokenReview (#4393 by @akshaychitneni)
• [release-2.4] fix(framework): validate TrainJob and OptimizationJob name length against JobSet pod limits (#4389 by @aniket2405)
• [release-2.4] fix(framework): set MPI resourcesPerNode on the nodes (#4375 by @matbun)
• fix(api): do not default union searchAlgorithm in OptimizationJob (#4366 by @aniket2405)
• fix(controller): grant optimizationjobs/finalizers to the controller (#4365 by @aniket2405)
• fix: cover existing ServiceAccount handling (#4362 by @pratik-naik003)
• fix: nightly security dependency updates (#4341 by @github-actions[bot])
• fix(api): validate OptimizationJob names and keep algorithm service names unique (#4155 by @harshrajdebug)
• fix(api): validate OptimizationJob parameter names (#4062 by @mimosapudical)
• fix(framework): default PRTE slots when NumProcPerNode is nil in MPI plugin (#4333 by @Ahmad-Faraj)
• fix(docs): use v2.3.0 in the installation guide (#4334 by @fjvbn2003)
• fix(initializer): bump certifi to 2026.10.23 to drop distrusted eMudhra roots (#4312 by @akshaychitneni)
• fix(docs): close Kind tip in installation guide (#4309 by @dlanov)
• fix: nightly security Python dependency updates (#4302 by @github-actions[bot])
• fix(runtimes): validate inherited TorchTune node counts (#4255 by @spicysharkco)
• fix(manifests): do not grant the manager create on runtimes and OptimizationJobs in the Helm chart (#4271 by @aarushitandon0)
• fix(framework): keep fileKeyRef when converting TrainJob env vars (#4197 by @anishmehta24)
• fix: nightly security Python dependency updates (#4287 by @github-actions[bot])
• fix(api): bound OptimizationJob random search seed to the sampler range (#4248 by @rshdhere)
• fix(initializer): download only the objects under the S3 prefix (#4116 by @reckless-sherixx)
• fix: nightly security Python dependency updates (#4278 by @github-actions[bot])
• fix(runtimes): install the CUDA 13 build of MLX in the MLX runtime image (#4246 by @matbun)
• fix(manifests): rename serviceMonitor value to prometheus to align with JobSet (#4244 by @robert-bell)
• fix(controller): break ties when picking the best OptimizationJob trial (#4192 by @btxu-db)
• fix: nightly security dependency updates (#4047 by @github-actions[bot])
• fix(docs): configure scheduler for Volcano gang scheduling (#4199 by @auraCodesKM)
• fix(api): reject podGroupPolicy with both coscheduling and volcano set (#4061 by @rishabhsai)
• fix(runtimes): correct runtimeRef field path casing in ClusterTrainingRuntime error messages (#4194 by @nikhil3495)
• fix(manifests): check out every text file with LF line endings (#4191 by @abhishekKokadwar)
• fix(runtimes): guard nil SSHAuthMountPath and MPIImplementation dereferences in MPI plugin (#4157 by @nikhil3495)
• fix: nightly security Python dependency updates (#4142 by @github-actions[bot])
• fix(runtimes): do not panic on nil NumProcPerNode in Flux plugin (#4159 by @100-JM)
• fix(framework): inject distributed envs when TrainJob spec.trainer is omitted (#4082 by @anishmehta24)
• fix(api): Add missing update validation to Training Runtime webhook (#3687 by @robert-bell)
• fix(controller): stop TrainJob reconcile panicking on an unsupported runtimeRef (#4088 by @abhishekKokadwar)
• fix(framework): validate all ReplicatedJobs in Volcano priorityClassName check (#3918 by @abhishekKokadwar)
• fix(controller): reject trailing JSON in status updates (#3971 by @DevaanshPathak)
• fix(framework): prevent potential nil pointer dereference on NumProcPerNode in MPI plugin (#4049 by @yush-1018)
• fix(cache): give workers with no data files a readiness path (#4058 by @reckless-sherixx)
• fix(cache): migrate data cache image to Debian bookworm (#4051 by @andreyvelich)
• fix: nightly security dependency updates (#4022 by @github-actions[bot])
• fix(framework): stop Flux reconcile panic on unset trainer fields (#3904 by @rjgoyln)
• fix: nightly security dependency updates (#3913 by @github-actions[bot])
• fix(controller): validate TrainJob updates against the runtime snapshot (#3960 by @abhishekKokadwar)
• fix(controller): reject oversized chunked status requests (#3973 by @DevaanshPathak)
• fix(runtimes): populate GVK on runtime snapshot before serialization to fix reconciliation failures (#3922 by @HarshPopat23)
• fix(framework): keep unnamed container ports when upserting (#3926 by @aarushitandon0)
• fix(controller): apply configured health probe endpoint names (#3886 by @rjgoyln)
• fix(api): validate coscheduling schedule timeout (#3872 by @DevaanshPathak)
• fix(initializer): improve dataset provider errors (#3826 by @HarshPopat23)
• fix(initializer): allow GGUF files in HuggingFace model snapshot download (#3910 by @sumedhcharjan)
• fix(api): reject zero and negative numNodes and numProcPerNode (#3809 by @sanskar-singh-2403)
• fix(operator): keep the TrainJob Failed condition terminal (#3891 by @XploY04)
• fix(manifests): install OptimizationJob CRD consistently (#3883 by @HarshPopat23)
• fix(api): CEL validation fix to restore CI pipeline (#3874 by @aniket2405)
• fix(ci): Fix the git tag range for the release workflow (#3860 by @andreyvelich)
• fix: nightly security Python dependency updates (#3852 by @github-actions[bot])
• fix: nightly security dependency updates (#3538 by @github-actions[bot])
• fix(operator): improve status server bearer token parsing and add helper tests (#3782 by @uditjainstjis)
• fix(ci): Update Kubeflow SDK installation for E2Es (#3823 by @andreyvelich)
• fix(ci): Remove concurrency from Helm publish action (#3791 by @andreyvelich)
• fix(initializer): validate cache storage URI (#3741 by @HarshPopat23)
• fix(ci): Publish Helm Charts in release workflow (#3788 by @andreyvelich)
• fix(ci): Publish images during the GitHub release workflow (#3785 by @andreyvelich)

⚙️ Miscellaneous Tasks

• feat(docs): add an MPI user guide (#4344 by @matbun)
• feat(framework): validate Volumes, VolumeMounts, and Tolerations in RuntimePatches (#3961 by @Prat260104)
• feat(controller): add TLS curve preferences to controller config (#4103 by @ugiordan)
• feat(docs): SKILL for Comprehensive PR Review (#4201 by @andreyvelich)
• feat(docs): add TTLSecondsAfterFinished and RunPolicy to KEP-2899 (#3824 by @XploY04)
• feat(framework): make plugin phase execution order deterministic (#3989 by @yindia)
• feat(docs): document MPIJob migration to TrainJob (#3968 by @hari-599)
• feat(runtimes): add deprecation warning for TrainingRuntime (#3966 by @Prat260104)
• feat(controller): secure metrics endpoint with TLS and RBAC-based auth (#3912 by @ugiordan)
• feat(controller): register TrainJobStatus server with controller manager healthz and readyz probes (#3338 by @krishdef7)
• feat: add .agents skill for add-new-crd (#3837 by @h0pers)
• feat(operator): Add PreComponentBuilder and PodSpec plugins (#3889 by @andreyvelich)
• feat(test): Add integration tests for status server (#3373 by @digvijay-y)
• chore(deps): Bump PyTorch to 2.15 and CUDA to 13.4 version (#4369 by @andreyvelich)
• chore(deps): bump transformers in /cmd/runtimes/deepspeed (#4349 by @dependabot[bot])
• chore(deps): bump shogo82148/actions-goveralls from 1.11.4 to 1.11.5 (#4355 by @dependabot[bot])
• chore(deps): bump tonic from 0.14.6 to 0.14.7 in /pkg/data_cache/test (#4350 by @dependabot[bot])
• chore(deps): bump docker/login-action from 4.6.0 to 4.7.0 (#4354 by @dependabot[bot])
• chore(deps): bump pypa/gh-action-pypi-publish from 1.14.2 to 1.14.3 (#4353 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.84.0 to 1.85.0 (#4352 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 2.1.1 to 2.3.0 in /cmd/initializers/model (#4330 by @dependabot[bot])
• chore(deps): bump tonic from 0.14.6 to 0.14.7 in /pkg/data_cache (#4351 by @dependabot[bot])
• chore(deps): add a two-day cooldown for Dependabot version updates (#4348 by @amengnew)
• chore(kep): KEP-3562: [OptimizationJob] Add kep.yaml for OptimizationJob (#4346 by @akshaychitneni)
• chore(deps): bump opendal from 0.47.10 to 0.47.12 in /cmd/initializers/dataset (#4329 by @dependabot[bot])
• chore(deps): bump actions/download-artifact from 8.0.1 to 8.0.2 (#4227 by @dependabot[bot])
• chore(deps): bump actions/setup-python from 7.0.0 to 7.1.0 (#4328 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.10 to 0.47.12 in /cmd/initializers/model (#4325 by @dependabot[bot])
• chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.4 to 4.1.6 (#4336 by @dependabot[bot])
• chore(deps): bump rustls from 0.23.45 to 0.23.46 in /pkg/data_cache (#4335 by @dependabot[bot])
• chore(deps): bump mlx from 0.32.3 to 0.32.4 in /cmd/runtimes/mlx (#4274 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 2.1.1 to 2.3.0 in /cmd/initializers/dataset (#4326 by @dependabot[bot])
• chore(deps): bump transformers from 5.18.0 to 5.20.2 in /cmd/runtimes/deepspeed (#4322 by @dependabot[bot])
• chore(deps): bump deepspeed from 0.19.7 to 0.19.8 in /cmd/runtimes/deepspeed (#4321 by @dependabot[bot])
• chore(api): generate OpenAPI v3 directly (#4126 by @dlanov)
• chore(deps): bump shogo82148/actions-goveralls from 1.11.3 to 1.11.4 (#4327 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.38.2 to 4.39.1 (#4323 by @dependabot[bot])
• chore(deps): bump tokio from 1.53.1 to 1.54.1 in /pkg/data_cache (#4320 by @dependabot[bot])
• chore(deps): bump tokio from 1.53.1 to 1.54.1 in /pkg/data_cache/test (#4319 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.4.2-devel-ubuntu22.04 to 13.4.3-devel-ubuntu22.04 in /cmd/runtimes/mlx (#4317 by @dependabot[bot])
• chore(deps): bump kubernetes from 36.0.3 to 37.0.1 in /cmd/initializers/dataset (#4235 by @dependabot[bot])
• chore(deps): bump actions/upload-artifact from 7.0.1 to 7.0.2 (#4234 by @dependabot[bot])
• chore(deps): bump datasets from 5.0.1 to 5.1.0 in /cmd/runtimes/deepspeed (#4230 by @dependabot[bot])
• chore(deps): bump datasets from 5.0.1 to 5.1.0 in /cmd/runtimes/mlx (#4229 by @dependabot[bot])
• chore(deps): bump hyper from 1.11.1 to 1.12.0 in /pkg/data_cache (#4228 by @dependabot[bot])
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.33.0 to 2.33.1 (#4224 by @dependabot[bot])
• chore(deps): bump sigs.k8s.io/scheduler-plugins from 0.35.4-devel to 0.36.4-devel in the kubernetes group across 1 directory (#4218 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.4.2-devel-ubuntu26.04 to 13.4.3-devel-ubuntu26.04 in /cmd/runtimes/deepspeed (#4318 by @dependabot[bot])
• chore(test): add Open MPI 4 e2e for the MPI runtime (#4282 by @matbun)
• chore(manifests): generate the Helm manager ClusterRole rules from the kustomize role (#4291 by @ganeshmshetty)
• chore(scripts): use go.mod go version for generation (#4295 by @robert-bell)
• chore(test): add unit tests for the status server token authorizer (#4200 by @shreyaabaranwal)
• chore: update controller tools to v0.23 and drop crdschema hack (#4249 by @kannon92)
• chore(deps): bump golang.org/x/crypto from 0.57.0 to 0.58.0 in the golang group (#4223 by @dependabot[bot])
• chore(test): add unit tests for TrainingRuntime and ClusterTrainingRuntime update validation (#4165 by @DsThakurRawat)
• chore(runtimes): add ValidateObjects test coverage for TrainingRuntime and ClusterTrainingRuntime (#3892 by @Prat260104)
• chore(deps): Bump JobSet to v0.13.0 version (#4217 by @andreyvelich)
• chore(deps): bump softprops/action-gh-release from 3.0.2 to 3.0.3 (#4038 by @dependabot[bot])
• chore(test): add unit tests for status server middleware and server helpers (#4164 by @DsThakurRawat)
• chore(deps): bump huggingface-hub from 1.31.0 to 2.1.1 in /cmd/initializers/dataset (#4183 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.9 to 0.47.10 in /cmd/initializers/dataset (#4137 by @dependabot[bot])
• chore(deps): bump deepspeed from 0.19.6 to 0.19.7 in /cmd/runtimes/deepspeed (#4093 by @dependabot[bot])
• chore(deps): bump github.com/onsi/gomega from 1.43.0 to 1.44.0 (#4133 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.83.2 to 1.84.0 (#4172 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.29.0 to 2.1.1 in /cmd/initializers/model (#4182 by @dependabot[bot])
• chore(deps): bump shogo82148/actions-goveralls from 1.11.2 to 1.11.3 (#4184 by @dependabot[bot])
• chore(deps): bump transformers from 5.16.1 to 5.18.0 in /cmd/runtimes/deepspeed (#4181 by @dependabot[bot])
• chore(deps): bump torch from 2.13.0 to 2.14.1 in /cmd/runtimes/deepspeed (#4179 by @dependabot[bot])
• chore(deps): bump mlx from 0.32.2 to 0.32.3 in /cmd/runtimes/mlx (#4177 by @dependabot[bot])
• chore(deps): bump futures from 0.3.33 to 0.3.34 in /pkg/data_cache/test (#4176 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.3.1-devel-ubuntu22.04 to 13.4.2-devel-ubuntu22.04 in /cmd/runtimes/deepspeed (#4175 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.3.1-devel-ubuntu22.04 to 13.4.2-devel-ubuntu22.04 in /cmd/runtimes/mlx (#4174 by @dependabot[bot])
• chore(deps): bump rust from 1.98-bookworm to 1.99-bookworm in /cmd/data_cache (#4173 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.6 to 0.47.10 in /cmd/initializers/model (#4138 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.38.0 to 4.38.2 (#4136 by @dependabot[bot])
• chore(deps): bump clap from 4.6.3 to 4.6.7 in /pkg/data_cache/test (#4095 by @dependabot[bot])
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.32.2 to 2.33.0 (#4089 by @dependabot[bot])
• chore(deps): bump golang.org/x/crypto from 0.56.0 to 0.57.0 in the golang group across 1 directory (#4065 by @dependabot[bot])
• chore(test): check MPI ranks run on both nodes in e2e (#4166 by @auraCodesKM)
• chore(ci): use common boilerplate check action (#4144 by @nightcityblade)
• chore(cache): upgrade datafusion deps (#4132 by @akshaychitneni)
• chore(deps): bump opendal from 0.47.6 to 0.47.9 in /cmd/initializers/dataset (#4074 by @dependabot[bot])
• chore(deps): bump anyio from 4.13.0 to 4.14.2 in /cmd/initializers/dataset (#4081 by @dependabot[bot])
• chore(deps): bump anyio from 4.13.0 to 4.14.2 in /cmd/initializers/model (#4080 by @dependabot[bot])
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.44.0 to 1.45.0 (#4079 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.29.0 to 1.31.0 in /cmd/initializers/dataset (#4072 by @dependabot[bot])
• chore(deps): bump actions-rust-lang/setup-rust-toolchain from 1.17.0 to 2.0.0 (#4071 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.37.9 to 4.38.0 (#4070 by @dependabot[bot])
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.32.1 to 2.32.2 (#4066 by @dependabot[bot])
• chore(deps): bump go.opentelemetry.io/otel/sdk from 1.44.0 to 1.45.0 (#4078 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.83.1 to 1.83.2 (#4046 by @dependabot[bot])
• chore(deps): bump github.com/coreos/go-oidc/v3 from 3.20.0 to 3.21.0 (#4036 by @dependabot[bot])
• chore: update trainer to k8s 1.37 (#4005 by @kannon92)
• chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 (#4023 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.28.0 to 1.29.0 in /cmd/initializers/dataset (#4018 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.28.0 to 1.29.0 in /cmd/initializers/model (#4017 by @dependabot[bot])
• chore(deps): bump pydantic from 2.13.4 to 2.13.5 in /api/python_api (#4016 by @dependabot[bot])
• chore(deps): bump shogo82148/actions-goveralls from 1.11.1 to 1.11.2 (#4015 by @dependabot[bot])
• chore(deps): bump transformers from 5.15.1 to 5.16.1 in /cmd/runtimes/deepspeed (#4012 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.37.8 to 4.37.9 (#4014 by @dependabot[bot])
• chore(deps): bump hyper from 1.11.0 to 1.11.1 in /pkg/data_cache (#4013 by @dependabot[bot])
• chore(deps): bump deepspeed from 0.19.5 to 0.19.6 in /cmd/runtimes/deepspeed (#4011 by @dependabot[bot])
• chore(deps): bump mlx from 0.32.1 to 0.32.2 in /cmd/runtimes/mlx (#4010 by @dependabot[bot])
• chore(deps): update bitsandbytes requirement from >=0.50.1 to >=0.50.2 in /cmd/trainers/torchtune (#4009 by @dependabot[bot])
• chore(deps): bump github.com/onsi/gomega from 1.42.1 to 1.43.0 (#4008 by @dependabot[bot])
• chore(docs): improve website outline and framework guide intros (#4004 by @andreyvelich)
• chore: add 1.37 e2e tests (#4007 by @kannon92)
• chore(deps): bump huggingface-hub from 1.27.0 to 1.28.0 in /cmd/initializers/model (#3984 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.5 to 0.47.6 in /cmd/initializers/model (#3986 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.37.6 to 4.37.8 (#3982 by @dependabot[bot])
• chore(deps): bump golang from 1.26 to 1.27 in /cmd/trainer-controller-manager (#3978 by @dependabot[bot])
• chore(deps): bump transformers from 5.15.0 to 5.15.1 in /cmd/runtimes/deepspeed (#3981 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.5 to 0.47.6 in /cmd/initializers/dataset (#3985 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.27.0 to 1.28.0 in /cmd/initializers/dataset (#3983 by @dependabot[bot])
• chore(deps): bump mlx from 0.30.6 to 0.32.1 in /cmd/runtimes/mlx (#3980 by @dependabot[bot])
• chore(deps): bump rust from 1.97-bullseye to 1.98-bullseye in /cmd/data_cache (#3977 by @dependabot[bot])
• chore(deps): bump the kubernetes group with 7 updates (#3976 by @dependabot[bot])
• chore(deps): bump quinn-proto from 0.11.14 to 0.11.17 in /pkg/data_cache (#3810 by @dependabot[bot])
• chore(controller): unit test TrainJob deadline reconciliation with a fake clock (#3915 by @sumedhcharjan)
• chore(docs): Add project board section to the Contributing guide (#3987 by @reckless-sherixx)
• chore(ci): add kep scope for PR titles (#3959 by @andreyvelich)
• chore(deps): bump deepspeed from 0.19.4 to 0.19.5 in /cmd/runtimes/deepspeed (#3941 by @dependabot[bot])
• chore(deps): bump golang.org/x/crypto from 0.54.0 to 0.55.0 in the golang group across 1 directory (#3934 by @dependabot[bot])
• chore(deps): bump github.com/google/cel-go from 0.26.0 to 0.29.0 (#3953 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.79.3 to 1.82.1 (#3952 by @dependabot[bot])
• chore(deps): bump Swatinem/rust-cache from e18b497796c12c097a38f9edb9d0641fb99eee32 to f0d9c3887740aee45f6153b24b3a6b815192ec16 (#3942 by @dependabot[bot])
• chore(deps): bump async-trait from 0.1.91 to 0.1.92 in /pkg/data_cache (#3940 by @dependabot[bot])
• chore(deps): update bitsandbytes requirement from >=0.49.2 to >=0.50.1 in /cmd/trainers/torchtune (#3939 by @dependabot[bot])
• chore(deps): bump futures from 0.3.33 to 0.3.34 in /pkg/data_cache/test (#3938 by @dependabot[bot])
• chore(deps): bump transformers from 5.14.1 to 5.15.0 in /cmd/runtimes/deepspeed (#3937 by @dependabot[bot])
• chore(deps): bump futures from 0.3.33 to 0.3.34 in /pkg/data_cache (#3936 by @dependabot[bot])
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.32.0 to 2.32.1 (#3935 by @dependabot[bot])
• chore(deps): bump datasets from 5.0.0 to 5.0.1 in /cmd/runtimes/deepspeed (#3842 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.24.0 to 1.27.0 in /cmd/initializers/dataset (#3903 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.3 to 0.47.5 in /cmd/initializers/model (#3848 by @dependabot[bot])
• chore(deps): bump the kubernetes group across 1 directory with 7 updates (#3811 by @dependabot[bot])
• chore(deps): bump github/codeql-action/upload-sarif from 4.37.1 to 4.37.6 (#3901 by @dependabot[bot])
• chore(deps): bump arrow-flight from 59.0.0 to 59.2.0 in /pkg/data_cache/test (#3896 by @dependabot[bot])
• chore(deps): bump actions/stale from 10.4.0 to 11.0.0 (#3850 by @dependabot[bot])
• chore(deps): bump docker/login-action from 4.4.0 to 4.6.0 (#3849 by @dependabot[bot])
• chore(deps): bump opendal from 0.47.3 to 0.47.5 in /cmd/initializers/dataset (#3844 by @dependabot[bot])
• chore(deps): bump datasets from 5.0.0 to 5.0.1 in /cmd/runtimes/mlx (#3843 by @dependabot[bot])
• chore(deps): bump huggingface-hub from 1.24.0 to 1.27.0 in /cmd/initializers/model (#3902 by @dependabot[bot])
• chore(deps): bump shogo82148/actions-goveralls from 1.11.0 to 1.11.1 (#3897 by @dependabot[bot])
• chore(deps): bump deepspeed from 0.19.2 to 0.19.4 in /cmd/runtimes/deepspeed (#3895 by @dependabot[bot])
• chore(deps): bump clap from 4.6.3 to 4.6.6 in /pkg/data_cache/test (#3894 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.3.0-devel-ubuntu22.04 to 13.3.1-devel-ubuntu22.04 in /cmd/runtimes/deepspeed (#3840 by @dependabot[bot])
• chore(deps): bump nvidia/cuda from 13.3.0-devel-ubuntu22.04 to 13.3.1-devel-ubuntu22.04 in /cmd/runtimes/mlx (#3839 by @dependabot[bot])
• chore(deps): bump pypa/gh-action-pypi-publish from 1.14.1 to 1.14.2 (#3851 by @dependabot[bot])
• chore(operator): drop unused test-only helpers from pkg/features (#3880 by @rjgoyln)
• chore: Nominate robert-bell as Kubeflow Trainer reviewer (#3873 by @andreyvelich)
• chore(api): upgrade controller-gen to 0.21 (#3853 by @robert-bell)
• chore(api): Remove duplicate TrainJob status patch (#3448 by @robert-bell)

New Contributors

• @danvau7 made their first contribution in #4402
• @pratik-naik003 made their first contribution in #4362
• @matbun made their first contribution in #4344
• @Prat260104 made their first contribution in #3961
• @amengnew made their first contribution in #4348
• @harshrajdebug made their first contribution in #4155
• @dlanov made their first contribution in #4126
• @mimosapudical made their first contribution in #4062
• @ugiordan made their first contribution in #4103
• @Ahmad-Faraj made their first contribution in #4333
• @fjvbn2003 made their first contribution in #4334
• @sfeng1996 made their first contribution in #4265
• @spicysharkco made their first contribution in #4255
• @ganeshmshetty made their first contribution in #4291
• @aarushitandon0 made their first contribution in #4271
• @anishmehta24 made their first contribution in #4197
• @daehyun-kim-94 made their first contribution in #4059
• @rshdhere made their first contribution in #4248
• @shreyaabaranwal made their first contribution in #4200
• @DsThakurRawat made their first contribution in #4165
• @btxu-db made their first contribution in #4192
• @auraCodesKM made their first contribution in #4199
• @rishabhsai made their first contribution in #4061
• @nikhil3495 made their first contribution in #4194
• @abhishekKokadwar made their first contribution in #4191
• @100-JM made their first contribution in #4159
• @nightcityblade made their first contribution in #4144
• @DevaanshPathak made their first contribution in #3971
• @yindia made their first contribution in #3989
• @yush-1018 made their first contribution in #4049
• @hari-599 made their first contribution in #3968
• @killianmuldoon made their first contribution in #3951
• @rjgoyln made their first contribution in #3904
• @sumedhcharjan made their first contribution in #3915
• @sanskar-singh-2403 made their first contribution in #3809
• @digvijay-y made their first contribution in #3373
• @uditjainstjis made their first contribution in #3782