If you want to upgrade to v2.4.0, remember to set hook.upgradeCrd=true when running helm upgrade. This will create a Helm pre-install/pre-upgrade Job to run kubectl apply --server-side to update CRDs.
Announcements
- We are going to deprecate Kustomize manifests after three months, see #2702. We suggest using Helm to manage spark-operator releases. Plase leave comments in the PR if you are still using Kustomize manifests.
Highlights
- SparkApplication now can be suspended/resumed by setting
.spec.suspendtotrue/falseand will have an integration with Kueue, see kubernetes-sigs/kueue#7268. - SparkConnect server service can be customized by
.spec.server.service.
Features
- fix(chart): add revisionHistoryLimit option (#2625 by @t3mi)
- Suspend/Resume feature on SparkApplication (#2387 by @everpeace)
- Set ControllerReference on driver pod and non-Controller OwnerReference on executor pod at submission time (#2670 by @everpeace)
- added SparkApp name validator to accept valid DNS-1035 format (#2711 by @aryankumar04)
- No propagate Kueue labels to driver and executor pods (#2714 by @everpeace)
- feat: add support for customizing connect service (#2709 by @ChenYi015)
Bug Fixes
- Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)
- Grant create events permissions to Controller (#2616 by @Rockawear)
- fix(connect): propagate executor memory setting to spark options (#2656 by @mrjoe7)
- [fix] add miss cli params. Add leader election params in chart (#2657 by @aagumin)
- fix: webhook encoder configuration (#2664 by @pvbouwel)
- bugfix: hook.upgradeCrd use cases (#2663 by @pvbouwel)
- Correct entrypoint.sh for Openshift (#2645 by @Reamer)
- Fix driver host configuration to handle IPv6 addresses (#2703 by @tiagotxm)
- Add permissions for sparkconnects/finalizers (#2727 by @ChenYi015)
- Remove redundant name and namespace context in logs (#2723 by @ChenYi015)
- Logging info when resources associated with SparkApplication still exist (#2725 by @ChenYi015)
Unit Tests
- Add kube-scheduler podgroup unit tests (#2689 by @shadowinlife)
- Generate HTML coverage report after running unit tests (#2691 by @shadowinlife)
- Avoid 409 (Conflict) error at e2e tests with retry. (#2695 by @shadowinlife)
- Add unit test for sparkapplication_validator (#2692 by @shadowinlife)
- Add unit test for scheduledsparkapplication_validator (#2694 by @shadowinlife)
- Refactor the unit tests for web_ui.go (#2696 by @shadowinlife)
Refactor
- unify SparkApplication defaulting logic (#2671 by @zhzhuang-zju)
- Replace strconv.Atoi with strconv.ParseInt (#2699 by @ChenYi015)
- refactor: use ptr.To to replace util.XxxPtr (#2693 by @ChenYi015)
Dependencies
- Bump actions/checkout from 4 to 5 (#2623 by @dependabot[bot])
- Bump actions/download-artifact from 4 to 5 (#2624 by @dependabot[bot])
- Bump helm.sh/helm/v3 from 3.18.4 to 3.18.5 (#2627 by @dependabot[bot])
- Bump github.com/go-viper/mapstructure/v2 from 2.3.0 to 2.4.0 (#2635 by @dependabot[bot])
- Bump actions/setup-go from 5 to 6 (#2651 by @dependabot[bot])
- Bump actions/stale from 9 to 10 (#2649 by @dependabot[bot])
- Bump aquasecurity/trivy-action from 0.32.0 to 0.33.1 (#2650 by @dependabot[bot])
- Bump github.com/golang/glog from 1.2.4 to 1.2.5 (#2620 by @dependabot[bot])
- Bump github.com/onsi/ginkgo/v2 from 2.23.3 to 2.26.0 (#2660 by @dependabot[bot])
- Bump sigs.k8s.io/scheduler-plugins from 0.31.8 to 0.32.7 (#2675 by @dependabot[bot])
- Bump github/codeql-action from 3 to 4 (#2672 by @dependabot[bot])
- Bump github.com/stretchr/testify from 1.10.0 to 1.11.1 (#2673 by @dependabot[bot])
- Bump golang.org/x/time from 0.9.0 to 0.14.0 (#2686 by @dependabot[bot])
- Bump github.com/prometheus/client_golang from 1.22.0 to 1.23.2 (#2682 by @dependabot[bot])
- Bump github.com/spf13/viper from 1.20.1 to 1.21.0 (#2683 by @dependabot[bot])
- Bump github/codeql-action from 3.29.2 to 4.31.0 (#2684 by @dependabot[bot])
- Bump github.com/containerd/containerd from 1.7.27 to 1.7.29 (#2712 by @dependabot[bot])
- Bump sigs.k8s.io/yaml from 1.5.0 to 1.6.0 (#2704 by @dependabot[bot])
- Bump github.com/spf13/cobra from 1.9.1 to 1.10.1 (#2705 by @dependabot[bot])
- Bump github/codeql-action from 4.31.0 to 4.31.2 (#2707 by @dependabot[bot])
- Bump actions/upload-artifact from 4 to 5 (#2681 by @dependabot[bot])
- Bump actions/download-artifact from 5 to 6 (#2680 by @dependabot[bot])
- Bump actions/checkout from 4.2.2 to 5.0.0 (#2685 by @dependabot[bot])
- Bump ossf/scorecard-action from 2.4.2 to 2.4.3 (#2687 by @dependabot[bot])
- Bump github.com/onsi/ginkgo/v2 from 2.26.0 to 2.27.2 (#2719 by @dependabot[bot])
- Bump golang.org/x/mod from 0.27.0 to 0.29.0 (#2720 by @dependabot[bot])
- Bump helm/chart-testing-action from 2.7.0 to 2.8.0 (#2722 by @dependabot[bot])
Misc
- Add changelog for v2.3.0 (#2614 by @ChenYi015)
- Update README (#2634 by @ChenYi015)
- Add Spark Operator Roadmap (#2622 by @ChenYi015)
- CI: Add Scorecard workflow (#2654 by @AdamKorcz)
- Update GitHub action workflow permissions (#2678 by @ChenYi015)