kubeflow/spark-operator v2.6.0-rc.0 on GitHub

Installation

helm install spark-operator oci://ghcr.io/kubeflow/helm-charts/spark-operator \
  --version 2.6.0-rc.0 --namespace spark-operator --create-namespace

Images: ghcr.io/kubeflow/spark-operator/controller:2.6.0-rc.0, ghcr.io/kubeflow/spark-operator/kubectl:2.6.0-rc.0

Python API: pip install kubeflow-spark-api==2.6.0rc0

What's Changed

Update workflow and docs for releasing Spark operator by @ChenYi015 in #2089
Use controller-runtime to reconsturct spark operator by @ChenYi015 in #2072
Fix broken integration test CI by @ChenYi015 in #2109
Fix CI: environment variable BRANCH is missed by @ChenYi015 in #2111
Increase frequency for Stale bot by @andreyvelich in #2124
Support gang scheduling with Yunikorn by @jacobsalway in #2107
Update Makefile for building sparkctl by @ChenYi015 in #2119
fix: Add default values for namespaces to match usage descriptions by @xiyoubird in #2128
Add changelog for v2.0.0-rc.0 by @ChenYi015 in #2126
Fix: Spark role binding did not render properly when setting spark service account name by @ChenYi015 in #2135
Added jacobsalway as a reviewer by @yuchaoran2011 in #2140
Reintroduce option webhook.enable by @ChenYi015 in #2142
Add default batch scheduler argument by @jacobsalway in #2143
Update release workflow and docs by @ChenYi015 in #2121
fix: unable to set controller/webhook replicas to zero by @ChenYi015 in #2147
Adding support for setting spark job namespaces to all namespaces by @ChenYi015 in #2123
Support extended kube-scheduler as batch scheduler by @ChenYi015 in #2136
Run e2e tests on Kind by @jacobsalway in #2148
Set schedulerName to Yunikorn by @jacobsalway in #2153
Create role and rolebinding for controller/webhook in every spark job namespace if not watching all namespaces by @ChenYi015 in #2129
Fix: e2e test failes due to webhook not ready by @ChenYi015 in #2149
Upgrade to Go 1.23.1 by @jacobsalway in #2155
Upgrade to Spark 3.5.2 by @jacobsalway in #2154
Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 by @dependabot[bot] in #2159
feat: support driver and executor pod use different priority by @Kevinz857 in #2146
Bump gocloud.dev from 0.37.0 to 0.39.0 by @dependabot[bot] in #2160
Update e2e tests by @ChenYi015 in #2161
fix: webhook not working when settings spark job namespaces to empty by @ChenYi015 in #2163
fix: The logger had an odd number of arguments, making it panic by @tcassaert in #2166
replace datamechanics -> spot.io as adopter (acquisition) by @ImpSy in #2168
Upgrade to Spark 3.5.2(#2012) by @ha2hi in #2157
Feature: Add pprof endpoint by @ImpSy in #2164
fix the make kind-delete-custer to avoid accidental kubeconfig deletion by @ImpSy in #2172
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 by @dependabot[bot] in #2174
Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 by @dependabot[bot] in #2173
implement workflow to scan latest released docker image by @ImpSy in #2177
Add specific error in log line when failed to create web UI service by @tcassaert in #2170
Account for spark.executor.pyspark.memory in Yunikorn gang scheduling by @jacobsalway in #2178
Fix: spark application does not respect time to live seconds by @ChenYi015 in #2165
FEATURE: build operator image as non-root by @ImpSy in #2171
Bump github.com/onsi/ginkgo/v2 from 2.19.0 to 2.20.2 by @dependabot[bot] in #2188
Bump github.com/onsi/gomega from 1.33.1 to 1.34.2 by @dependabot[bot] in #2189
Update controller RBAC for ConfigMap and PersistentVolumeClaim by @ChenYi015 in #2187
Group monthly dependabot updates for gomod by @jacobsalway in #2190
Add changelog for v2.0.1 and update README by @ChenYi015 in #2195
Add yaml file to create serviceaccount by @ha2hi in #2191
FEATURE: add cli argument to modify controller workqueue ratelimiter by @ImpSy in #2186
Fix ingress capability discovery by @jacobsalway in #2201
Revert "Group monthly dependabot updates for gomod (#2190)" by @jacobsalway in #2203
Bump github.com/aws/aws-sdk-go-v2 from 1.30.5 to 1.31.0 by @dependabot[bot] in #2207
Bump golang.org/x/net from 0.28.0 to 0.29.0 by @dependabot[bot] in #2205
Bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible by @dependabot[bot] in #2125
Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.3 to 1.63.3 by @dependabot[bot] in #2206
Update integration test workflow and add golangci lint check by @ChenYi015 in #2197
Bump github.com/aws/aws-sdk-go-v2 from 1.31.0 to 1.32.0 by @dependabot[bot] in #2229
Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 by @dependabot[bot] in #2228
Bump manusa/actions-setup-minikube from 2.11.0 to 2.12.0 by @dependabot[bot] in #2226
Bump golang.org/x/time from 0.6.0 to 0.7.0 by @dependabot[bot] in #2227
fix: imagePullPolicy was ignored by @missedone in #2222
fix: spark-submission failed due to lack of permission by user spark by @missedone in #2223
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.33 to 1.27.42 by @dependabot[bot] in #2231
Bump github.com/prometheus/client_golang from 1.19.1 to 1.20.4 by @dependabot[bot] in #2204
Remove cap_net_bind_service from image by @jacobsalway in #2216
Add @ImpSy as reviewer by @ChenYi015 in #2235
fix: webhook panics due to logging by @ChenYi015 in #2232
Add check for generating manifests and code by @ChenYi015 in #2234
Add changelog for v2.0.2 by @ChenYi015 in #2239
implement an upper bound limit to the number of tracked executor by @ImpSy in #2181
Upgrade to Spark 3.5.3 by @jacobsalway in #2202
Minor fixes to e2e test make targets by @Tom-Newton in #2242
Added off heap memory to calculation for YuniKorn gang scheduling by @guangyu-yang-rokt in #2209
remove redundant test.sh file by @ChenYi015 in #2243
Add permissions to controller serviceaccount to list and watch ingresses by @tcassaert in #2246
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.42 to 1.27.43 by @dependabot[bot] in #2252
Bump manusa/actions-setup-minikube from 2.12.0 to 2.13.0 by @dependabot[bot] in #2247
Bump golang.org/x/net from 0.29.0 to 0.30.0 by @dependabot[bot] in #2251
feat: support archives param for spark-submit by @kaka-zb in #2256
Bump aquasecurity/trivy-action from 0.24.0 to 0.27.0 by @dependabot[bot] in #2248
Bump gocloud.dev from 0.39.0 to 0.40.0 by @dependabot[bot] in #2250
Add Quick Start guide to README by @jacobsalway in #2259
Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.63.3 to 1.65.3 by @dependabot[bot] in #2249
Make sure enable-ui-service flag is set to false when controller.uiService.enable is set to false by @Roberdvs in #2261
omitempty corrections by @Tom-Newton in #2255
Add release badge to README by @jacobsalway in #2263
Bump helm.sh/helm/v3 from 3.16.1 to 3.16.2 by @dependabot[bot] in #2275
Bump github.com/prometheus/client_golang from 1.20.4 to 1.20.5 by @dependabot[bot] in #2274
Bump cloud.google.com/go/storage from 1.44.0 to 1.45.0 by @dependabot[bot] in #2273
Run e2e tests with Kubernetes version matrix by @jacobsalway in #2266
Bump aquasecurity/trivy-action from 0.27.0 to 0.28.0 by @dependabot[bot] in #2270
Allow --ingress-class-name to be specified in chart by @jacobsalway in #2278
Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.65.3 to 1.66.0 by @dependabot[bot] in #2271
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.43 to 1.28.0 by @dependabot[bot] in #2272
Fix retries by @Tom-Newton in #2241
Update default container security context by @ChenYi015 in #2265
Support pod template for Spark 3.x applications by @ChenYi015 in #2141
Add workflow for releasing sparkctl binary by @ChenYi015 in #2264
Update issue and pull request templates by @ChenYi015 in #2287
Allow setting automountServiceAccountToken by @Aransh in #2298
Fix: executor container security context does not work by @ChenYi015 in #2306
Fix: should not add emptyDir sizeLimit conf if it is nil by @ChenYi015 in #2305
Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true by @npgretz in #2282
Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil by @Cian911 in #2316
Bump volcano.sh/apis to 1.10.0 by @jacobsalway in #2320
Truncate UI service name if over 63 characters by @jacobsalway in #2311
Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 by @dependabot[bot] in #2332
Bump github.com/onsi/ginkgo/v2 from 2.20.2 to 2.22.0 by @dependabot[bot] in #2335
The webhook-key-name command-line param isn't taking effect by @c-h-afzal in #2344
Robustness to driver pod taking time to create by @Tom-Newton in #2315
Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh by @Aakcht in #2312
Move sparkctl to cmd directory by @ChenYi015 in #2347
Add changelog for v2.1.0 by @ChenYi015 in #2355
Promoted jacobsalway from reviewer to approver by @yuchaoran2011 in #2361
Bump golang.org/x/net from 0.30.0 to 0.32.0 by @dependabot[bot] in #2350
Bump golang.org/x/crypto from 0.30.0 to 0.31.0 by @dependabot[bot] in #2365
Add Ninja Van to adopters by @hongshaoyang in #2377
Adding seccompProfile RuntimeDefault by @tarekabouzeid in #2397
add an example of using prometheus servlet by @nabuskey in #2403
Remove dependency on k8s.io/kubernetes by @jacobsalway in #2398
fix make deploy and install by @nabuskey in #2412
Add helm unittest step to integration test workflow by @ChenYi015 in #2424
Add option for disabling leader election by @ChenYi015 in #2423
ensure passed context is used by @nabuskey in #2432
Controller should only be granted event permissions in spark job namespaces by @ChenYi015 in #2426
Bump manusa/actions-setup-minikube from 2.13.0 to 2.13.1 by @dependabot[bot] in #2390
Bump helm/chart-testing-action from 2.6.1 to 2.7.0 by @dependabot[bot] in #2391
Bump golang.org/x/mod from 0.21.0 to 0.23.0 by @dependabot[bot] in #2427
Bump github.com/golang/glog from 1.2.2 to 1.2.4 by @dependabot[bot] in #2411
Bump golang.org/x/net from 0.32.0 to 0.35.0 by @dependabot[bot] in #2428
Support Kubernetes 1.32 by @jacobsalway in #2416
docs: Add information about KEP process by @anishasthana in #2440
use cmd context in sparkctl by @nabuskey in #2447
Make image optional by @jbhalodia-slack in #2439
Support non-standard Spark container names by @jbhalodia-slack in #2441
specify branch name in chart testing by @nabuskey in #2451
add support for metrics-job-start-latency-buckets flag in helm by @nabuskey in #2450
fix: webhook fail to add lifecycle to Spark3 executor pods by @pvbouwel in #2458
Bump golang.org/x/net from 0.35.0 to 0.36.0 by @dependabot[bot] in #2470
change env in executorSecretOption by @TQJADE in #2467
Bump aquasecurity/trivy-action from 0.29.0 to 0.30.0 by @dependabot[bot] in #2475
Bump golang.org/x/net from 0.35.0 to 0.37.0 by @dependabot[bot] in #2472
Bump github.com/containerd/containerd from 1.7.19 to 1.7.27 by @dependabot[bot] in #2476
Bump k8s.io/apimachinery from 0.32.0 to 0.32.3 by @dependabot[bot] in #2474
Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.66.0 to 1.78.2 by @dependabot[bot] in #2473
Bump github.com/aws/aws-sdk-go-v2/config from 1.28.0 to 1.29.9 by @dependabot[bot] in #2463
Bump sigs.k8s.io/scheduler-plugins from 0.29.8 to 0.30.6 by @dependabot[bot] in #2444
Add APRA AMCOS to adopters by @shuch3ng in #2485
Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 by @dependabot[bot] in #2488
Bump github.com/prometheus/client_golang from 1.20.5 to 1.21.1 by @dependabot[bot] in #2487
Bump sigs.k8s.io/controller-runtime from 0.20.1 to 0.20.4 by @dependabot[bot] in #2486
Deprecating sparkctl by @vikas-saxena02 in #2484
Upgrade to Spark 3.5.5 by @jacobsalway in #2490
fix: add webhook cert validity checking by @teejaded in #2489
Changing image repo from docker.io to ghcr.io by @vikas-saxena02 in #2483
Add timeZone to ScheduledSparkApplication by @jacobsalway in #2471
Upgrade Golang to 1.24.1 and golangci-lint to 1.64.8 by @jacobsalway in #2494
Bump helm.sh/helm/v3 from 3.16.2 to 3.17.3 by @dependabot[bot] in #2503
Add changelog for v2.1.1 by @ChenYi015 in #2504
Remove sparkctl by @ChenYi015 in #2466
Enable the override of MemoryLimit through webhook by @danielrsfreitas in #2478
Bump github.com/spf13/viper from 1.19.0 to 1.20.1 by @dependabot[bot] in #2496
Bump golang.org/x/net from 0.37.0 to 0.38.0 by @dependabot[bot] in #2505
Remove clientset, informer and listers generated by code-generator by @ChenYi015 in #2506
Remove v1beta1 API by @ChenYi015 in #2516
add unit tests for driver and executor configs by @nabuskey in #2521
Adding securityContext to spark examples by @tarekabouzeid in #2530
Add ShuffleTrackingEnabled to DynamicAllocation struct to allow disabling shuffle tracking by @jbhalodia-slack in #2511
fix and add back unit tests by @nabuskey in #2532
fix volcano tests by @nabuskey in #2533
Bump github.com/spf13/cobra from 1.8.1 to 1.9.1 by @dependabot[bot] in #2497
Bump golang.org/x/mod from 0.23.0 to 0.24.0 by @dependabot[bot] in #2495
Adding Manabu to the reviewers by @vara-bonthu in #2522
Bump manusa/actions-setup-minikube from 2.13.1 to 2.14.0 by @dependabot[bot] in #2523
Define SparkApplicationSubmitter interface to allow customizing submitting mechanism by @ChenYi015 in #2500
Add support for using cert manager to generate webhook certificates by @ChenYi015 in #2373
Add v2 to module path by @ChenYi015 in #2515
Bump k8s.io dependencies to v0.32.5 by @ChenYi015 in #2540
#2525 spark metrics in depends on prometheus by @blcksrx in #2529
Pass the correct LDFLAGS when building the operator image by @ChenYi015 in #2541
Add changelog for v2.2.0 by @ChenYi015 in #2547
add driver ingress unit tests by @nabuskey in #2552
Bump aquasecurity/trivy-action from 0.30.0 to 0.31.0 by @dependabot[bot] in #2557
Customize ingress URL with Spark application ID by @ChenYi015 in #2554
Make default ingress tls and annotations congurable in the helm config by @Tom-Newton in #2513
Get logger from context by @ChenYi015 in #2551
Update golangci lint by @joshuacuellar1 in #2560
Use code-generator for clientset, informers, listers by @jbhalodia-slack in #2563
Bump github.com/prometheus/client_golang from 1.21.1 to 1.22.0 by @dependabot[bot] in #2548
Bump sigs.k8s.io/scheduler-plugins from 0.30.6 to 0.31.8 by @dependabot[bot] in #2549
Bump golang.org/x/mod from 0.24.0 to 0.25.0 by @dependabot[bot] in #2566
Bump github.com/go-logr/logr from 1.4.2 to 1.4.3 by @dependabot[bot] in #2567
Add changelog for v2.2.1 by @ChenYi015 in #2570
chore: Adding OpenSSF Badge by @franciscojavierarceo in #2571
Add support for Spark Connect by @ChenYi015 in #2569
upgrade to Spark 4.0.0 by @nabuskey in #2564
Splat recordExecutorEvent args for cleaner event messages by @matschaffer-roblox in #2582
Add SparkConnect e2e test by @ChenYi015 in #2578
Make logging encoder configurable by @ChenYi015 in #2580
Bump aquasecurity/trivy-action from 0.31.0 to 0.32.0 by @dependabot[bot] in #2585
fix: should add executor env when driver env is empty by @ChenYi015 in #2586
Include pod.Status.Message in recordExecutorEvent by @matschaffer-roblox in #2589
Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 by @dependabot[bot] in #2572
feat(docs): Guide to report security vulnerabilities by @andreyvelich in #2593
Bump sigs.k8s.io/yaml from 1.4.0 to 1.5.0 by @dependabot[bot] in #2577
Add print columns for Spark Connect by @nabuskey in #2592
Bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 by @dependabot[bot] in #2587
chore: update prometheus pattern and labels for structured streaming driver by @yahwang in #2581
Add Helm hook to upgrade CRDs by @ChenYi015 in #2371
Read Helm version from go.mod file by @ChenYi015 in #2598
Add web UI configurations when enabling UI service and ingress by @ChenYi015 in #2599
Bump golang.org/x/mod from 0.25.0 to 0.26.0 by @dependabot[bot] in #2608
Bump github.com/onsi/gomega from 1.36.1 to 1.37.0 by @dependabot[bot] in #2607
Grant get/patch permissions for SparkConnect CRD to hook ClusterRole by @ChenYi015 in #2605
Add changelog for v2.3.0 by @ChenYi015 in #2614
Grant create events permissions to Controller by @Rockawear in #2616
Bump actions/checkout from 4 to 5 by @dependabot[bot] in #2623
Bump actions/download-artifact from 4 to 5 by @dependabot[bot] in #2624
Bump helm.sh/helm/v3 from 3.18.4 to 3.18.5 by @dependabot[bot] in #2627
Update README by @ChenYi015 in #2634
fix(chart): add revisionHistoryLimit option by @t3mi in #2625
Bump github.com/go-viper/mapstructure/v2 from 2.3.0 to 2.4.0 by @dependabot[bot] in #2635
fix(connect): propagate executor memory setting to spark options by @mrjoe7 in #2656
[fix] add miss cli params. Add leader election params in chart by @aagumin in #2657
Suspend/Resume feature on SparkApplication by @everpeace in #2387
Bump actions/setup-go from 5 to 6 by @dependabot[bot] in #2651
Bump actions/stale from 9 to 10 by @dependabot[bot] in #2649
fix: webhook encoder configuration by @pvbouwel in #2664
Bump aquasecurity/trivy-action from 0.32.0 to 0.33.1 by @dependabot[bot] in #2650
Bump github.com/golang/glog from 1.2.4 to 1.2.5 by @dependabot[bot] in #2620
bugfix: hook.upgradeCrd use cases by @pvbouwel in #2663
Bump github.com/onsi/ginkgo/v2 from 2.23.3 to 2.26.0 by @dependabot[bot] in #2660
Add Spark Operator Roadmap by @ChenYi015 in #2622
Correct entrypoint.sh for Openshift by @Reamer in #2645
Bump sigs.k8s.io/scheduler-plugins from 0.31.8 to 0.32.7 by @dependabot[bot] in #2675
Bump github/codeql-action from 3 to 4 by @dependabot[bot] in #2672
Set ControllerReference on driver pod and non-Controller OwnerReference on executor pod at submission time by @everpeace in #2670
CI: Add Scorecard workflow by @AdamKorcz in #2654
Bump github.com/stretchr/testify from 1.10.0 to 1.11.1 by @dependabot[bot] in #2673
unify SparkApplication defaulting logic by @zhzhuang-zju in #2671
Update GitHub action workflow permissions by @ChenYi015 in #2678
Bump golang.org/x/time from 0.9.0 to 0.14.0 by @dependabot[bot] in #2686
Bump github.com/prometheus/client_golang from 1.22.0 to 1.23.2 by @dependabot[bot] in #2682
Bump github.com/spf13/viper from 1.20.1 to 1.21.0 by @dependabot[bot] in #2683
Add kube-scheduler podgroup unit tests by @shadowinlife in #2689
Bump github/codeql-action from 3.29.2 to 4.31.0 by @dependabot[bot] in #2684
Generate HTML coverage report after running unit tests by @shadowinlife in #2691
Avoid 409 (Conflict) error at e2e tests with retry. by @shadowinlife in #2695
Add unit test for sparkapplication_validator by @shadowinlife in #2692
Add unit test for scheduledsparkapplication_validator by @shadowinlife in #2694
Refactor the unit tests for web_ui.go by @shadowinlife in #2696
Replace strconv.Atoi with strconv.ParseInt by @ChenYi015 in #2699
refactor: use ptr.To to replace util.XxxPtr by @ChenYi015 in #2693
added SparkApp name validator to accept valid DNS-1035 format by @aryankumar04 in #2711
Bump github.com/containerd/containerd from 1.7.27 to 1.7.29 by @dependabot[bot] in #2712
No propagate Kueue labels to driver and executor pods by @everpeace in #2714
Bump sigs.k8s.io/yaml from 1.5.0 to 1.6.0 by @dependabot[bot] in #2704
Bump github.com/spf13/cobra from 1.9.1 to 1.10.1 by @dependabot[bot] in #2705
Bump github/codeql-action from 4.31.0 to 4.31.2 by @dependabot[bot] in #2707
feat: add support for customizing connect service by @ChenYi015 in #2709
Bump actions/upload-artifact from 4 to 5 by @dependabot[bot] in #2681
Bump actions/download-artifact from 5 to 6 by @dependabot[bot] in #2680
Bump actions/checkout from 4.2.2 to 5.0.0 by @dependabot[bot] in #2685
Bump ossf/scorecard-action from 2.4.2 to 2.4.3 by @dependabot[bot] in #2687
Fix driver host configuration to handle IPv6 addresses by @tiagotxm in #2703
Bump github.com/onsi/ginkgo/v2 from 2.26.0 to 2.27.2 by @dependabot[bot] in #2719
Bump golang.org/x/mod from 0.27.0 to 0.29.0 by @dependabot[bot] in #2720
Bump helm/chart-testing-action from 2.7.0 to 2.8.0 by @dependabot[bot] in #2722
Add permissions for sparkconnects/finalizers by @ChenYi015 in #2727
Remove redundant name and namespace context in logs by @ChenYi015 in #2723
Logging info when resources associated with SparkApplication still exist by @ChenYi015 in #2725
Add changelog for v2.4.0 by @ChenYi015 in #2731
feat(chart): add hostUsers (user namespace) option by @t3mi in #2721
Add GitHub action workflow for publishing latest spark-operator and kubectl images by @ChenYi015 in #2710
Update ENVTEST_K8S_VERSION for e2e and unit tests by @ChenYi015 in #2751
Reset SparkApplication status when transitioning from SUCCEEDING/FAILING to PENDING_RERUN by @ChenYi015 in #2773
Use custom log constructors for controllers by @ChenYi015 in #2756
Bump golang from 1.24.1 to 1.24.10 by @ChenYi015 in #2732
Bump actions/download-artifact from 5 to 7 by @dependabot[bot] in #2776
Bump actions/upload-artifact from 4 to 6 by @dependabot[bot] in #2777
Bump github/codeql-action from 4.31.2 to 4.31.8 by @dependabot[bot] in #2778
Helm: Add toleration, affinity, and nodeselector for upgrade hook by @MonkeyCanCode in #2780
feat: introduce feature gate mechanism by @Kevinz857 in #2794
feat: skip reconcile for webhook-patched executor fields by @Kevinz857 in #2786
Fix : Propagate batch scheduler initialization errors to trigger retries by @houyuting in #2783
Server-side apply CRDs by forcing conflicts by @ChenYi015 in #2800
refactor: replace util.ContainsString with slices.Contains by @ChenYi015 in #2796
Refactor webhook logging by @ChenYi015 in #2799
feature: Add alpha feature gate LoadSparkDefaults by @ChenYi015 in #2798
Update to spark 4.0.1 by @nabuskey in #2804
Fix SparkConnect nil executor template panic by @sameerdattav in #2814
feat: watch namespaces based on labels by @tariq-hasan in #2808
Bump golang.org/x/crypto from 0.42.0 to 0.45.0 by @dependabot[bot] in #2743
refactor: reuse version.Info from k8s.io/component-base/version by @ChenYi015 in #2797
feat: Expose metrics-labels flag via Spark Operator Helm chart by @IgorSteps in #2817
Set appProtocol on SparkConnect server service ports by @wkd-woo in #2823
feat: add welcome-new-contributors workflow by @ChenYi015 in #2824
docs(readme): update Slack channel name in community section by @ChenYi015 in #2829
fix: correct filename typo in metrics package by @puwun in #2830
fix: handle nil Executor.Instances in GetExecutorRequestResource by @puwun in #2834
refactor: replace glog with controller-runtime structured logging by @puwun in #2836
fix: correct schedule parse error logging in ScheduledSparkApplication controller by @puwun in #2841
feat: add configurable timestamp precision for ScheduledSparkApplication names by @puwun in #2827
feat: add Python API generation for Spark Operator CRDs by @tariq-hasan in #2828
chore(api): bump Python API version to 2.4.0 by @tariq-hasan in #2853
Release v2.5.0-rc.0 by @vara-bonthu in #2854
Fix duplicate webhook patch and add missing ScheduledSparkApplication Patches by @RobuRishabh in #2820
feat(webhook): Add webhook validation for SparkConnect CRD by @tariq-hasan in #2862
fix: Add label selector to ConfigMap cache to prevent OOM via informer flooding by @RobuRishabh in #2878
chore(deps): bump actions/upload-artifact from 6.0.0 to 7.0.0 by @dependabot[bot] in #2861
Bump actions/checkout from 5.0.0 to 6.0.2 by @dependabot[bot] in #2821
Update integration test k8s matrix by @nabuskey in #2874
chore(deps): bump aquasecurity/trivy-action from 0.33.1 to 0.34.0 in /.github/workflows by @dependabot[bot] in #2847
chore(deps): bump google.golang.org/grpc from 1.69.4 to 1.79.3 by @dependabot[bot] in #2884
fix: avoid duplicate web UI service/ingress creation during reconciliation by @YassinNouh21 in #2868
feat(controller): add --kube-api-qps and --kube-api-burst flags by @houyuting in #2865
pin github actions to hashes by @nabuskey in #2892
chore(ci): Remove trivy - Spark-Operator by @sameerdattav in #2894
chore(ci): Remove Trivy Action by @nabuskey in #2893
fix(crd): Add missing SparkConnect CRD resource and patches to kustomization by @RobuRishabh in #2890
fix(rbac): Replace hardcoded namespace: default with namespace: system by @RobuRishabh in #2898
feat(pdb): Add PodDisruptionBudget Kustomize resource for controller-manager by @Vedant-Deshpande in #2918
fix(rbac): Tighten controller RBAC markers to enforce least privilege by @RobuRishabh in #2914
fix(rbac): Fix Helm RBAC bugs, add trailing newline, and add unit test by @RobuRishabh in #2917
feat(prometheus): Add Prometheus PodMonitor Kustomize resource by @Vedant-Deshpande in #2922
feat(kustomize): Add core deployment infrastructure and E2E test suite by @RobuRishabh in #2919
fix: EmptyDir medium field is not forwarded to Spark configuration by @MathildeLeroi in #2925
Chore/remove standalone kustomize by @RobuRishabh in #2929
fix(webhook): Fix path for mutating ScheduledSparkApplication, correct sideEffects, remove update verb in manifests by @Vedant-Deshpande in #2915
fix(kustomize): Add namespaceSelector to webhook configurations via kustomize patches by @RobuRishabh in #2941
fix(kustomize,helm): Align Helm and Kustomize RBAC, args, and sideEffects via code tracing by @RobuRishabh in #2936
Promote nabuskey to approver by @nabuskey in #2944
Add spark_application_submit_latency_seconds metric by @venkomirisetti in #2942
fix(metrics): use sanitized labels for executor/application metric vecs by @dineshkumar181094 in #2931
feat(ci): Implement drift detection as Go test in test/drift/ by @RobuRishabh in #2933
fix(kustomize): Add missing --metrics-job-submit-latency-buckets arg to controller deployment by @RobuRishabh in #2950
Fix(SparkConnect): Add BlockManager port to match Service definition by @Vedant-Deshpande in #2948
promote roburishabh to reviewers by @RobuRishabh in #2969
feat(controller): emit Kubernetes event when driver pod missing after grace period by @RobuRishabh in #2971
fix(controller): avoid invalidating apps on TTL updates by @dineshkumar181094 in #2956
Replace tini in entrypoint.sh with catatonit by @alimaredia in #2968
Add labels and annotations support for upgrade hook by @MonkeyCanCode in #2940
fix: handle uppercase units in ConvertJavaMemoryStringToK8sMemoryString by @tariq-hasan in #2937
feat(controller): log when SparkApplication is marked Failing after driver-pod grace period by @dineshkumar181094 in #2947
docs(readme): Add installation instructions via kustomize manifests by @alimaredia in #2951
(chore) use command in entrypoint.sh by @alimaredia in #2975
Use driverIngressOptions annotations and TLS instead of sparkUIOptions by @shruthis4 in #2973
fix(kustomize): Add webhook subPath and expand Helm/Kustomize drift checks by @RobuRishabh in #2982
feat: add TLS version and cipher suite configuration flags by @ugiordan in #2986
Bump k8s to v1.35.4 and controller-runtime to v0.23.3 by @everpeace in #2946
REST-based spark submitter service as pluggable submission strategy by @venkomirisetti in #2976
Bump Go to 1.25.11 by @nabuskey in #2998
tighten GH workflows permissions by @nabuskey in #2981
docs: Add dedicated Spark Operator documentation website by @vara-bonthu in #2988
Add dependabot config for website by @nabuskey in #3007
fix: always set NextProtos regardless of enableHTTP2 flag by @ugiordan in #2990
feat(chart): make Helm hook Job resources configurable by @maxpschonder in #3015
feat: Add Spark Connect server probes by @Shekharrajak in #2995
ci: add shfmt and shellcheck checks for shell scripts by @tariq-hasan in #2993
chore(deps): bump docker/login-action from 3.7.0 to 4.2.0 by @dependabot[bot] in #3008
feat(helm): add RestSubmitter feature gate with controller integration by @venkomirisetti in #2999
Add deploymentAnnotations to controller and webhook deployments by @cHiv0rz in #3001
Scope make e2e-test to a dedicated Kind cluster by @nabuskey in #3004
docs: point documentation links to spark.kubeflow.org by @vara-bonthu in #3020
feat: add per-application driver PodDisruptionBudget (#2949) by @vjanelle in #2954
Scope codegen paths to workload-operator packages by @RobuRishabh in #3021
Update setup-helm github action versions by @nabuskey in #3030
fix(chart): grant webhook leader election event permissions by @immanuwell in #3027
chore(deps): bump golang.org/x/crypto from 0.46.0 to 0.52.0 by @dependabot[bot] in #3039
test(e2e): add ScheduledSparkApplication workload e2e test by @sahana-sreeram in #3029
fix: harden controller-runtime cache against OOMKill vulnerabilities by @RobuRishabh in #3032
feat(webhook): add sparkConf key validation by @sahana-sreeram in #3043
chore(deps): bump google.golang.org/grpc from 1.79.3 to 1.82.1 by @dependabot[bot] in #3054
adding my company RAICS.AI to adopters list by @vikas-saxena02 in #3056
update spark image to 4.0.4 by @nabuskey in #3052
chore: Adding Tariq Hasan as new reviewer to the repo by @vara-bonthu in #3064
ci: add gh-workflow-approve workflow for /ok-to-test on fork PRs by @RobuRishabh in #3066
fix: remove unused Provider.TLSConfig() method by @ugiordan in #3062
test(e2e): add SparkConnect query e2e test by @sahana-sreeram in #3031
test(e2e): add Prometheus metrics and Spark UI e2e tests by @sahana-sreeram in #3047
(Chore): Move e2e tests into own submodule for dependency isolation by @vjanelle in #3071
chore(deps): bump github.com/containerd/containerd from 1.7.30 to 1.7.33 in /test/e2e by @dependabot[bot] in #3087
chore(deps): bump oras.land/oras-go/v2 from 2.6.1 to 2.6.2 in /test/e2e by @dependabot[bot] in #3088
chore(deps): bump github.com/onsi/gomega from 1.38.2 to 1.42.1 in /test/e2e by @dependabot[bot] in #3085
fix: preserve shell-sensitive Spark Connect configuration values by @jayanth86 in #3051
Chore(Remove go card from readme) - #3058 by @Tankdoggo in #3060
chore(deps): bump github.com/prometheus/common from 0.67.5 to 0.70.1 in /test/e2e by @dependabot[bot] in #3083
chore(deps): bump actions/checkout from 6.0.2 to 7.0.1 by @dependabot[bot] in #3086
feat(ci): Publish Spark Operator binaries with GitHub releases by @vikas-saxena02 in #3078
feat: garbage-collect terminated SparkApplications via operator default TTL by @dineshkumar181094 in #3042
chore(deps): bump github.com/onsi/ginkgo/v2 from 2.27.2 to 2.32.1 in /test/e2e by @dependabot[bot] in #3095
docs: document webhook failure when a service mesh sidecar is injected by @vikas-saxena02 in #3100
fix(webhook): generate valid volume names for ConfigMaps by @immanuwell in #3099
feat(ci): publish the operator binary extracted from the released image by @vikas-saxena02 in #3098
fix(sparkconnect): resolve images from selected template containers by @immanuwell in #3103
docs: add Spark Connect user guide by @reckless-sherixx in #3091
feat(controller): add default service account fallback for SparkApplication driver by @vikas-saxena02 in #3092
fix(webhook): generate valid volume name for Prometheus ConfigMap by @rjgoyln in #3111
refactor: centralize Spark container selection by @rohith1X8 in #3115
feat(sparkconnect): add service account fallback for Spark Connect server pod by @vikas-saxena02 in #3109
chore(ci): migrate vulnerability scanning to OSV-Scanner by @vikas-saxena02 in #3110
chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 in /test/e2e by @dependabot[bot] in #3136
chore(deps): bump tornado from 6.5.7 to 6.5.8 in /docs/website by @dependabot[bot] in #3135
Make webhook leader election timings configurable by @AxcelBlaze003 in #3065
fix(webhook): validate ConfigMap references in SparkApplication specs by @rjgoyln in #3117
docs: replace Enterprise Gateway notebooks guide with Kubeflow SDK by @vikas-saxena02 in #3141
docs: add SparkConnect API reference to api-docs.md by @Fumer057 in #3126
fix(webhook): validate ConfigMap mount-path uniqueness by @pujitha24 in #3140
docs: add SparkConnect to the website API reference by @sxivansx in #3138
fix: stop adding SparkApplication owner reference to executor pods by @KR-Ravindra in #3152
test(util): add unit tests for CreateValidMetricNameLabel by @magic-peach in #3159
chore: remove unused pkg/util/interrupt.go by @pujitha24 in #3162
feat(sparkconnect): support GPU resources in SparkConnect CRD by @Rajadi16 in #3169
fix(sparkconnect): pass server options as individual container arguments by @vikas-saxena02 in #3188
fix(controller): always clean up pod template files by @Aakcht in #3178
fix(sparkconnect): update GPU specs for per-argument container args by @vikas-saxena02 in #3212
test(util): add unit tests for InitializeIngressCapabilities by @magic-peach in #3156
fix: default the retry interval when the webhook has not set it by @pratik-naik003 in #3209
test: add unit tests for SumResourceList by @pratik-naik003 in #3200
docs: clarify inherited dependency security boundary by @thisis-Shitanshu in #3214
chore(deps): bump urllib3 from 2.7.0 to 2.8.0 in /docs/website by @dependabot[bot] in #3220
chore(deps): bump tornado from 6.5.8 to 6.5.9 in /docs/website by @dependabot[bot] in #3219
fix: name the actual flag in TLS setup errors by @pratik-naik003 in #3204
fix(scheduledsparkapplication): recover from FailedValidation state when spec is corrected by @rautaditya2606 in #3175
docs: add REST submitter integration guide by @venkomirisetti in #3216
feat: add support for managedBy field on SparkApplication by @amirialy in #3221
test: add unit tests for LogConstructor by @magic-peach in #3181
docs: add AGENTS.md and CLAUDE.md for AI coding agents by @Abhayanthk in #3192
feat!: Add webhook namespace-selector for multi-tenant deployments by @RobuRishabh in #3217
docs: add 2026 roadmap by @tariq-hasan in #3179
feat(controller): allow Spark 4.2 NetworkPolicy creation by @thisis-Shitanshu in #3240
test(e2e): run SparkConnect query on the Connect server, not a local … by @vikas-saxena02 in #3239
fix(controller): authorize events.k8s.io for SparkApplication events by @RobuRishabh in #3227
Adding Vikas and Vincent to reviewers by @vikas-saxena02 in #3246
fix(controller): mark missing executors completed when the driver succeeds by @rishabhsai in #3244
Add comment policy to AGENTS.md by @nabuskey in #3247
feat: support physical CPU request and limit in SparkConnect CRD by @DUKartik in #3096
feat: Expose Spark Operator control-plane version via public ConfigMap by @tariq-hasan in #3225
chore(deps): bump certifi from 2026.5.20 to 2026.10.19 in /docs/website by @dependabot[bot] in #3255
docs: show how to set runtimeClassName with pod templates by @highlyavailable in #3248
fix(scheduledsparkapplication): don't panic on negative history limits by @rautaditya2606 in #3229
feat(ci): add required boilerplate (license header) check by @reckless-sherixx in #3213
test(scheduler): add unit tests for the batch scheduler registry by @vikas-saxena02 in #3232
feat(sparkconnect): add a Dependencies field to SparkConnectSpec by @Mr-Neutr0n in #3120
chore: fix CI failures on master (boilerplate headers, ubuntu-26.04 runner) by @vikas-saxena02 in #3271
fix(sparkconnect): use server.cores as server pod CPU request by @vikas-saxena02 in #3252
feat(ci): automate the release process by @tariq-hasan in #3226
feat: upgrade Spark runtime to 4.2.1 by @thisis-Shitanshu in #3245
feat(api): deprecate client deploy mode for SparkApplication by @vikas-saxena02 in #3270
chore: release v2.6.0-rc.0 by @tariq-hasan in #3287

New Contributors

@xiyoubird made their first contribution in #2128
@dependabot[bot] made their first contribution in #2159
@Kevinz857 made their first contribution in #2146
@tcassaert made their first contribution in #2166
@ha2hi made their first contribution in #2157
@missedone made their first contribution in #2222
@Tom-Newton made their first contribution in #2242
@guangyu-yang-rokt made their first contribution in #2209
@kaka-zb made their first contribution in #2256
@Roberdvs made their first contribution in #2261
@npgretz made their first contribution in #2282
@c-h-afzal made their first contribution in #2344
@tarekabouzeid made their first contribution in #2397
@anishasthana made their first contribution in #2440
@pvbouwel made their first contribution in #2458
@TQJADE made their first contribution in #2467
@shuch3ng made their first contribution in #2485
@teejaded made their first contribution in #2489
@danielrsfreitas made their first contribution in #2478
@blcksrx made their first contribution in #2529
@joshuacuellar1 made their first contribution in #2560
@franciscojavierarceo made their first contribution in #2571
@yahwang made their first contribution in #2581
@Rockawear made their first contribution in #2616
@mrjoe7 made their first contribution in #2656
@aagumin made their first contribution in #2657
@everpeace made their first contribution in #2387
@Reamer made their first contribution in #2645
@AdamKorcz made their first contribution in #2654
@zhzhuang-zju made their first contribution in #2671
@shadowinlife made their first contribution in #2689
@aryankumar04 made their first contribution in #2711
@tiagotxm made their first contribution in #2703
@MonkeyCanCode made their first contribution in #2780
@houyuting made their first contribution in #2783
@sameerdattav made their first contribution in #2814
@tariq-hasan made their first contribution in #2808
@IgorSteps made their first contribution in #2817
@wkd-woo made their first contribution in #2823
@puwun made their first contribution in #2830
@RobuRishabh made their first contribution in #2820
@YassinNouh21 made their first contribution in #2868
@Vedant-Deshpande made their first contribution in #2918
@MathildeLeroi made their first contribution in #2925
@venkomirisetti made their first contribution in #2942
@dineshkumar181094 made their first contribution in #2931
@alimaredia made their first contribution in #2968
@shruthis4 made their first contribution in #2973
@ugiordan made their first contribution in #2986
@maxpschonder made their first contribution in #3015
@Shekharrajak made their first contribution in #2995
@cHiv0rz made their first contribution in #3001
@vjanelle made their first contribution in #2954
@immanuwell made their first contribution in #3027
@sahana-sreeram made their first contribution in #3029
@jayanth86 made their first contribution in #3051
@Tankdoggo made their first contribution in #3060
@reckless-sherixx made their first contribution in #3091
@rjgoyln made their first contribution in #3111
@rohith1X8 made their first contribution in #3115
@AxcelBlaze003 made their first contribution in #3065
@Fumer057 made their first contribution in #3126
@pujitha24 made their first contribution in #3140
@sxivansx made their first contribution in #3138
@KR-Ravindra made their first contribution in #3152
@magic-peach made their first contribution in #3159
@Rajadi16 made their first contribution in #3169
@pratik-naik003 made their first contribution in #3209
@thisis-Shitanshu made their first contribution in #3214
@rautaditya2606 made their first contribution in #3175
@amirialy made their first contribution in #3221
@Abhayanthk made their first contribution in #3192
@rishabhsai made their first contribution in #3244
@DUKartik made their first contribution in #3096
@highlyavailable made their first contribution in #3248
@Mr-Neutr0n made their first contribution in #3120

Full Changelog: spark-operator-chart-1.4.6...v2.6.0-rc.0