kubeflow/spark-operator v2.6.0

on GitHub

Breaking Changes

• feat!: Add webhook namespace-selector for multi-tenant deployments (#3217 by @RobuRishabh)
  • spark.jobNamespaceSelector no longer accepts a string; it must be a label selector object. The webhook now uses the same namespace matcher as the controller.

Deprecations

• feat(api): deprecate client deploy mode for SparkApplication (#3270 by @vikas-saxena02)
  • spec.mode: client still works but is deprecated; the webhook returns a warning and the controller logs one on submission.

Features

• feat: upgrade Spark runtime to 4.2.1 (#3245 by @thisis-Shitanshu)
• feat(sparkconnect): add a Dependencies field to SparkConnectSpec (#3120 by @Mr-Neutr0n)
• feat: Expose Spark Operator control-plane version via public ConfigMap (#3225 by @tariq-hasan)
• feat: support physical CPU request and limit in SparkConnect CRD (#3096 by @DUKartik)
• feat(controller): allow Spark 4.2 NetworkPolicy creation (#3240 by @thisis-Shitanshu)
• feat: add support for managedBy field on SparkApplication (#3221 by @amirialy)
• feat(sparkconnect): support GPU resources in SparkConnect CRD (#3169 by @Rajadi16)
• Make webhook leader election timings configurable (#3065 by @AxcelBlaze003)
• feat(sparkconnect): add service account fallback for Spark Connect server pod (#3109 by @vikas-saxena02)
• feat(controller): add default service account fallback for SparkApplication driver (#3092 by @vikas-saxena02)
• feat: garbage-collect terminated SparkApplications via operator default TTL (#3042 by @dineshkumar181094)
• feat: add per-application driver PodDisruptionBudget (#2949) (#2954 by @vjanelle)
• Add deploymentAnnotations to controller and webhook deployments (#3001 by @cHiv0rz)
• feat(helm): add RestSubmitter feature gate with controller integration (#2999 by @venkomirisetti)
• feat: Add Spark Connect server probes (#2995 by @Shekharrajak)
• feat(chart): make Helm hook Job resources configurable (#3015 by @maxpschonder)
• REST-based spark submitter service as pluggable submission strategy (#2976 by @venkomirisetti)
• feat: add TLS version and cipher suite configuration flags (#2986 by @ugiordan)
• feat(controller): log when SparkApplication is marked Failing after driver-pod grace period (#2947 by @dineshkumar181094)
• Add labels and annotations support for upgrade hook (#2940 by @MonkeyCanCode)
• feat(controller): emit Kubernetes event when driver pod missing after grace period (#2971 by @RobuRishabh)

Bug Fixes

• fix(controller): resolve {{SPARK_VERSION}} in images with the application's Spark version (#3292 by @vara-bonthu)
• fix(kustomize): remove stale OPERATOR_VERSION from controller deployment (#3291 by @tariq-hasan)
• fix(sparkconnect): use server.cores as server pod CPU request (#3252 by @vikas-saxena02)
• fix(scheduledsparkapplication): don't panic on negative history limits (#3229 by @rautaditya2606)
• fix(controller): mark missing executors completed when the driver succeeds (#3244 by @rishabhsai)
• fix(controller): authorize events.k8s.io for SparkApplication events (#3227 by @RobuRishabh)
• fix(scheduledsparkapplication): recover from FailedValidation state when spec is corrected (#3175 by @rautaditya2606)
• fix: name the actual flag in TLS setup errors (#3204 by @pratik-naik003)
• fix: default the retry interval when the webhook has not set it (#3209 by @pratik-naik003)
• fix(sparkconnect): update GPU specs for per-argument container args (#3212 by @vikas-saxena02)
• fix(controller): always clean up pod template files (#3178 by @Aakcht)
• fix(sparkconnect): pass server options as individual container arguments (#3188 by @vikas-saxena02)
• fix: stop adding SparkApplication owner reference to executor pods (#3152 by @KR-Ravindra)
• fix(webhook): validate ConfigMap mount-path uniqueness (#3140 by @pujitha24)
• fix(webhook): validate ConfigMap references in SparkApplication specs (#3117 by @rjgoyln)
• fix(webhook): generate valid volume name for Prometheus ConfigMap (#3111 by @rjgoyln)
• fix(sparkconnect): resolve images from selected template containers (#3103 by @immanuwell)
• fix(webhook): generate valid volume names for ConfigMaps (#3099 by @immanuwell)
• fix: preserve shell-sensitive Spark Connect configuration values (#3051 by @jayanth86)
• fix: harden controller-runtime cache against OOMKill vulnerabilities (#3032 by @RobuRishabh)
• fix: always set NextProtos regardless of enableHTTP2 flag (#2990 by @ugiordan)
• fix(kustomize): Add webhook subPath and expand Helm/Kustomize drift checks (#2982 by @RobuRishabh)
• Use driverIngressOptions annotations and TLS instead of sparkUIOptions (#2973 by @shruthis4)
• fix: handle uppercase units in ConvertJavaMemoryStringToK8sMemoryString (#2937 by @tariq-hasan)

Refactor

• chore: remove unused pkg/util/interrupt.go (#3162 by @pujitha24)
• refactor: centralize Spark container selection (#3115 by @rohith1X8)
• fix: remove unused Provider.TLSConfig() method (#3062 by @ugiordan)
• Scope codegen paths to workload-operator packages (#3021 by @RobuRishabh)

Tests

• test(scheduler): add unit tests for the batch scheduler registry (#3232 by @vikas-saxena02)
• test(e2e): run SparkConnect query on the Connect server, not a local … (#3239 by @vikas-saxena02)
• test: add unit tests for LogConstructor (#3181 by @magic-peach)
• test: add unit tests for SumResourceList (#3200 by @pratik-naik003)
• test(util): add unit tests for InitializeIngressCapabilities (#3156 by @magic-peach)
• test(util): add unit tests for CreateValidMetricNameLabel (#3159 by @magic-peach)
• (Chore): Move e2e tests into own submodule for dependency isolation (#3071 by @vjanelle)
• test(e2e): add Prometheus metrics and Spark UI e2e tests (#3047 by @sahana-sreeram)
• test(e2e): add SparkConnect query e2e test (#3031 by @sahana-sreeram)
• test(e2e): add ScheduledSparkApplication workload e2e test (#3029 by @sahana-sreeram)
• Scope make e2e-test to a dedicated Kind cluster (#3004 by @nabuskey)

Documentation

• docs: show how to set runtimeClassName with pod templates (#3248 by @highlyavailable)
• Add comment policy to AGENTS.md (#3247 by @nabuskey)
• docs: add 2026 roadmap (#3179 by @tariq-hasan)
• docs: add AGENTS.md and CLAUDE.md for AI coding agents (#3192 by @Abhayanthk)
• docs: add REST submitter integration guide (#3216 by @venkomirisetti)
• docs: clarify inherited dependency security boundary (#3214 by @thisis-Shitanshu)
• docs: add SparkConnect to the website API reference (#3138 by @sxivansx)
• docs: add SparkConnect API reference to api-docs.md (#3126 by @Fumer057)
• docs: replace Enterprise Gateway notebooks guide with Kubeflow SDK (#3141 by @vikas-saxena02)
• docs: add Spark Connect user guide (#3091 by @reckless-sherixx)
• docs: document webhook failure when a service mesh sidecar is injected (#3100 by @vikas-saxena02)
• Chore(Remove go card from readme) - #3058 (#3060 by @Tankdoggo)
• adding my company RAICS.AI to adopters list (#3056 by @vikas-saxena02)
• docs: point documentation links to spark.kubeflow.org (#3020 by @vara-bonthu)
• docs: Add dedicated Spark Operator documentation website (#2988 by @vara-bonthu)

CI and Release

• feat(ci): automate the release process (#3226 by @tariq-hasan)
• chore: fix CI failures on master (boilerplate headers, ubuntu-26.04 runner) (#3271 by @vikas-saxena02)
• feat(ci): add required boilerplate (license header) check (#3213 by @reckless-sherixx)
• chore(ci): migrate vulnerability scanning to OSV-Scanner (#3110 by @vikas-saxena02)
• feat(ci): publish the operator binary extracted from the released image (#3098 by @vikas-saxena02)
• feat(ci): Publish Spark Operator binaries with GitHub releases (#3078 by @vikas-saxena02)
• ci: add gh-workflow-approve workflow for /ok-to-test on fork PRs (#3066 by @RobuRishabh)
• Update setup-helm github action versions (#3030 by @nabuskey)
• ci: add shfmt and shellcheck checks for shell scripts (#2993 by @tariq-hasan)
• Add dependabot config for website (#3007 by @nabuskey)
• tighten GH workflows permissions (#2981 by @nabuskey)

Dependencies

• chore(deps): bump certifi from 2026.5.20 to 2026.10.19 in /docs/website (#3255 by @dependabot[bot])
• chore(deps): bump tornado from 6.5.8 to 6.5.9 in /docs/website (#3219 by @dependabot[bot])
• chore(deps): bump urllib3 from 2.7.0 to 2.8.0 in /docs/website (#3220 by @dependabot[bot])
• chore(deps): bump tornado from 6.5.7 to 6.5.8 in /docs/website (#3135 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 in /test/e2e (#3136 by @dependabot[bot])
• chore(deps): bump github.com/onsi/ginkgo/v2 from 2.27.2 to 2.32.1 in /test/e2e (#3095 by @dependabot[bot])
• chore(deps): bump actions/checkout from 6.0.2 to 7.0.1 (#3086 by @dependabot[bot])
• chore(deps): bump github.com/prometheus/common from 0.67.5 to 0.70.1 in /test/e2e (#3083 by @dependabot[bot])
• chore(deps): bump github.com/onsi/gomega from 1.38.2 to 1.42.1 in /test/e2e (#3085 by @dependabot[bot])
• chore(deps): bump oras.land/oras-go/v2 from 2.6.1 to 2.6.2 in /test/e2e (#3088 by @dependabot[bot])
• chore(deps): bump github.com/containerd/containerd from 1.7.30 to 1.7.33 in /test/e2e (#3087 by @dependabot[bot])
• chore(deps): bump google.golang.org/grpc from 1.79.3 to 1.82.1 (#3054 by @dependabot[bot])
• chore(deps): bump golang.org/x/crypto from 0.46.0 to 0.52.0 (#3039 by @dependabot[bot])
• chore(deps): bump docker/login-action from 3.7.0 to 4.2.0 (#3008 by @dependabot[bot])
• Bump Go to 1.25.11 (#2998 by @nabuskey)
• Bump k8s to v1.35.4 and controller-runtime to v0.23.3 (#2946 by @everpeace)

Misc

• Adding Vikas and Vincent to reviewers (#3246 by @vikas-saxena02)
• chore: Adding Tariq Hasan as new reviewer to the repo (#3064 by @vara-bonthu)
• (chore) use command in entrypoint.sh (#2975 by @alimaredia)
• Replace tini in entrypoint.sh with catatonit (#2968 by @alimaredia)
• promote roburishabh to reviewers (#2969 by @RobuRishabh)

Full Changelog

Installation

helm install spark-operator oci://ghcr.io/kubeflow/helm-charts/spark-operator \
  --version 2.6.0 --namespace spark-operator --create-namespace

Images: ghcr.io/kubeflow/spark-operator/controller:2.6.0, ghcr.io/kubeflow/spark-operator/kubectl:2.6.0

Python API: pip install kubeflow-spark-api==2.6.0