🚀 Spark on Kubernetes Operator v2.5.0

We're excited to announce the release of Spark on Kubernetes Operator v2.5.0! This release introduces
alpha feature gates, namespace label-based watching, Python API generation, SparkConnect webhook validation,
and several important bug fixes including an OOM prevention fix.

✨ New Features

• Feature gate mechanism — Alpha feature gates are now supported, enabling experimental features to be
  toggled safely (#2794)
• LoadSparkDefaults feature gate (Alpha) — Adds --load-spark-defaults to spark-submit for
  Spark 4.0+ (#2798)
• PartialRestart feature gate (Alpha) — Skips reconcile for webhook-patched executor fields,
  reducing unnecessary restarts (#2786)
• Watch namespaces by labels — Operator can now watch namespaces based on label selectors
  (#2808)
• Configurable timestamp precision for ScheduledSparkApplication names
  (#2827)
• Python API generation for Spark Operator CRDs
  (#2828)

🐛 Bug Fixes

• OOM prevention — Add label selector to ConfigMap cache to prevent OOM via informer flooding
  (#2881)
• Fix duplicate webhook patch and add missing ScheduledSparkApplication patches
  (#2875)
• Fix: correct schedule parse error logging in ScheduledSparkApplication controller
  (#2841)
• Fix: handle nil Executor.Instances in GetExecutorRequestResource
  (#2834)
• Fix: correct filename typo in metrics package
  (#2830)
• Fix: propagate batch scheduler initialization errors to trigger retries
  (#2783)
• Reset SparkApplication status when transitioning from SUCCEEDING/FAILING → PENDING_RERUN
  (#2773)
• Server-side apply CRDs by forcing conflicts
  (#2800)
• Fix SparkConnect nil executor template panic
  (#2814)

⛵ Helm Chart

• Add hostUsers (user namespace) option
  (#2721)
• Expose metrics-labels flag via Helm chart
  (#2817)
• Add toleration, affinity, and nodeSelector support for upgrade hook
  (#2780)

⚡ SparkConnect

• Add webhook validation for SparkConnect CRD
  (#2877)
• Set appProtocol on SparkConnect server service ports
  (#2823)
• Add permissions for sparkconnects/finalizers
  (#2727)

🔧 Improvements & Refactoring

• Replace glog with controller-runtime structured logging
  (#2836)
• Refactor webhook logging
  (#2799)
• Use custom log constructors for controllers
  (#2756)
• Replace util.ContainsString with slices.Contains
  (#2796)
• Reuse version.Info from k8s.io/component-base/version
  (#2797)
• Remove redundant name and namespace context in logs
  (#2723)
• Log info when resources associated with SparkApplication still exist
  (#2725)
• Add welcome-new-contributors GitHub Actions workflow
  (#2824)
• Add GitHub Actions workflow for publishing latest images
  (#2710)
• Update ENVTEST_K8S_VERSION for e2e and unit tests
  (#2751)
• Update Slack channel name in community section
  (#2829)

📦 Dependencies

• Update to Spark 4.0.1 (#2804)
• Bump Go from 1.24.1 → 1.24.10 (#2732)
• Bump golang.org/x/crypto from 0.42.0 → 0.45.0 (#2743)
• Bump golang.org/x/mod from 0.27.0 → 0.29.0 (#2720)
• Bump Python API version to 2.4.0 (#2853)
• Bump github.com/onsi/ginkgo/v2 from 2.26.0 → 2.27.2 (#2719)
• Bump actions/download-artifact from 5 → 7 (#2776)
• Bump actions/upload-artifact from 4 → 6 (#2777)
• Bump github/codeql-action from 4.31.2 → 4.31.8 (#2778)
• Bump helm/chart-testing-action from 2.7.0 → 2.8.0 (#2722)

🌟 New Contributors

• @MonkeyCanCode made their first contribution in #2780
• @sameerdattav made their first contribution in #2814
• @IgorSteps made their first contribution in #2817
• @wkd-woo made their first contribution in #2823
• @puwun made their first contribution in #2830

📥 Installation

helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm repo update
helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \                                                                                                                                                                                       
  --create-namespace \                                                                                                                                                                                               
  --version 2.5.0                                                                                                                                                                                                    
                                                                                                                                                                                                                     
🐳 Images                                                                                                                                                                                                            
 
┌────────────────────────────────────────────┬────────┐                                                                                                                                                              
│                   Image                    │  Tag   │
├────────────────────────────────────────────┼────────┤
│ ghcr.io/kubeflow/spark-operator/controller │ v2.5.0 │
├────────────────────────────────────────────┼────────┤
│ ghcr.io/kubeflow/spark-operator/kubectl    │ v2.5.0 │                                                                                                                                                              
└────────────────────────────────────────────┴────────┘                                                                                                                                                              
                                                                                                                                                                                                                     
---     
Full Changelog: https://github.com/kubeflow/spark-operator/compare/v2.4.0...v2.5.0