Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.4.0 (App Version) with Helm chart 1.2.0 (Chart Version).
Notable Features
- Support K8s v1.31, v1.32, and v1.33
- Support Apache Spark 3.5 and 4.0
- Support
SparkAppandSparkClusterCRDssparkapplications.spark.apache.org(v1beta1)sparkclusters.spark.apache.org(v1beta1)
- Support
HPAforSparkCluster - Support
Artifact Hub - Provide various examples
Docker Image
docker pull apache/spark-kubernetes-operator:0.4.0
Install Helm Chart
$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
spark default 1 2025-06-04 11:06:24.61408 -0700 PDT deployed spark-kubernetes-operator-1.2.0 0.4.0
Full Changelog:
Resolved Issues
[SPARK-52378] - Release Spark Kubernetes Operator 0.4.0
[SPARK-52086] - Upgrade operator-sdk to 5.1.1
[SPARK-52379] - Bump version to 0.4.0-SNAPSHOT in main branch
[SPARK-52416] - Upgrade Gradle to 8.14.2
[SPARK-52435] - Update CIs to use actions/checkout@v4 consistently
[SPARK-52464] - Update .asf.yaml with new README.md link
[SPARK-52466] - Clean up unnecessary comments in e2e tests
[SPARK-52467] - Add dfs-read-write and localstack examples
[SPARK-52471] - Add Swift-based SparkPi K8s CronJob example
[SPARK-52474] - Limit GHA job execution time to up to 20 minutes
[SPARK-52477] - Upgrade JUnit to 5.13.1 and Mockito to 5.18.0
[SPARK-52478] - Use GCS Maven Central mirror as the primary repository
[SPARK-52479] - Upgrade the minimum K8s version to v1.31
[SPARK-52480] - Remove scalaVersion from Spark 4+ examples
[SPARK-52481] - Add Spark History Server example
[SPARK-52512] - Add pi-with-eventlog example
[SPARK-52513] - Add a streaming word count example with rolling event logs
[SPARK-52526] - Add Spark Thrift Server example
[SPARK-52555] - Enforce UnusedLocalVariable rule
[SPARK-52558] - Lower SparkOperatorConfManager log level to WARN for FileNotFoundException
[SPARK-52559] - Synchronize SparkOperatorConfManager.getValue
[SPARK-52581] - Revise AppCleanUpStep to include cleanup logic for all states
[SPARK-52602] - Document all public classes
[SPARK-52603] - Update GitHub Action YAML file in branch-0.4