apache/spark-kubernetes-operator 0.7.0

on GitHub

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.7.0 (App Version) with Helm chart 1.5.0 (Chart Version).

Notable Features

• Support K8s v1.33, v1.34, and v1.35
• Support Apache Spark 3.5, 4.0, 4.1
• Support SparkApp and SparkCluster CRDs
  • sparkapplications.spark.apache.org (v1)
  • sparkclusters.spark.apache.org (v1)
• Support HPA for SparkCluster
• Support Artifact Hub
  • https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
• Provide various examples

Docker Image

docker pull apache/spark-kubernetes-operator:0.7.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list

Full Changelog:

• https://github.com/apache/spark-kubernetes-operator/commits/0.7.0

Resolved Issues

• [SPARK-54168] Bump version to 0.7.0-SNAPSHOT in main branch
• [SPARK-54328] Add configurable startupProbe and enhance liveness/readiness probes in Helm chart
• [SPARK-54428] Upgrade Spark to 4.1.0-preview4
• [SPARK-54432] Use 4.1.0-preview4-java21-scala image for preview examples
• [SPARK-54658] Add Apache DataFusion Comet example
• [SPARK-54659] Enhance gradle task 'generateConfPropsDoc' to account for recent changes
• [SPARK-54667] Upgrade Gradle to 9.2.1
• [SPARK-54671] Upgrade operator-sdk to 5.2.1
• [SPARK-54672] Upgrade mockito to 5.21.0
• [SPARK-54674] Fix AppValidateStep to prevent ClientMode deployment correctly
• [SPARK-54676] Upgrade Spotless to 8.1.0
• [SPARK-54677] Upgrade pmd to 7.19.0 and apply EnumComparison rules
• [SPARK-54678] Upgrade spotbugs-tool to 4.9.8 and spotbugs-plugin to 6.4.7
• [SPARK-54680] Upgrade actions/checkout to v6
• [SPARK-54747] Upgrade Spark to 4.1.0
• [SPARK-54755] Promote all preview examples with 4.1.0
• [SPARK-54757] Update SparkClusterResourceSpec to use apache/spark:4.1.0 as the default value of container image
• [SPARK-54764] Use 4.1.0 instead of 4.0.1 for tests and benchmark
• [SPARK-54765] Make README.md and examples up-to-date with 4.1.0
• [SPARK-54766] Update CI to test K8s 1.35
• [SPARK-54788] Upgrade the minimum K8s version to v1.33
• [SPARK-54789] Support label selector based filter on resources to reconcile
• [SPARK-54791] Increase setup-minikube resources to 3 cores and 10240m
• [SPARK-54792] Update setup-minikube to v0.0.21
• [SPARK-54795] Suppress Hadoop warnings in History Server example
• [SPARK-54813] Remove cores configs from tests/e2e/watched-namespaces/spark-example.yaml
• [SPARK-54814] Set parallel to 1 for K8s integration tests
• [SPARK-54898] Upgrade Iceberg to 1.10.1 in spark-connect-server-iceberg.yaml
• [SPARK-54926] Improve Helm chart for adding labels and annotations for operator config map
• [SPARK-54927] Add Volcano example
• [SPARK-54931] Support Pod Disruption Budget for Spark cluster workers
• [SPARK-54932] Use (pi|pi-with-driver-timeout).yaml to show ttlAfterStopMillis example
• [SPARK-54964] Bump Java Operator SDK to version 5.2.2
• [SPARK-54983] Upgrade Spark to 4.1.1
• [SPARK-54997] Use Spark 4.1.1 in docs, examples, tests, and default values
• [SPARK-55001] Update GitHub Action YAML file in branch-0.7