apache/spark-kubernetes-operator 0.8.0

on GitHub

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.8.0 (App Version) with Helm chart 1.6.0 (Chart Version).

Notable Features

• Support K8s v1.33, v1.34, and v1.35
• Support Apache Spark 3.5, 4.0, 4.1
• Support SparkApp and SparkCluster CRDs
  • sparkapplications.spark.apache.org (v1)
  • sparkclusters.spark.apache.org (v1)
• Support HPA for SparkCluster
• Support Artifact Hub
  • https://artifacthub.io/packages/helm/spark-kubernetes-operator/spark-kubernetes-operator/
• Provide various examples

Docker Image

docker pull apache/spark-kubernetes-operator:0.8.0

Install Helm Chart

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list

Full Changelog:

• https://github.com/apache/spark-kubernetes-operator/commits/0.8.0

Resolved Issues

• [SPARK-55000] Release Spark Kubernetes Operator 0.8.0
• [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily
• [SPARK-55002] Bump version to 0.8.0-SNAPSHOT in main branch
• [SPARK-55067] Update Spark 3 examples and tests to use 3.5.8
• [SPARK-55069] Upgrade kubernetes-client to 7.5.1 and update CRDs
• [SPARK-55078] Upgrade Gradle to 9.3.0
• [SPARK-55079] Log Java Version
• [SPARK-55080] Improve MANIFEST.MF to include Implementation-(Title|Version)
• [SPARK-55081] Log Version
• [SPARK-55082] Upgrade Dropwizard metrics to 4.2.37
• [SPARK-55083] Upgrade checkstyle to 13.0.0
• [SPARK-55084] Update docs to recommend K8s 1.33+
• [SPARK-55085] Support NetworkPolicy for SparkApplication
• [SPARK-55094] Enable spark.authenticate by default
• [SPARK-55095] Enable spark.io.encryption.enabled by default
• [SPARK-55099] Support NetworkPolicy for SparkCluster
• [SPARK-55100] Enhance Spark Resource Security
• [SPARK-55102] Upgrade com.gradleup.shadow to 8.3.9
• [SPARK-55165] Upgrade kubernetes-client to 7.5.2
• [SPARK-55190] Improve ClusterInitStep to use Server-Side Apply APIs
• [SPARK-55192] Improve HTTP response security
• [SPARK-55268] Fix ConfigOption.getValue not to invoke resolveValue twice
• [SPARK-55269] Fix a wrong comment of testHandleSentinelResourceReconciliation
• [SPARK-55270] Disallow all HTTP methods except GET and HEAD
• [SPARK-55274] Set -XX:+AlwaysPreTouch by default
• [SPARK-55288] Upgrade Netty to 4.2.9.Final
• [SPARK-55292] Add discover latency metric to track operator processing delay
• [SPARK-55298] Fix "Argument list too long" error in assertGeneratedCRDMatchesHelmChart task
• [SPARK-55310] Upgrade Gradle to 9.3.1
• [SPARK-55329] Upgrade Apache DataFusion Comet to 0.13.0
• [SPARK-55343] Simplify HealthProbe class
• [SPARK-55344] Support spark.kubernetes.operator.metrics.path
• [SPARK-55352] Use K8s Garbage Collection to delete executor pods
• [SPARK-55371] Increase Gradle retry setting to stablize CIs
• [SPARK-55374] Remove vendor requirement from Java toolchain
• [SPARK-55380] Upgrade Iceberg example to use Spark 4.0.2
• [SPARK-55417] Add create_spark_jira.py script
• [SPARK-55419] Upgrade Netty to 4.2.10.Final
• [SPARK-55421] Increase livenessProbe.failureThreshold to 3
• [SPARK-55422] Fix the default value of readinessProbe.failureThreshold to 1
• [SPARK-55425] Set strategy.max-parrallel to 20 for all GitHub Action jobs
• [SPARK-55468] Log Built-in Spark Version
• [SPARK-55470] Add a Checkstyle rule to enforce symbolic placeholder for logging
• [SPARK-55486] Fix StatusRecorder.patchAndStatusWithVersionLocked not to log errors
• [SPARK-55499] Update pi-with-eventlog to generate multiple log files
• [SPARK-55512] Update README.md with YuniKorn 1.8.0
• [SPARK-55527] Remove labeler GitHub Actions job
• [SPARK-55537] Check spark.dynamicAllocation.enabled before overriding deleteOnTermination
• [SPARK-55543] Change SentinelManager.getSentinelResources to package private
• [SPARK-55545] Improve create_spark_jira.py to support -p to set the parent JIRA ID
• [SPARK-55546] Enable GitHub Issues feature
• [SPARK-55550] Add Apache Celeborn example
• [SPARK-55563] Add JWSFilter-enabled Spark History Server example
• [SPARK-55566] Add SparkClusterReconcilerTest
• [SPARK-55567] Remove HttpServletRequest usage
• [SPARK-55592] Revise subtask creating message
• [SPARK-55598] Update rsync-deployments tag to 8.0.3
• [SPARK-55602] Update setup-java to v5
• [SPARK-55611] Add RunningWithPartialCapacity state for SparkApplication CRD
• [SPARK-55623] Add granular restart control with consecutive failure tracking
• [SPARK-55633] Improve create_spark_jira.py to support TYPE parameter -t
• [SPARK-55634] Use Mermaid for (Application|Cluster) State Transition
• [SPARK-55641] Add connection timeout to JIRA client to prevent hanging and enable retries
• [SPARK-55644] Add instanceConfig SparkApplication example
• [SPARK-55651] Improve create_spark_jira.py to support the version parameter
• [SPARK-55660] Add On-Demand PVC example
• [SPARK-55676] Upgrade Hadoop to 3.4.3
• [SPARK-55680] Remove spark.kubernetes.driver.pod.excludedFeatureSteps config from examples
• [SPARK-55698] Upgrade operator-sdk to 5.2.3
• [SPARK-55733] Upgrade PMD to 7.21.0 to support Java 26
• [SPARK-55734] Upgrade Dropwizard metrics to 4.2.38
• [SPARK-55735] Upgrade checkstyle to 13.2.0 and spotbugs-plugin to 6.4.8
• [SPARK-55737] Add Java 26-ea to GitHub Action
• [SPARK-55738] Fix ResourceRetainPolicy javadoc location
• [SPARK-55750] Retry creating secondary resources on 408/500/502/503/504 error codes and network timeouts
• [SPARK-55758] Use ubuntu-slim for GitHub Actions license-check job
• [SPARK-55804] Expose informer cache sync timeout as operator config
• [SPARK-55807] Use Java 25 Compact Object Headers
• [SPARK-55808] Use -XX:+UseCompactObjectHeaders for Java 25+ GitHub Actions build-test jobs
• [SPARK-55840] Upgrade Gradle to 9.4.0
• [SPARK-55845] Add .claude to .gitignore
• [SPARK-55847] Upgrade kubernetes-client to 7.6.0 and update CRDs
• [SPARK-55900] Simplify SparkAppStatusUtils by adjusting import statements
• [SPARK-55906] Remove unused SparkAppStatusUtils.hasReachedState
• [SPARK-55911] Improve ModelUtils to support isClientMode method
• [SPARK-55912] Add since Javadoc tag for all public spark-operator-api classes
• [SPARK-55937] Upgrade kubernetes-client to 7.6.1
• [SPARK-55942] Update GitHub Actions YAML file in branch-0.8