github apache/spark-kubernetes-operator 0.9.0
on GitHub

6 hours ago

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. This release includes Apache Spark K8s Operator 0.9.0 (App Version) with Helm chart 1.7.0 (Chart Version).

Notable Features

  • Support K8s v1.34, v1.35, and v1.36
  • Support Apache Spark 3.5, 4.0, 4.1, and tested with 4.2.0-preview5
  • Upgrade Spark to 4.2.0-preview5 to support heterogeneous K8s executor management
  • Promote native acceleration for Spark 4 on Kubernetes
  • Support Java 26

Docker Image 

docker pull apache/spark-kubernetes-operator:0.9.0

Install Helm Chart 

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator
$ helm list

Full Changelog:

Resolved Issues

  • SPARK-52251: Use crd-generator-cli to generate CRDs
  • SPARK-53640: Operator Helm Chart: support additionalContainers, volumes, volumeMounts as array
  • SPARK-55554: Add Apache Gluten example and integration test
  • SPARK-55905: Exponential backoff for getOrCreateSecondaryResource attempt
  • SPARK-55944: Bump version to 0.9.0-SNAPSHOT in main branch
  • SPARK-56004: Upgrade java-operator-sdk library to 5.3.0
  • SPARK-56005: Propagate MDC context in SparkClusterReconciler
  • SPARK-56037: Use Java 26 instead of 26-ea
  • SPARK-56078: Expose failureThreshold and timeoutSeconds in operator livenessProbe schema
  • SPARK-56087: Use Java 26 toolchain while supporting Java 17+
  • SPARK-56088: Support Java 26
  • SPARK-56091: Upgrade Apache DataFusion Comet to 0.14.0
  • SPARK-56096: Upgrade mockito to 5.23.0
  • SPARK-56099: Upgrade spotless-plugin to 8.4.0
  • SPARK-56103: Upgrade junit to 6.0.3
  • SPARK-56105: Upgrade shadow-jar-plugin to 8.3.10
  • SPARK-56107: Remove docker-entrypoint.sh
  • SPARK-56108: Remove Super-Linter from GitHub Actions jobs
  • SPARK-56114: Use Java 26 always except build-test matrix job
  • SPARK-56116: Upgrade Gradle to 9.4.1
  • SPARK-56117: Make spotless ignore .claude directory
  • SPARK-56119: Sync docker-related GitHub Actions versions to the ASF approved patterns
  • SPARK-56127: Use zulu-openjdk-alpine to minimize the Docker image
  • SPARK-56129: Exclude Netty QUIC native libraries to minimize Docker image
  • SPARK-56130: Make values.schema.json up-to-date
  • SPARK-56131: Use jlink custom JRE runtime to minimize Docker image size
  • SPARK-56132: Increase emptyDir sizeLimit to 200Mi for pi-with-comet
  • SPARK-56136: Add PySpark 4.1.1 integration test coverage
  • SPARK-56138: Replace ubuntu-latest with ubuntu-slim in build-test GitHub Actions job
  • SPARK-56139: Add E2E test for ApplicationTimeoutConfig with driverStartTimeoutMillis
  • SPARK-56141: Cache Gradle Java Toolchain in CI
  • SPARK-56142: Add worker StatefulSet replica validation to SparkCluster e2e test
  • SPARK-56144: Remove no-op spark.master.rest.enabled because it is enabled by default since Apache Spark 4.1
  • SPARK-56146: Add e2e test for resourceRetainPolicy: OnFailure with a failed SparkApp
  • SPARK-56162: Prevent Jackson from using reflection on final fields during deserialization for JEP-500
  • SPARK-56192: Exclude spotbugs-annotations from the uber jar
  • SPARK-56193: Fix spark-operator-api to use compileOnly for Lombok
  • SPARK-56194: Upgrade Lombok to 1.18.44
  • SPARK-56195: Fix BaseStatus to handle when K8s API returns null for stateTransitionHistory
  • SPARK-56198: Update Operator to Java Operator SDK 5.3.2
  • SPARK-56210: Remove lombok.SneakyThrows from StatusRecorder
  • SPARK-56211: Ban lombok.(SneakyThrows|val|var)
  • SPARK-56212: Fix spark-versions and python e2e test flakiness by explicit clean-up
  • SPARK-56213: Upgrade Netty to 4.2.12.Final
  • SPARK-56215: Exclude netty-codec-*-quic dependencies
  • SPARK-56218: Exclude Netty io_uring-related dependencies
  • SPARK-56228: Fix StatusRecorder to refresh resourceVersion on 409 Error before retrying
  • SPARK-56229: Remove io.netty.noUnsafe according to Netty changes
  • SPARK-56267: Use testCompileOnly for spotbugs-annotations in spark-operator
  • SPARK-56268: Remove unnecessary log4j dependencies from spark-operator-api
  • SPARK-56269: Fix Gradle 10 deprecation warnings
  • SPARK-56274: Simplify SparkClusterSubmissionWorker.getResourceSpec
  • SPARK-56276: Fix file descriptor leak in SparkAppResourceSpecFactory
  • SPARK-56281: Upgrade PMD to 7.23.0
  • SPARK-56282: Exclude unnecessary avro dependency
  • SPARK-56283: Exclude HDFS/YARN UI resources
  • SPARK-56285: Add minimize to shadowJar task
  • SPARK-56289: Upgrade minimum Java runtime version to 21
  • SPARK-56291: Make BaseStateSummary a sealed interface
  • SPARK-56292: Make (App|Cluster)ReconcileStep abstract classes sealed
  • SPARK-56293: Make BaseAppDriverObserver abstract class sealed
  • SPARK-56294: Apply exhaustive switch for ResourceRetainPolicy and ClusterStateSummary
  • SPARK-56297: Use Virtual Threads (JEP 444) for unbounded reconciliation thread pool
  • SPARK-56299: Fix Dockerfile for jdeps to use 21
  • SPARK-56308: Remove invalid log4j2.contextSelector from test log4j2.properties
  • SPARK-56309: Upgrade log4j to 2.25.4
  • SPARK-56365: Remove hard-coded APP_VERSION from Dockerfile
  • SPARK-56366: Update README.md by removing 17
  • SPARK-56379: Upgrade Hadoop to 3.5.0
  • SPARK-56394: Upgrade the minimum K8s version to v1.34
  • SPARK-56396: Support dependencyUpdates Gradle task
  • SPARK-56401: Use setup-gradle GitHub Actions in CIs
  • SPARK-56405: Enable Gradle configuration cache
  • SPARK-56408: Introduce OwnerResourceDecorator and use it instead (Driver|Cluster)Decorator
  • SPARK-56409: Consolidate Netty exclusions into root build.gradle
  • SPARK-56417: Consolidate Lombok and JUnitPlatform configurations into root build.gradle
  • SPARK-56418: Unify getOrCreateLocalFileFor(Driver|Executor)Spec to getOrCreateLocalFileForSpec
  • SPARK-56433: Upgrade operator-sdk to 5.3.3
  • SPARK-56434: Add Deployment Topology diagram
  • SPARK-56435: Upgrade Spark to 4.2.0-preview4
  • SPARK-56442: Use Java-friendly APIs of JavaMainAppResource
  • SPARK-56443: Use Java-friendly API of KubernetesDriverSpec
  • SPARK-56445: Exclude unused LoosePackageCoupling PMD rule
  • SPARK-56446: Use Java Optional in SparkAppDriverConf.create method signature
  • SPARK-56473: Rename create_spark_jira.py to create_jira_and_branch.py
  • SPARK-56487: Update spotbugs-gradle-plugin to 6.5.0
  • SPARK-56544: Update gradle-versions-plugin to 0.54.0
  • SPARK-56545: Update spotbugs-gradle-plugin to 6.5.1
  • SPARK-56552: Update cyclonedx to 3.2.4
  • SPARK-56571: Upgrade Apache DataFusion Comet to 0.15.0
  • SPARK-56621: Update CI to test K8s 1.36
  • SPARK-56628: Update lombok to 1.18.46
  • SPARK-56644: Support periodic GC
  • SPARK-56665: Use gradle:9.4.1-jdk26-noble Docker image for builder stage
  • SPARK-56667: Upgrade Gradle to 9.5.0
  • SPARK-56679: Fix Helm chart to render valid YAML for operator ConfigMap
  • SPARK-56701: Upgrade Apache DataFusion Comet example to use Spark 4.0.2
  • SPARK-56715: Use CrashOnOutOfMemoryError with stderr redirection
  • SPARK-56723: Use the latest docker/* GitHub Actions
  • SPARK-56727: Add Apache DataFusion Comet K8s integration test
  • SPARK-56728: Simplify chainsaw test manifests via apply syntax
  • SPARK-56732: Upgrade Spark to 4.2.0-preview5
  • SPARK-56733: Improve SparkAppResourceSpec to use Java-friendly KubernetesDriverSpec APIs
  • SPARK-56735: Improve SparkClusterResourceSpec to use Java-friendly SparkConf API
  • SPARK-56747: Documentation fixes
  • SPARK-56751: Promote native acceleration for Spark 4 on Kubernetes (Comet, Gluten)
  • SPARK-56752: Enable spark.kubernetes.driver.annotateExitException by default
  • SPARK-56776: Upgrade Netty to 4.2.13.Final
  • SPARK-56777: Upgrade log4j to 2.26.0
  • SPARK-56778: Upgrade checkstyle to 13.4.2
  • SPARK-56779: Upgrade spotbugs-gradle-plugin to 6.5.4
  • SPARK-56780: Upgrade PMD to 7.24.0
  • SPARK-56785: Use the latest actions/* GitHub Actions
  • SPARK-56786: Add pi-preview.yaml with Apache Spark 4.2.0-preview5
  • SPARK-56787: Add Deployment-based SparkApp example
  • SPARK-56789: Add ExecutorResizePlugin example
  • SPARK-56823: Update GitHub Actions YAML file in branch-0.9
Check out latest releases or
releases around apache/spark-kubernetes-operator 0.9.0

Don't miss a new spark-kubernetes-operator release

NewReleases is sending notifications on new releases.

Get notifications