github apache/spark-kubernetes-operator v0.1.0
on GitHub

latest releases: 0.4.0, 0.4.0-rc.1, 0.3.0...
4 months ago

Apache Spark™ K8s Operator is a subproject of Apache Spark and aims to extend K8s resource manager to manage Apache Spark applications via Operator Pattern. v0.1.0 is the initial release of Apache Spark K8s Operator.

Notable Features

Docker Image 

docker pull apache/spark-kubernetes-operator:0.1.0

Install Helm Chart 

$ helm repo add spark https://apache.github.io/spark-kubernetes-operator
$ helm repo update
$ helm install spark spark-kubernetes-operator/spark-kubernetes-operator

Full Changelog:

Resolved Issues

  • [SPARK-47745] Add License to Spark Operator repository
  • [SPARK-47889] Setup gradle as build tool for operator repository
  • [SPARK-47929] Setup Static Analysis for Operator
  • [SPARK-47943] Add GitHub Action CI for Java Build and Test
  • [SPARK-47950] Add Java API Module for Spark Operator
  • [SPARK-48015] Update build.gradle to fix deprecation warnings
  • [SPARK-48017] Add Spark application submission worker for operator
  • [SPARK-48120] Enable autolink to SPARK jira issue
  • [SPARK-48326] Use the official Apache Spark 4.0.0-preview1
  • [SPARK-48382] Add reconciler to spark-operator module
  • [SPARK-48398] Add Helm Chart
  • [SPARK-48528] Refine K8s Operator merge_spark_pr.py to use kubernetes-operator-x.y.z version
  • [SPARK-48606] Upgrade google-java-format to 1.22.0
  • [SPARK-48679] Upgrade checkstyle and spotbugs version
  • [SPARK-48786] Define Config Loading Framework for Spark Operator Controller
  • [SPARK-48952] Use BasePluginExtension in spark-operator/build.gradle
  • [SPARK-48953] Upgrade Gradle to 8.9
  • [SPARK-48968] Avoid unnecessary task configuration in spark-operator-api
  • [SPARK-48969] Fix spark-operator module to define test framework explicitly
  • [SPARK-48971] Ban org.apache.commons.collections in favor of collections4
  • [SPARK-48984] Add Controller Metrics System and Utils
  • [SPARK-49019] Use try-with-resources to test KubernetesClientFactory.buildKubernetesClient
  • [SPARK-49020] Avoid raw type usage
  • [SPARK-49036] Exclude JUnitAssertionsShouldIncludeMessage/JUnitTestContainsTooManyAsserts PMD rules and simplify test code
  • [SPARK-49045] Add docker image build for operator
  • [SPARK-49052] Add SparkOperator class and tests
  • [SPARK-49129] Fix ENTRYPOINT to point /opt/spark-operator/operator/docker-entrypoint.sh
  • [SPARK-49130] Verify built images in build-image CI job via docker run test
  • [SPARK-49132] Minimize docker image by removing redundant chown commands
  • [SPARK-49144] Use the latest setup-java v4 with cache feature
  • [SPARK-49148] Use the latest PMD 6.x rules instead of the deprecated ones
  • [SPARK-49153] Increase Gradle JVM memory to 4g like Spark repo
  • [SPARK-49156] Enforce ImmutableField and UselessParentheses rules
  • [SPARK-49157] Enforce SignatureDeclareThrowsException and AvoidThrowingRawExceptionTypes rules
  • [SPARK-49158] Enforce ConfusingTernary and PrematureDeclaration rules
  • [SPARK-49159] Enforce FieldDeclarationsShouldBeAtStartOfClass, LinguisticNaming and ClassWithOnlyPrivateConstructorsShouldBeFinal rules
  • [SPARK-49165] Fix RestartPolicyTest to cover SchedulingFailure
  • [SPARK-49167] Enforce UseUtilityClass rule
  • [SPARK-49168] Add OpenContainers Annotations to docker image
  • [SPARK-49214] Add e2e test in operator workflow
  • [SPARK-49236] Update README.md with build/test/CI info
  • [SPARK-49237] Speed up docker image building by excluding check instead of test
  • [SPARK-49238] Upgrade Gradle to 8.10
  • [SPARK-49239] Add k8s-integration-tests GitHub Action CI job
  • [SPARK-49253] Install and test Helm chart in K8s integration test CI
  • [SPARK-49256] Upgrade kubernetes-client to 6.13.3 and commons-lang3 to 3.16.0
  • [SPARK-49258] Add pi.yaml example and update README.md
  • [SPARK-49270] Add sql.yaml example
  • [SPARK-49271] Improve gradlew to support both curl and wget
  • [SPARK-49276] Use API Group spark.apache.org
  • [SPARK-49277] Refactor RestartPolicyTest to test per case
  • [SPARK-49278] Revise reconcilesteps package and SparkAppReconciler
  • [SPARK-49279] Revise observers package
  • [SPARK-49280] Add pi-with-one-pod.yaml example
  • [SPARK-49289] Fix docker-entrypoint.sh to quote the environment variables
  • [SPARK-49290] Remove commons-lang3 dependency from spark-operator-api
  • [SPARK-49291] Fix javadoc generation and add lint test pipeline to prevent
  • [SPARK-49293] Add _MESSAGE postfix to DRIVER_(READY|RUNNING)
  • [SPARK-49295] Add NOTICE, NOTICE-binary, LICENSE-binary files and update *.gradle files
  • [SPARK-49296] Add deploy.gradle to support publish-related tasks
  • [SPARK-49298] Reduce spark-operator fat jar size by excluding dependencies
  • [SPARK-49299] Add buildDockerImage Gradle Task
  • [SPARK-49304] Add -SNAPSHOT postfix to Spark Operator version
  • [SPARK-49305] Revise Spark Operator docker image
  • [SPARK-49309] Use HTTP_* constant variables instead of magic numbers
  • [SPARK-49315] Generalize relocateGeneratedCRD Gradle Task to handle *.spark.apache.org-v1.yml
  • [SPARK-49316] Generalize printer-columns.sh to handle *.spark.apache.org-v1.yml files
  • [SPARK-49317] Add pi-on-yunikorn.yaml example and update README.md
  • [SPARK-49319] Add SparkCluster to spark-operator-api module and examples
  • [SPARK-49322] Add SparkCluster to spark-submission-worker module
  • [SPARK-49324] Add SparkApplication state transition e2e test
  • [SPARK-49325] Add SparkCluster to spark-operator module
  • [SPARK-49328] Propagate Spark configurations to SparkCluster
  • [SPARK-49329] Support user provided spec for SparkCluster
  • [SPARK-49330] Revise InstanceConfig to ExecutorInstanceConfig class
  • [SPARK-49331] Use setup-minikube GitHub Action
  • [SPARK-49332] Add K8s service for Workers to SparkClusterResourceSpec
  • [SPARK-49340] Document SparkCluster and add submit-pi-to-prod.sh example
  • [SPARK-49343] Document nightly versions of operator image and Helm Chart
  • [SPARK-49346] Add publish_snapshot_dockerhub.yml Daily GitHub Action job
  • [SPARK-49348] Support schedulerName for SparkCluster
  • [SPARK-49351] Add publish_snapshot_chart.yml GitHub Action job
  • [SPARK-49360] Use rsync to upload to nightlies
  • [SPARK-49361] Fix Dockerfile by removing unused ARG from builder and moving default value
  • [SPARK-49362] Simplify snapshot HelmChart to use apache/spark-kubernetes-operator:main-snapshot by default
  • [SPARK-49376] Introduce ClusterToleration and WorkerInstanceConfig
  • [SPARK-49377] Fix e2e catch step application not found
  • [SPARK-49379] Enable Pull Request Labeler
  • [SPARK-49380] Add cluster-with-template.yaml example
  • [SPARK-49388] Remove SPARK_NO_DAEMONIZE in favor of live log UIs
  • [SPARK-49389] Support master|worker container templates
  • [SPARK-49395] Add 8081 port to Worker resource spec
  • [SPARK-49397] Add Clean Up section to README.md
  • [SPARK-49399] Add pi-scala.yaml and pyspark-pi.yaml
  • [SPARK-49400] Use Gradle Version Catalog
  • [SPARK-49403] Add log4j2 default setting to values.yaml
  • [SPARK-49404] Adjust ERROR-level log messages
  • [SPARK-49464] Add documentations
  • [SPARK-49468] Add Hot Properties Reload Test
  • [SPARK-49514] Fix K8s version in GitHub Action CI
  • [SPARK-49516] Upgrade the minimum K8s version to v1.28
  • [SPARK-49527] Add ConfOptionDocGenerator to generate Spark Operator Config Property Doc
  • [SPARK-49590] Fix javaVersion to jdkVersion in E2E spark-versions test template
  • [SPARK-49619] Upgrade Gradle to 8.10.1
  • [SPARK-49623] Refactor prefix appResources to workloadResources
  • [SPARK-49625] Add SparkCluster state transition test
  • [SPARK-49645] Update e2e/python/chainsaw-test.yaml to use non-R image
  • [SPARK-49657] Add multi instances e2e
  • [SPARK-49658] Refactor e2e tests pipelines
  • [SPARK-49705] Use spark-examples.jar instead of spark-examples_2.13-4.0.0-preview1.jar
  • [SPARK-49706] Use apache/spark images instead of spark
  • [SPARK-49714] Add Java 21-based SparkPi example
  • [SPARK-49715] Add Java 21-based SparkCluster example
  • [SPARK-49724] Add java21 to the e2e tests
  • [SPARK-49735] Upgrade Spark to 4.0.0-preview2
  • [SPARK-49742] Upgrade README, examples, tests to use preview2
  • [SPARK-49754] Support HPA for SparkCluster
  • [SPARK-49778] Remove (master|worker) prefix from field names of (Master|Worker)Spec
  • [SPARK-49790] Support HPA template for SparkCluster
  • [SPARK-49818] Update E2E tests to use Spark 3.5.3
  • [SPARK-49830] Add required fields for sparkApplicationSentinel
  • [SPARK-49831] Provide empty RuntimeVersions object to ClusterSpec.runtimeVersions by default
  • [SPARK-49832] Make o.a.s.k8s.operator.utils.Utils argument naming consistent
  • [SPARK-49838] Add spark-version label to Spark Cluster resources
  • [SPARK-49887] Update cluster-with-template.yaml example with pod annotation
  • [SPARK-49888] Use registry.k8s.io/pause:3.9 to avoid failure deterministically
  • [SPARK-50305] Upgrade Gradle to 8.11
  • [SPARK-50384] Update docs with YuniKorn 1.6.0 and first-time installation guide
  • [SPARK-50401] Upgrade kubernetes-client to 6.13.4 and log4j to 2.24.2
  • [SPARK-50670] Use Spark 3.5.4 in E2E tests
  • [SPARK-50723] Upgrade kubernetes-client to 7.0.1
  • [SPARK-50724] Update gradle to 8.12
  • [SPARK-50725] Update CI to test K8s 1.32
  • [SPARK-50999] Update gradle to 8.12.1
  • [SPARK-51002] Upgrade the minimum K8s version to v1.30
  • [SPARK-51075] Use container resource for HPA
  • [SPARK-51220] Upgrade kubernetes-client to 7.1.0
  • [SPARK-51347] Enable Ingress and Service Support for Spark Driver
  • [SPARK-51352] Use Spark 3.5.5 in E2E tests
  • [SPARK-51382] Add submit-spark-connect-server.sh example
  • [SPARK-51405] Upgrade build-tools to use Ubuntu 24.04 LTS instead of 22.04 LTS docker image
  • [SPARK-51406] Remove no-op spark.log.structuredLogging.enabled=false
  • [SPARK-51410] Add Spark Connect Plugin example
  • [SPARK-51412] Upgrade Gradle to 8.13
  • [SPARK-51507] Support creating config map that can be mounted by Spark pods for apps
  • [SPARK-51520] Use JRE to reduce size of Docker image
  • [SPARK-51526] Upgrade Gradle shadow plugin to 8.3.6
  • [SPARK-51538] Add spark-connect-server-with-spark-cluster.yaml example
  • [SPARK-51770] Set Content-Type headers for Prometheus v3
  • [SPARK-51783] Support ubuntu-24.04-arm in build-test test pipeline
  • [SPARK-51828] Update README.md with YuniKorn 1.6.2
  • [SPARK-51893] Upgrade CI to test K8s 1.33
  • [SPARK-51894] Update setup-minikube to v0.0.19
  • [SPARK-51951] Upgrade Gradle to 8.14
  • [SPARK-51952] Disable fail-fast in Build Test CI matrix
  • [SPARK-51965] Update Compatibility section of docs/operations.md
  • [SPARK-51978] Upgrade kubernetes-client to 7.2.0 for K8s 1.33
  • [SPARK-52002] Update GitHub Action YAML file in branch-0.1
Check out latest releases or
releases around apache/spark-kubernetes-operator v0.1.0

Don't miss a new spark-kubernetes-operator release

NewReleases is sending notifications on new releases.

Get notifications