This is an alpha release with API version v1alpha1 for Spark 2.3.x and up (including the Spark master branch or 2.4.0-SNAPSHOT). It includes the following list of features and changes:
- Support for exposing application-level (exposed and served by the operator) as well as driver and executor metrics to Prometheus.
- Support for automatically uploading local application dependencies to S3 in
sparkctl. - New mutating admission webhook that replaced the old initializer for pod customization.
- Support for using pod affinity/anti-affinity, mounting arbitrary volumes to the driver/executors, using image pull secret (requires a Spark image built from the Spark master).
- PySpark support (requires a Spark image built from the Spark master).
- Documentation enhancements.