delta-io/delta v3.1.0

Delta Lake 3.1.0

on GitHub

We are excited to announce the release of Delta Lake 3.1.0. This release includes several exciting new features.

Few Highlights

• Delta-Spark: Support for merge with deletion vectors to reduce the write overhead for merge operations. This feature improves the performance of merge by several folds.
• Delta-Spark: Support for optimizing min/max aggregation queries using the table metadata which improves the performance of simple aggregations queries (e.g SELECT min(x) FROM deltaTable) by up to 100x.
• Delta-Spark: Support for querying tables shared through Delta Sharing protocol.
• Kernel: Support for data skipping for given query predicates to reduce the number of files read during the table scan.
• Uniform: Enhanced Iceberg support for Delta tables that enables MAP and LIST types and ease of use improvements to enable Uniform on a Delta table.
• Delta-Flink: Flink write job startup time latency improvement using Kernel.

Details by each component.

Delta Spark

Delta Spark 3.1.0 is built on Apache Spark™ 3.5. Similar to Apache Spark, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

• Documentation: https://docs.delta.io/3.1.0/index.html
• API documentation: https://docs.delta.io/3.1.0/delta-apidoc.html#delta-spark
• Maven artifacts: delta-spark_2.12, delta-spark_2.13, delta-contribs_2.12, delta_contribs_2.13, delta-storage, delta-storage-s3-dynamodb, delta-iceberg_2.12, delta-iceberg_2.13
• Python artifacts: https://pypi.org/project/delta-spark/3.1.0/

The key features of this release are:

• Support for merge with deletion vectors to reduce the write overhead for merge operations. This feature improves the performance of merge by several folds. Refer to the documentation on deletion vectors for more information.
• Support for optimizing min/max aggregation queries using the table metadata which improves the performance of simple aggregations queries (e.g SELECT min(x) FROM deltaTable) by up to 100x.
• (Preview) Liquid clustering for better table layout Now Delta allows clustering the data in a Delta table for better data skipping. Currently this is an experimental feature. See documentation and example for how to try out this feature.
• Support for DEFAULT value columns. Delta supports defining default expressions for columns on Delta tables. Delta will generate default values for columns when users do not explicitly provide values for them when writing to such tables, or when the user explicitly specifies the DEFAULT SQL keyword for any such column. See documentation on how to enable this feature and try out.
• Support for Hive Metastore schema sync. Adds a mechanism for syncing the table schema to HMS. External tools can now directly consume the schema from HMS instead of accessing it from the Delta table directory. See the documentation on how to enable this feature.
• Auto compaction to address the small files problem during table writes. Auto compaction which runs at the end of the write query combines small files within partitions to large files to reduce the metadata size and improve query performance. See the documentation for details on how to enable this feature.
• Optimized write is an optimization that repartitions and rebalances data before writing them out to a Delta table. Optimized writes improve file size and reduce the small file problem as data is written and benefit subsequent reads on the table. See the documentation for details on how to enable this feature.

Other notable changes include:

• Peformance improvement by removing redundant jobs when performing DML operations with deletion vectors.
• Update command now writes deletions vectors by default when the table has deletion vectors enabled.
• Support for writing partition columns to data files.
• Support for phaseout of v2 checkpoint table feature.
• Fix an issue with case-sensitive column names in Merge.
• Make VACCUM command to be Delta protocol aware so that it can only vacuum tables with protocol that it supports.

Delta Sharing Spark

• Documentation: https://docs.delta.io/3.1.0/delta-sharing.html
• Maven artifacts: delta-sharing-spark_2.12, delta-sharing-spark_2.13

This release of Delta adds a new module called delta-sharing-spark which enables reading Delta tables shared using the Delta Sharing protocol in Apache Spark™. It is migrated from https://github.com/delta-io/delta-sharing/tree/main/spark repository to https://github.com/delta-io/delta/tree/master/sharing repository. Last release version of delta-sharing-spark is 1.0.4 from the previous location. Next release of delta-sharing-spark is with the current release of Delta which is 3.1.0.

Supported read types are: read snapshot of the table, incrementally read the table using streaming or read the changes (Change Data Feed) between two versions of the table.

“Delta Format Sharing” is newly introduced since delta-sharing-spark 3.1, which supports reading shared Delta tables with advanced Delta features such as deletion vectors and column mapping.

Below is an example of reading a Delta table shared using the Delta Sharing protocol in a Spark environment. For more examples refer to the documentation.

import org.apache.spark.sql.SparkSession

val spark = SparkSession
  .builder()
  .appName("...")
  .master("...")
  .config(
     "spark.sql.extensions",
      "io.delta.sql.DeltaSparkSessionExtension"
  ).config(
     "spark.sql.catalog.spark_catalog",
      "org.apache.spark.sql.delta.catalog.DeltaCatalog"
  ).getOrCreate()

val tablePath = "<profile-file-path>#<share-name>.<schema-name>.<table-name>"

// Batch query
spark.read
  .format("deltaSharing")
  .option("responseFormat", "delta")
  .load(tablePath)
  .show(10)

Delta Universal Format (UniForm)

• Documentation: https://docs.delta.io/3.1.0/delta-uniform.html
• Maven artifacts: delta-iceberg_2.12, delta-iceberg_2.13

Delta Universal Format (UniForm) allows you to read Delta tables from Iceberg and Hudi (coming soon) clients. Delta 3.1.0 provided the following improvements:

• Enhanced Iceberg support through IcebergCompatV2. IcebergCompatV2 adds support forLIST and MAP data types and improves compatibility with popular Iceberg reader clients.
• Easier retrieval of the Iceberg metadata file location via familiar SQL syntax DESCRIBE EXTENDED TABLE.
• A new SQL command to enable UniForm REORG TABLE table APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2)) on existing Delta tables. See the documentation for details.
• Delta file statistics conversion to Iceberg including max/min/rowCount/nullCount which enables efficient data skipping when the tables are read as Iceberg in queries containing predicates.

Delta Kernel

• API documentation: https://docs.delta.io/3.1.0/api/java/kernel/index.html
• Maven artifacts: delta-kernel-api, delta-kernel-defaults

The Delta Kernel project is a set of Java libraries (Rust will be coming soon!) for building Delta connectors that can read (and, soon, write to) Delta tables without the need to understand the Delta protocol details).

• Delta 3.0.0 released the first version of Kernel. In this release, read support is further enhanced and APIs are solidified by taking into account the feedback received from connectors trying out the first version of Kernel in Delta 3.0.0.
• Support for data skipping for given query predicates. Now Kernel can prune the list of files to scan for a given query predicate using the file level statistics stored in the Delta metadata. This helps connectors read less data than usual.
• Improved Delta table reconstruction latency. Kernel now can read load the initial protocol and metadata several times faster due to improved table state reconstruction.
• Support for column mapping id mode. Now tables with column mapping id mode can be read by Kernel.
• Support for slf4j logging

For more information, refer to:

• User guide on step by step process of using Kernel in a standalone Java program or in a distributed processing connector.
• Slides explaining the rationale behind Kernel and the API design.
• Example Java programs that illustrate how to read Delta tables using the Kernel APIs.
• Table and default TableClient API Java documentation

Delta Flink

• Delta-Flink 3.1.0 is built on top of Apache Flink™ 1.16.1.
• Documentation: https://github.com/delta-io/delta/tree/branch-3.1/connectors/flink
• API Documentation: https://docs.delta.io/3.1.0/api/java/flink/index.html
• Maven artifact: delta-flink

The key features of this release are

• Flink write job startup time latency improvement using Kernel In this version, Flink has an option to use Kernel to load the Delta table metadata (i.e table schema) which helps the reduce the startup time by up to 45x. To enable this set io.delta.flink.kernel.enabled to true in the Hadoop configuration you pass when creating the Flink Sink.

Delta Standalone

• Documentation: https://docs.delta.io/3.1.0/delta-standalone.html
• API Documentation: https://docs.delta.io/3.1.0/api/java/standalone/index.html
• Maven artifacts: delta-standalone_2.12, delta-standalone_2.13

There are no updates to Standalone in this release.

Credits

Ala Luszczak, Allison Portis, Ami Oka, Amogh Akshintala, Andreas Chatzistergiou, Bart Samwel, BjarkeTornager, Christos Stavrakakis, Costas Zarifis, Daniel Tenedorio, Dhruv Arya, EJ Song, Eric Maynard, Felipe Pessoto, Fred Storage Liu, Fredrik Klauss, Gengliang Wang, Gerhard Brueckl, Haejoon Lee, Hao Jiang, Jared Wang, Jiaheng Tang, Jing Wang, Johan Lasperas, Kaiqi Jin, Kam Cheung Ting, Lars Kroll, Li Haoyi, Lin Zhou, Lukas Rupprecht, Mark Jarvin, Max Gekk, Ming DAI, Nick Lanham, Ole Sasse, Paddy Xu, Patrick Leahey, Peter Toth, Prakhar Jain, Renan Tomazoni Pinzon, Rui Wang, Ryan Johnson, Sabir Akhadov, Scott Sandre, Serge Rielau, Shixiong Zhu, Tathagata Das, Thang Long Vu, Tom van Bussel, Venki Korukanti, Vitalii Li, Wei Luo, Wenchen Fan, Xin Zhao, jintao shen, panbingkun