delta-io/delta v4.4.0

Delta Lake 4.4.0

on GitHub

We are excited to announce the release of Delta Lake 4.4.0. This release adds Apache Spark 4.2 support, expands integration with the UC Delta Table API, and adds identity-column and generated-column support to Delta Spark.

Highlights

• [Spark] Apache Spark 4.2 support: Delta Spark, Delta Connect, and Delta Sharing now support Apache Spark 4.2.0.
• [Kernel] [Flink] UC Delta API integration: Delta Kernel and the experimental Flink connector can use the UC Delta API for catalog-managed tables.
• [Spark] Identity columns in SQL DDL: CREATE TABLE now supports GENERATED ALWAYS AS IDENTITY and GENERATED BY DEFAULT AS IDENTITY.
• [Spark] Generated columns with Unity Catalog: Delta tables created through Unity Catalog can now define generated columns.

Delta Spark

Delta Spark 4.4.0 is built for Apache Spark 4.2.0, Apache Spark 4.1.0, and Apache Spark 4.0.1. As with Apache Spark, Maven artifacts are published for Scala 2.13.

• Maven artifacts for Spark 4.2.0:
  • delta-spark_4.2_2.13
  • delta-connect-client_4.2_2.13
  • delta-connect-common_4.2_2.13
  • delta-connect-server_4.2_2.13
• Maven artifacts for Spark 4.1.0:
  • delta-spark_4.1_2.13
  • delta-connect-client_4.1_2.13
  • delta-connect-common_4.1_2.13
  • delta-connect-server_4.1_2.13
• Maven artifacts for Spark 4.0.1:
  • delta-spark_4.0_2.13
  • delta-connect-client_4.0_2.13
  • delta-connect-common_4.0_2.13
  • delta-connect-server_4.0_2.13
• Backward-compatibility artifacts without a Spark version in the name default to Spark 4.2.0:
  • delta-spark_2.13
  • delta-connect-client_2.13
  • delta-connect-common_2.13
  • delta-connect-server_2.13
• Other Maven artifacts:
  • delta-contribs_2.13
  • delta-storage
  • delta-storage-s3-dynamodb
• Python artifact:
  • delta-spark

The key features of this release are:

• Apache Spark 4.2 support (#7223): Delta Spark, Delta Connect, and Delta Sharing are now built and tested with Spark 4.2.0 in addition to Spark 4.0.1 and 4.1.0.
• Identity columns in Spark SQL DDL (#7062): CREATE TABLE now accepts Spark's GENERATED ALWAYS AS IDENTITY and GENERATED BY DEFAULT AS IDENTITY syntax, including custom start and increment values.
• Generated columns with Unity Catalog (#7290): Generated-column expressions are now preserved during table creation. Delta computes or validates generated values during writes using the declared expressions.
• SHOW PARTITIONS support (#6102): Users can inspect the partitions of a partitioned Delta table with the standard Spark SQL command.
• VOID column support (#6965, #6966): On Spark 4.1 and later, Delta reads now preserve VOID (NullType) columns instead of failing or silently dropping them. This includes time-travel and path-based reads.

Delta Kernel

Delta Kernel is a set of Java libraries for building Delta connectors that read and write Delta tables without requiring each connector to implement the Delta protocol directly.

• Maven artifacts:
  • delta-kernel-api
  • delta-kernel-defaults
  • delta-kernel-unitycatalog

The key changes in this release are:

• UC Delta API integration (#7159): The Unity Catalog committer can use the UC Delta API for catalog-managed tables, including structured protocol, metadata, and domain-metadata updates.
• Safer concurrent writes (#7163): Concurrent transactions that remove or update the same data file now conflict instead of allowing duplicate active files or lost deletion-vector updates.
• Correct partition output for column-mapped tables (#7212): Kernel write contexts now translate logical partition names to physical in target paths and AddFile.partitionValues, allowing partitioned writes to name- and id-mapped tables to round-trip correctly.
• Protocol-recommended timestamp partitions (#7287): Newly written TIMESTAMP partition values use UTC ISO-8601 strings. Pre-epoch TIMESTAMP_NTZ values are also serialized correctly.
• Fewer object-store metadata requests (#6856, #7099): Known file lengths are passed when opening files, and redundant existence probes are avoided during log listing. This reduces HEAD requests and prevents failures with scoped credentials that permit listing but not directory-level HEAD operations.

Delta UniForm

Delta UniForm's delta-iceberg module keeps Apache Iceberg metadata synchronized with Delta commits, allowing Iceberg readers to query Delta tables without duplicating data.

• Maven artifacts:
  • delta-iceberg_2.13

Note: In Delta 4.4, delta-iceberg_2.13 supports Spark 4.1 and is not compatible with Spark 4.2.

The key changes in this release are:

• Atomic table initialization (#6880): UniForm metadata can be initialized atomically with Delta table creation.
• Improved Iceberg conversion correctness (#7047, #7187): Metadata-only statistics commits are handled correctly, and UniForm table properties are preserved during saveAsTable overwrite.

Delta Sharing

Delta Sharing is a Spark DataSource that supports batch, streaming, CDF, and time-travel reads on tables shared through the Delta Sharing protocol.

• Maven artifacts for Spark 4.2.0:
  • delta-sharing-spark_4.2_2.13
• Maven artifacts for Spark 4.1.0:
  • delta-sharing-spark_4.1_2.13
• Maven artifacts for Spark 4.0.1:
  • delta-sharing-spark_4.0_2.13
• Backward-compatibility artifact without a Spark version in the name defaults to Spark 4.2.0:
  • delta-sharing-spark_2.13

The key changes in this release are:

• Delta Sharing client 1.4.1 (#7281): The Spark connector now uses Delta Sharing client 1.4.1.
• Protocol changes within streaming and CDF ranges (#7305, #7319): Opt-in historical-protocol support allows a shared streaming or CDF query to cross a table protocol upgrade without reconstructing an inconsistent local log. The switches are disabled by default and require corresponding server support.
• Independent CDF streaming format resolution (#7003): CDF streaming can control automatic Parquet-versus-Delta response-format resolution independently from ordinary streaming.

Delta Flink

The Kernel-based delta-flink connector remains experimental. Delta 4.4.0 supports Apache Flink 2.0.2, 2.1.3, 2.2.1, and 2.3.0.

• Maven artifacts:
  • delta-flink_2.0
  • delta-flink_2.1
  • delta-flink_2.2
  • delta-flink_2.3

The key changes in this release are:

• Primary-key upserts from Flink SQL (#6933, #6935, #6977, #6997): CREATE TABLE ... WITH ('write.mode' = 'upsert') uses the declared primary key to process Flink changelog rows. INSERT is treated as a new key, UPDATE_AFTER replaces an existing key, DELETE removes it, and UPDATE_BEFORE is ignored. A primary key is required in upsert mode.
• Merge-on-read updates and deletes (#6964, #6977): The default upsert strategy uses deletion vectors to remove old row versions while appending new rows, avoiding full data-file rewrites.
• UC Delta API integration (#7229, #7244): Catalog-managed table loads, existence checks, commits, and storage-credential vending now use the UC Delta API. Catalog browsing continues to use the existing catalog API.
• Ambient and customer-provided storage credentials (#7045, #7048): Set credentials.source=ambient to use workload identity, instance profiles, application-default credentials, or filesystem configuration instead of Unity Catalog credential vending. Table options beginning with fs. are now passed to the Kernel engine.
• Path handling fix (#7027): Storage authorities containing underscores or user-info components are preserved instead of being dropped during URI normalization.
• Correct handling of reused Flink rows (#7475): Delta-Flink now copies rows before buffering them, preventing later row reuse from changing previously accepted values.
• Version-specific Flink artifacts (#7484): Delta now publishes separate artifacts for Flink 2.0, 2.1, 2.2, and 2.3.

Protocol

• VOID behavior (#6966): The protocol now defines how readers and writers handle VOID columns, complementing Spark 4.1+'s ability to preserve missing columns in query output.
• Interval-type and column-mapping clarifications (#7078, #6939): The protocol clarifies interval-type behavior and the uniqueness requirement for column-mapping identifiers.

For the complete commit history, see the comparison between v4.3.1 and v4.4.0.

Credits

Adam Reeve, Ala Luszczak, Aleksei Shishkin, Alex Moschos, Alexandru Mihai, Amogh Jahagirdar, Annie Wang, Anoop Johnson, Ayush Raj, Bilal Akhtar, Brooks Walls, Charlene Lyu, Chen Wang, ChengJi, Chiin Luen Quah, Cuong Nguyen, Dhruv Arya, Eames Trinh, Eduard Tudenhoefner, Felipe Pessoto, foss-contributor, Gengliang Wang, Hao Jiang, Hao Sun, Hari Lamichhane, Itamar Turner-Trauring, Ivan Sadikov, Jason Chen, Jiayuan Chen, Jinhua Song, Johan Lasperas, Juliusz Sompolski, Kaiqi Jin, Lars Kroll, Leonid Lygin, littlegrasscao, Lukas Rupprecht, Miles Cole, Murali Ramanujam, ni-mi, Nimrod Ofek, Omar Elhadidy, openinx, OussamaSaoudi, Paddy Xu, PhilPlato, Prakhar Jain, Pratham Manja, Rajesh Parangi, Rakesh Veeramacheneni, Ryan Johnson, Ryan Liao, Sandro Sp, seewishnew, songhang, sotikoug83, Steven Yu, Sun Cao, Sushanth Sathish Kumar, Thang Long Vu, Thinh Bui, Timothy Wang, Uros Bojanic, Utkarsh, Xin Huang, Xintong (Oscar) Zhou, Yi Li, You Zhou, Yumingxuan Guo, yyanyy, Zhen Li, Zihao Xu, Ziya Mukhtarov