github delta-io/delta v4.1.0
Delta Lake 4.1.0
on GitHub

18 hours ago

We are excited to announce the release of Delta Lake 4.1.0! This release includes significant new features, performance improvements, and important platform upgrades.

Highlights

  • [Spark] Apache Spark 4.1.0 Support. The default build of Delta 4.1.0 leverages Apache Spark 4.1.0; however, it retains compatibility with Apache Spark 4.0.1.
  • [Spark] Catalog managed table enhancements (preview): Support UC managed table creation, batch read/write, streaming read/write.
  • [Spark] Spark V2 connector based on Delta Kernel API : A new Spark DataSource V2 connector backed by Delta Kernel, supporting streaming reads for catalog-managed tables.
  • [Spark] Server-Side Planning (preview): Delegate scan planning to catalog servers following the Apache Iceberg REST Catalog API. Supported filter, projection, and limits are pushed down to do the query planning.
  • [Spark] Conflict-free feature enablement: Enable Deletion Vectors and Column Mapping on existing tables without blocking or conflicting with concurrent writes.
  • [Kernel] Full support for catalog-managed tables, enabling Kernel-based connectors to interact with catalog-managed Delta tables (e.g., via Unity Catalog).

Delta Spark

Delta Spark 4.1.0 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

Starting in Delta 4.1.0, Maven artifacts include a Spark version suffix (e.g., delta-spark_4.1_2.13 instead of delta-spark_2.13), with backward compatibility preserved in this release but dependency updates recommended. Separate artifacts are now published for Spark 4.1 and Spark 4.0 so users can choose the version matching their Spark runtime.

The key features of this release are:

  • Catalog-managed table enhancements: Delta Spark has added more support for Unity Catalog managed Delta tables via catalogManaged feature, enabling table creation, batch and streaming reads/writes (including time travel, and DML operations), history inspection, and OAuth-based authentication. This is still in preview and production usage is not recommended.
  • Delta v2 Spark Connector: A new Spark DataSource V2 connector backed by Delta Kernel supporting streaming read for catalog-managed table.
  • Server-Side Planning (preview): Delegate scan planning to an external catalog server, with filter, projection, and limit pushdown and multi-cloud credential support.
  • Atomic CTAS: CREATE TABLE AS SELECT for UC managed delta tables (MANAGED and EXTERNAL) is now fully atomic, working with UC 0.4.0. Other operations (including REPLACE TABLE, REPLACE TABLE AS SELECT, CREATE OR REPLACE TABLE, Dynamic Partition Overwrite) now fail fast instead of running in best-effort mode.
  • Conflict-free Deletion Vector enablement: Enable Deletion Vectors on existing tables without conflicting with concurrent transactions or requiring a maintenance window.
  • Conflict-free Column Mapping enablement: Enable Column Mapping on existing tables without conflicting with concurrent transactions or requiring a maintenance window.
  • Type widening enhancements: New decimal coercion modes during schema evolution (e.g., integer to decimal). The default automatic widening mode is now always.

Other notable changes:

  • Support writing AddFile stats as struct in checkpoints.
  • Fix multiple MERGE and INSERT struct null expansion bugs (#5612, #5658, #5759, #4759).
  • Fix timezone conversion for non-UTC timestamp partition values.
  • Fix metadata cleanup to retain files required for log reconstruction.
  • Fix DomainMetadata handling for RESTORE/CLONE and concurrent REPLACE.

Delta Kernel

The Delta Kernel project is a set of Java and Rust libraries for building Delta connectors that can read and write to Delta tables without the need to understand the Delta protocol details.

The key features of this release are:

  • Full catalog-managed tables support: enabling Kernel-based connectors to read from and write to Delta tables managed by a catalog such as Unity Catalog. This includes snapshot loading, time travel, change data feeds, commit publishing, and a reference Unity Catalog integration. Catalog-managed tables shift commit coordination from the filesystem to a managing catalog, unlocking capabilities like multi-table transactions.
  • Type widening support: Read tables with type widening enabled, including Iceberg compatibility checks for type changes.
  • Variant read/write support including variant stats.
  • Collation support: Writing collated tables, partition pruning, and data skipping with collations.
  • Checkpoint protection support to preserve table history during feature drops.

Delta Universal Format (UniForm)

Delta UniForm is enabled for Spark 4.0 in this release, restoring full Iceberg interoperability that was listed as a limitation in Delta 4.0.0. Both hudi and iceberg are currently not compatible with Spark 4.1, as support depends on upcoming releases providing Spark 4.1 compatible integration bundles.

The key features of this release are:

  • Upgraded Iceberg dependency to 1.10.1.
  • Support converting Delta column default values to Iceberg.

Delta Sharing

The key features of this release are:

Compatibility

  • Java 17 Required: Delta 4.1.0 requires Java 17 or higher.
  • Spark 3.5 Dropped: Support for Spark 3.5 has been removed. Delta 4.1.0 requires Spark 4.0.1 or higher.
  • Check constraints validated earlier: Invalid check constraint and generated column expressions now fail at table creation and on writes.
  • Manual execution of VACUUM is blocked for catalog-managed tables, as data lifecycle must be managed through the catalog.
  • Non-null partition columns: Null values in non-null partition columns are now rejected on commit.

Credits

Abdelrahman Orief, Ada Ma, AlSchlo, Alden Lau, Aleksei Shishkin, Alex Khakhlyuk, Allison Portis, Amanda Liu, Amogh Jahagirdar, Andreas Chatzistergiou, Anoop Johnson, Anudeep Konaboina, AnudeepKonaboina, Artur Owczarek, Bilal Akhtar, Calvin Qin, Carmen Kwan, ChengJi, ChengJi-db, Chirag Singh, Christos Stavrakakis, Cuong Nguyen, David, Dhruv Arya, Drake Lin, Eames Trinh, Felipe Pessoto, Fred Storage Liu, GH-JamesD, Gene Pang, Gengliang Wang, Hao Jiang, Harsh Motwani, Jake Bellacera, Jerry Zheng, Johan Lasperas, Juliusz Sompolski, Kaiqi Jin, Lars Kroll, Lennart Behme, Leon Windheuser, Lukas Rupprecht, Marco Kroll, Marius Grama, Mark Jarvin, Marko Ilić, Ming DAI, Murali Ramanujam, Noritaka Sekiyama, Omar Elhadidy, OussamaSaoudi, Paddy Xu, Philip Zhu, Qianru Lao, Qiyuan Dong, Rajesh Parangi, Rakesh Veeramacheneni, Robert Dillitz, Robert Pack, Sagar Mittal, Scott Sandre, Sebastian Baunsgaard, Shani Solomon, Sumeet Varma, Tathagata Das, Thang Long Vu, Tim Armstrong, Timothy Wang, Tom Zhu, Tom van Bussel, Tomasz Marzec, Venki Korukanti, Vitalii Li, Vrinda Jindal, Wei Luo, Wenchen Fan, Xi Liang, Xin Huang, Yi Li, Yingyi Bu, You Zhou, Yuchuan Huang, Yufa, Yumingxuan Guo, Yuya Ebihara, Zhen Li, Zhipeng Mao, Zihao Xu, Zikang Han, Ziya Mukhtarov, aleksandr-chernousov-db, anniedde, dengsh12, emkornfield, jiahao-db, ju-klein, littlegrasscao, mollyo-openai, openinx, richardc-db, rqureshi-openai, uros7251brick, yaoforx, Yan Yan

Check out latest releases or
releases around delta-io/delta v4.1.0

Don't miss a new delta release

NewReleases is sending notifications on new releases.

Get notifications