delta-io/delta v4.0.1

on GitHub

We are excited to announce the release of Delta Lake 4.0.1! This release contains important bug fixes to 4.0.0 and it is recommended that users update to 4.0.1.

Highlights

• [Spark] Breaking change: rename managed table feature from catalogOwned-preview to catalogManaged; legacy ucTableId has also transitioned to the new managed-table io.unitycatalog.tableId
• [Spark] Unity Catalog OAuth: add OAuth authentication via catalog-scoped auth.* configs; tokens are acquired and refreshed automatically; static tokens remain supported.
• [Spark] Unity Catalog managed table table creation: enable creating UC-managed Delta table and always send table properties to the UC server as source of truth.
• [Spark] Spark compatibility fix: addressed NoSuchMethodError in REORG TABLE … APPLY (PURGE) when running with Spark 4.0.1.

Component-specific bug fixes are detailed below.

Delta Spark

Delta Spark 4.0.1 is built on Apache Spark™ 4.0.1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

• Maven artifacts: delta-spark_2.13, delta_contribs_2.13, delta-storage, delta-storage-s3-dynamodb, delta-connect-client_2.13, delta-connect-common_2.13, delta-connect-server_2.13
• Python artifacts: https://pypi.org/project/delta-spark/4.0.1/

The key features of this release are:

• Catalog Managed Tables: managed table feature rename. The feature formerly named catalogOwned‑preview is standardized as catalogManaged. The associated Unity Catalog table ID property is updated accordingly (ucTableId → io.unitycatalog.tableId) ; calls that still send the legacy key are handled for compatibility during creation.
• Fix Spark binary compatibility issue that caused REORG TABLE … APPLY (PURGE) to fail with NoSuchMethodError on Spark 4.0.1 by switching to the stable constructor and retrieving SQL configs via SparkSession.active.sessionState.conf in executors
• Introduced end‑to‑end tests for Unity Catalog DML on the 4.0.x line to validate UC‑backed execution paths; scope is test‑only with no public API changes.

Unity Catalog Support

• Official compatibility with UC 0.3.1. Delta 4.0.1 is officially tested with UC integration tests validating end‑to‑end behavior with UC 0.3.1. See the UC 0.3.1 release for the corresponding connector capabilities and APIs.

• Support Unity Catalog OAuth authentication. Use catalog‑scoped auth.* configuration; tokens are automatically acquired and refreshed, avoiding embedded static tokens. Legacy static‑token configs continue to work for backward compatibility.

  Enable OAuth on a Spark catalog alias that points to UC:

  # Point a Spark catalog alias at Unity Catalog
  spark.sql.catalog.mycatalog = "io.unitycatalog.connectors.spark.UCSingleCatalog"
  spark.sql.catalog.mycatalog.uri = "https://<your-workspace-host>"
  
  # OAuth (dynamic tokens) — supported keys
  spark.sql.catalog.mycatalog.auth.type               = "oauth"
  spark.sql.catalog.mycatalog.auth.oauth.uri          = "https://<auth-server-endpoint>"
  spark.sql.catalog.mycatalog.auth.oauth.clientId     = "<client-id>"
  spark.sql.catalog.mycatalog.auth.oauth.clientSecret = "<client-secret>"
  
  # Static token (legacy-compatible)
  spark.sql.catalog.mycatalog.auth.type  = "static"
  spark.sql.catalog.mycatalog.auth.token = "<personal-access-token>"
  # Legacy key also supported: spark.sql.catalog.mycatalog.token

  And run with Delta’s Spark extensions as usual:

  --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

• Support Unity Catalog Managed Delta table creation. You can now create UC‑managed Delta tables via standard CREATE TABLE on a UC‑backed Spark catalog; at creation time, Delta sends table properties to the UC server so the server is the source of truth.

  Example:

  CREATE TABLE mycatalog.my_schema.events (
    id BIGINT,
    ts TIMESTAMP,
    data STRING
  ) 
  USING delta 
  TBLPROPERTIES (
    'delta.feature.catalogManaged' = 'supported'
  );

Compatibility

• Hadoop version: When using the Delta 4.0.1 release with Spark 4.0.1 please use Hadoop 3.4.x; Using Hadoop 3.3.x may lead to dependency conflicts and unsupported/runtime issues. Please update any 3.3.x references in commands and dependency coordinates to 3.4.x.

Credits

Allison Portis, Anudeep Konaboina, Dhruv Arya, Felipe Pessoto, Lukas Rupprecht, Oussama Saoudi, Tathagata Das, Timothy Wang, Yi Li, Hao Jiang, Zheng Hu