delta-io/delta v3.3.3

Delta Lake 3.3.3

on GitHub

We are pleased to announce the release of Delta Lake 3.3.3, a patch release on top of 3.3.2 with targeted fixes for Delta Sharing cache correctness, transaction log retention safety, and a Kernel Parquet-reader configuration bug, plus a Delta Sharing client upgrade.

Note: this patch release does not publish delta-iceberg (UniForm/Iceberg support) — see Delta UniForm below for details and follow-up plan.

Highlights

• [Spark] Fix transaction log retention during metadata cleanup: Log cleanup could previously delete transaction log files still needed to reconstruct table versions within the retention window on tables with infrequent commits/checkpoints, breaking time travel and CDF reads.
• [Delta Sharing] Fix deletion vector cache refresh bug: Cache refresh could drop deletion-vector fileId→URL mappings, causing long-running queries against shared DV tables to fail with "file id not found" errors.
• [Delta Sharing] Upgrade delta-sharing-client 1.2.2 → 1.2.8: Picks up more resilient OAuth token parsing and improved retry/error-reporting behavior.
• [Spark] Add RANDOMIZE_FILE_PREFIXES support: Wires up the previously-declared delta.randomizeFilePrefixes / delta.randomPrefixLength table properties on the write path, to spread S3 object keys for high-throughput workloads. Opt-in; default behavior unchanged.
• [Spark] Fix Spark 3.5.6 compatibility for TRUNCATE TABLE: Spark 3.5.6 tightened V2Writes validation; Delta tables now correctly declare TRUNCATE/OVERWRITE_BY_FILTER capability.
• [Kernel] Fix missing Hadoop configuration in Parquet reader: The Hadoop Configuration was not being passed when Kernel opened Parquet footers, so custom filesystem/credential settings could be silently ignored.

Delta Spark

Delta Spark 3.3.3 is built on Apache Spark 3.5.6. Maven artifacts are published for both Scala 2.12 and 2.13.

• Maven artifacts:
  • delta-spark_2.12 / delta-spark_2.13
  • delta-contribs_2.12 / delta-contribs_2.13
  • delta-storage
  • delta-storage-s3-dynamodb
• Python artifact:
  • delta-spark

Fixes and changes in this release:

• Fix transaction log retention during metadata cleanup: Delete eligible log files only if a newer checkpoint exists before the retention cutoff, so time travel and CDF reads to retained versions keep working. Tables with a low commit/checkpoint rate may retain log files slightly beyond the nominal cutoff as a result.
• Add RANDOMIZE_FILE_PREFIXES support: delta.randomizeFilePrefixes / delta.randomPrefixLength table properties are now honored on writes, placing new data files under a random subdirectory prefix to spread S3 request load. Opt-in; default write path is unchanged.
• Fix Spark 3.5.6 compatibility for TRUNCATE TABLE: StagedDeltaTableV2 now implements SupportsTruncate and declares TRUNCATE/OVERWRITE_BY_FILTER capabilities, fixing TRUNCATE TABLE under Spark 3.5.6's stricter V2Writes validation.

Delta Kernel

The Delta Kernel project is a set of Java libraries for building Delta connectors without needing to implement the Delta protocol directly.

• Maven artifacts:
  • delta-kernel-api
  • delta-kernel-defaults

Fixes in this release:

• Fix missing Hadoop configuration parameter when creating the Parquet reader: A follow-up to an earlier IllegalAccessError fix had dropped the Hadoop Configuration when constructing the ParquetReader used to read footers, so custom Hadoop filesystem settings could be ignored during Kernel reads.

Delta UniForm

Delta UniForm's delta-iceberg and delta-hudi modules keep Apache Iceberg and Apache Hudi metadata in sync with Delta commits.

• delta-iceberg is not published in 3.3.3. There will be no delta-iceberg artifact in this release. There are no changes to delta-iceberg from the previous version. Existing delta-iceberg_2.12/delta-iceberg_2.13 v3.3.2 artifacts remain available and are forward-compatible with delta-spark 3.3.3.
• Maven artifacts:
  • delta-hudi_2.12 / delta-hudi_2.13

No changes to Delta Hudi in this patch release (rebuilt against the updated build toolchain only).

Delta Sharing

Delta Sharing is a Spark DataSource for batch, streaming, CDF, and time-travel reads on tables shared via the Delta Sharing protocol.

• Maven artifacts:
  • delta-sharing-spark_2.12 / delta-sharing-spark_2.13

Fixes and changes in this release:

• Fix deletion vector cache refresh dropping fileId→URL entries: Cache refresh previously dropped deletion-vector file mappings registered during initial cache setup, causing long-running queries against DV-enabled shared tables to fail with "file id not found" errors after a refresh.
• Upgrade delta-sharing-client to 1.2.8 (from 1.2.2): brings more resilient OAuth expires_in parsing, adaptive retry/backoff on retryable server errors, and clearer error reporting from streaming responses.

Delta Standalone

• Maven artifacts:
  • delta-standalone_2.12 / delta-standalone_2.13

No changes to Delta Standalone in this patch release.

Delta Hive Connector

• Maven artifacts:
  • delta-hive_2.12 / delta-hive_2.13

No changes to the Delta Hive connector in this patch release.

Delta Flink

The Kernel-based delta-flink connector (experimental).

• Maven artifact:
  • delta-flink

No changes to Delta Flink in this patch release.

Credits

Daniel Mattos, Felipe Pessoto, Geeta Krishna Panda, littlegrasscao, Sun Cao, Venki Korukanti, Vishnu Chandrashekhar, Yingyi Bu