github delta-io/delta v2.0.2
Delta Lake 2.0.2

latest releases: v3.2.1, v3.2.1rc3, v3.2.1rc2...
22 months ago

We are excited to announce the release of Delta Lake 2.0.2 on Apache Spark 3.2. This release contains important bug fixes and a few high-demand usability improvements over 2.0.1 and it is recommended that users update to 2.0.2. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

This release includes the following bug fixes and improvements:

  • Record VACUUM operation in the transaction log. With this feature, VACUUM operations and their associated metrics (e.g. numDeletedFiles) will now show up in table history.
  • Support idempotent writes for DML operations. This feature adds idempotency to INSERTS/DELETE/UPDATE/MERGE etc. operations using SQL configurations spark.databricks.delta.write.txnAppId and spark.databricks.delta.write.txnVersion.
    Support passing Hadoop configurations via DeltaTable API
    from delta.tables import DeltaTable
    hadoop_config = {
      "fs.azure.account.auth.type": "OAuth",
      "fs.azure.account.oauth.provider.type": "...",
      "fs.azure.account.oauth2.client.id": "...",
      "fs.azure.account.oauth2.client.secret": "...",
      "fs.azure.account.oauth2.client.endpoint": "..."
    }
    delta_table = DeltaTable.forPath(spark, <table-path>, hadoop_config)
  • Minor convenience improvement to the DeltaTableBuilder:executeZOrderBy Java API which allows users to pass in varargs instead of a List.
  • Fail fast on malformed delta log JSON entries. Previously, Delta queries could return inaccurate results whenever JSON commits in the _delta_log were malformed. For example, an add action with a missing } would be skipped. Now, queries will fail fast, preventing inaccurate results.
  • Fix “Could not find active SparkSession” bug by passing in the SparkSession when resolving tables in the DeltaTableBuilder.

Credits:
Helge Brügner, Jiaheng Tang, Mitchell Riley, Ryan Johnson, Scott Sandre, Venki Korukanti, Jintao Shen, Yann Byron

Don't miss a new delta release

NewReleases is sending notifications on new releases.