delta-io/delta v2.0.2

Delta Lake 2.0.2

on GitHub

We are excited to announce the release of Delta Lake 2.0.2 on Apache Spark 3.2. This release contains important bug fixes and a few high-demand usability improvements over 2.0.1 and it is recommended that users update to 2.0.2. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

• Documentation: https://docs.delta.io/2.0.2/index.html
• Maven artifacts: delta-core_2.12, delta-core_2.13, delta-contribs_2.12 delta_contribs_2.13, delta-storage, delta-storage-s3-dynamodb
• Python artifacts: https://pypi.org/project/delta-spark/2.0.2/

This release includes the following bug fixes and improvements:

• Record VACUUM operation in the transaction log. With this feature, VACUUM operations and their associated metrics (e.g. numDeletedFiles) will now show up in table history.
• Support idempotent writes for DML operations. This feature adds idempotency to INSERTS/DELETE/UPDATE/MERGE etc. operations using SQL configurations spark.databricks.delta.write.txnAppId and spark.databricks.delta.write.txnVersion.
  Support passing Hadoop configurations via DeltaTable API

  from delta.tables import DeltaTable
  hadoop_config = {
    "fs.azure.account.auth.type": "OAuth",
    "fs.azure.account.oauth.provider.type": "...",
    "fs.azure.account.oauth2.client.id": "...",
    "fs.azure.account.oauth2.client.secret": "...",
    "fs.azure.account.oauth2.client.endpoint": "..."
  }
  delta_table = DeltaTable.forPath(spark, <table-path>, hadoop_config)

• Minor convenience improvement to the DeltaTableBuilder:executeZOrderBy Java API which allows users to pass in varargs instead of a List.
• Fail fast on malformed delta log JSON entries. Previously, Delta queries could return inaccurate results whenever JSON commits in the _delta_log were malformed. For example, an add action with a missing } would be skipped. Now, queries will fail fast, preventing inaccurate results.
• Fix “Could not find active SparkSession” bug by passing in the SparkSession when resolving tables in the DeltaTableBuilder.

Credits:
Helge Brügner, Jiaheng Tang, Mitchell Riley, Ryan Johnson, Scott Sandre, Venki Korukanti, Jintao Shen, Yann Byron