We are excited to announce the release of Delta Lake 2.0.2 on Apache Spark 3.2. This release contains important bug fixes and a few high-demand usability improvements over 2.0.1 and it is recommended that users update to 2.0.2. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.
- Documentation: https://docs.delta.io/2.0.2/index.html
- Maven artifacts: delta-core_2.12, delta-core_2.13, delta-contribs_2.12 delta_contribs_2.13, delta-storage, delta-storage-s3-dynamodb
- Python artifacts: https://pypi.org/project/delta-spark/2.0.2/
This release includes the following bug fixes and improvements:
- Record VACUUM operation in the transaction log. With this feature, VACUUM operations and their associated metrics (e.g.
numDeletedFiles
) will now show up in table history. - Support idempotent writes for DML operations. This feature adds idempotency to INSERTS/DELETE/UPDATE/MERGE etc. operations using SQL configurations
spark.databricks.delta.write.txnAppId
andspark.databricks.delta.write.txnVersion
.
Support passing Hadoop configurations via DeltaTable APIfrom delta.tables import DeltaTable hadoop_config = { "fs.azure.account.auth.type": "OAuth", "fs.azure.account.oauth.provider.type": "...", "fs.azure.account.oauth2.client.id": "...", "fs.azure.account.oauth2.client.secret": "...", "fs.azure.account.oauth2.client.endpoint": "..." } delta_table = DeltaTable.forPath(spark, <table-path>, hadoop_config)
- Minor convenience improvement to the
DeltaTableBuilder:executeZOrderBy
Java API which allows users to pass in varargs instead of a List. - Fail fast on malformed delta log JSON entries. Previously, Delta queries could return inaccurate results whenever JSON commits in the
_delta_log
were malformed. For example, anadd
action with a missing}
would be skipped. Now, queries will fail fast, preventing inaccurate results. - Fix “Could not find active SparkSession” bug by passing in the SparkSession when resolving tables in the DeltaTableBuilder.
Credits:
Helge Brügner, Jiaheng Tang, Mitchell Riley, Ryan Johnson, Scott Sandre, Venki Korukanti, Jintao Shen, Yann Byron