We are excited to announce the release of Delta Lake 1.0.1 on Apache Spark™ 3.1, which back-ports bug fixes from Delta Lake 1.1.0 to Delta Lake 1.0.0.
The details of the fixed bugs are as follows:
-
Fix for rare data corruption issue on GCS - Experimental GCS support released in Delta Lake 1.0 has a rare bug that can lead to Delta tables being unreadable due to partially written transaction log files. This issue has now been fixed (1, 2).
-
Fix for the incorrect return object in Python
DeltaTable.convertToDelta()- This existing API now returns the correct Python object of typedelta.tables.DeltaTableinstead of an incorrectly-typed, and therefore unusable object. -
Fix for incorrect handling of special characters (e.g. spaces) in paths by MERGE/UPDATE/DELETE operations
-
Fix for Hadoop configurations not being used to write checkpoints
-
Improvements to DeltaTableBuilder API introduced in Delta 1.0.0
- Fix for bug that prevented passing of multiple partition columns in Python
DeltaTableBuilder.partitionBy. - Throw error when column data type is not specified.
- Fix for bug that prevented passing of multiple partition columns in Python
Credits
Jarred Parrett, Shixiong Zhu, Tathagata Das, Tom Lynch, Yijia Cui, Yaohua Zhao, gurunath