jorgecarleitao/arrow2 v0.11.0

on GitHub

Arrow2 v0.11.0 is out!! 🎉🎉🎉

This release is mainly focus on improving upon the previous one on better parquet support. In particular, we have the main ingredients to read indexed parquet pages, which allow skipping deserializing individual pages, and since this version parquet files are written with page indexes. There is still some work to improve the frontend API to skip pages via statistics, which will be left for the next version.

This version also contains multiple bug fixes.

Thanks everyone that contributed to this release (individual PRs below)! 🙇

Changelog

Full Changelog

Breaking changes:

• Refactored parquet statistics deserialization #962 (jorgecarleitao)
• Made GroupFilter Send + Sync #947 (jorgecarleitao)

New features:

• Added support for non-ordered projections to IPC reading #961 (jorgecarleitao)
• Added support for reading indexed parquet pages #923 (jorgecarleitao)

Fixed bugs:

• Parquet regression: exceptions.ArrowErrorException: NotYetImplemented("Can't read Dictionary(UInt32, LargeUtf8, false) from parquet") #955
• Reading Parquet binary column panics during deserialization 'attempt to subtract with overflow` #944
• Reading Parquet file written by pyarrow with lz4 compression fails with OutOfSpec("Thrift out of range") #940
• Issues when trying to create a parquet file with FixedSizedListArray #691
• Fixed bug in writing csv with buffer resizing #965 (ritchie46)
• Fixed bug in reading binary parquet #945 (jorgecarleitao)
• Fixed error in writing fixedSizeListArray to parquet #941 (jorgecarleitao)
• Fixed support to read dict nested binary parquet #924 (jorgecarleitao)

Enhancements:

• Reduced memory usage in reading parquet #964 (jorgecarleitao)
• Simpler IPC code #939 (jorgecarleitao)
• don't allocate string when writing to csv #935 (ritchie46)
• Removed un-needed generic parameter #927 (jorgecarleitao)
• update to odbc-api 0.36.0 #925 (pacman82)

Documentation updates:

• Fixed example of parallel read via rayon #958 (jorgecarleitao)
• Fixed guide deployment #931 (jorgecarleitao)
• Typo fix #919 (bkmgit)

Testing updates:

• Fixed patch of integration tests #960 (jorgecarleitao)
• Added test for MapArray #942 (jorgecarleitao)
• Fixed wrong clippy warning #938 (jorgecarleitao)