jorgecarleitao/arrow2 v0.4.0

on GitHub

A new release is here! 🎉🎉🎉

This one marked by a lot of enhancements to existing functionality. Of special mention:

• 🚀 improved performance of integer division by 4x-10x via strength division (@sundy-li and @ritchie46)
• 🚀 improved performance of concatenating nullable arrays by 4x
• 🚀 improved performance of comparisons by 2x-14x
• 🔧 moved most tests to a separate directory
• 🔧 Increased test coverage to over 80%
• 🔧 Made multiversion, lexical-core and serde-derive dependencies optional
• ✅ Added support for UnionArray (including FFI and IPC tests)
• ✅ Added support for FFI of Field

(full list below)

As usual, there is a small number of backward incompatible changes. The associated issues include the migration paths.

Finally, thank you to all contributors and reporters 🙇 In particular, thank you to polars and datafuse teams for the 🐛 reports. They help tremendously 💯

Full Changelog

Breaking changes:

• Change dictionary iterator of values from Arrays of one element to Scalars #335
• Align FFI API with arrow's C++ API #328
• Make *_compare_scalar not return Result #316
• Make io::print, get_value_display and get_display not return Result #286
• Add MetadataVersion to IPC interfaces #282
• Change DataType::Union to enable round trips in IPC #281
• Removed clone requirement in StructArray -> RecordBatch #307 (jorgecarleitao)
• Fixed error in reading a non-finished IPC stream. #302 (jorgecarleitao)
• Generalized ZipIterator to accept a BitmapIter #296 (jorgecarleitao)

New features:

• Added API to FFI Field #321 (jorgecarleitao)
• Added compare_scalar #317 (jorgecarleitao)
• Add UnionArray #283 (jorgecarleitao)

Fixed bugs:

• SliceIterator of last bytes is not correct #292
• Fixed error in displaying dictionaries with nulls in values #334 (jorgecarleitao)
• Fixed error in dict equality #333 (jorgecarleitao)
• Fixed small inconsistencies between compute::cast and compute::can_cast #295 (jorgecarleitao)
• Removed order implementation for days_ms / Interval(DayTime) #285 (jorgecarleitao)

Enhancements:

• Added support for remaining non-nested datatypes #336 (jorgecarleitao)
• Made multiversion and lexical-core optional #324 (jorgecarleitao)
• Improved performance of utf8 comparison (1.7x-4x) #322 (jorgecarleitao)
• Improved performance of boolean comparison (5x-14x) #318 (jorgecarleitao)
• Added trait TryPush #314 (jorgecarleitao)
• Added cast date32 -> i64 and date64 -> i32 #308 (ritchie46)
• Improved performance of comparison with SIMD feature flag (2x-3.5x) #305 (jorgecarleitao)
• Added support to read json to BinaryArray #304 (jorgecarleitao)
• Improved MutableFixedSizeBinaryArray #303 (jorgecarleitao)
• Improved MutablePrimitiveArray and MutableUtf8Array #299 (jorgecarleitao)
• Improved MutableBooleanArray #297 (jorgecarleitao)
• Improved performance of concatenating non-aligned validities (15x) #291 (jorgecarleitao)
• Added support for timestamps with tz and interval to io::print::write #287 (jorgecarleitao)
• Improved debug repr of buffers and bitmaps. #284 (jorgecarleitao)
• Cleaned up internals of json integration #280 (jorgecarleitao)
• Removed serde_derive dependency #279 (jorgecarleitao)
• Simplified IPC code. #277 (jorgecarleitao)
• Reduced dependencies from confi-table and enabled wasm on io_print feature. #276 (jorgecarleitao)
• Improve performance of rem_scalar/div_scalar for integer types (4x-10x) #275 (ritchie46)

Documentation updates:

• Cleaned examples and docs from old API. #330 (jorgecarleitao)
• Improved documentation #306 (jorgecarleitao)

Testing updates:

• Improved naming of testing workflows #315 (jorgecarleitao)
• Added tests to scalar API #300 (jorgecarleitao)
• Made CSV and JSON tests not use files. #290 (jorgecarleitao)
• Moved tests to integration tests #289 (jorgecarleitao)

Closed issues:

• Make parquet_read_record support async #331
• Panic due to SIMD comparison #312
• Bitmap::mutable line 155 may Panic/segfault #309
• IPC's StreamReader may abort due to excessive memory by overflowing a usized variable #301
• Improve performance of rem_scalar/div_scalar for integer types (4x-10x) #259