jorgecarleitao/arrow2 v0.8.0

on GitHub

A new release is here 🚀🚀🚀

This release has so many important new features and bug fixes that will be summarized as: thank you everyone for all the issues and PRs that resulted in this release (in order of appearance) 🙇🙇🙇🙇:

• yjshen
• sundy-li
• simonvandel
• illumination-k
• Dandandan
• flaneur2020
• 1aguna
• ritchie46
• yjhmelody

Full Changelog

Breaking changes:

• Made CSV write options use chrono formatting by default #624
• Add compression to IpcWriteOptions #570
• Made cast accept CastOptions parameter #569
• Simplified ArrowError #640 (jorgecarleitao)
• Use DynComparator for lexsort and partition #637 (yjshen)
• Split "compute" feature #634 (jorgecarleitao)
• Removed unneeded trait. #628 (jorgecarleitao)
• Sealed 2 traits to forbid downstream implementations #621 (jorgecarleitao)
• Simplified arithmetics compute #607 (jorgecarleitao)
• Refactored comparison Operator #604 (jorgecarleitao)
• Simplified dictionary indexes #584 (jorgecarleitao)
• Simplified IPC APIs #576 (jorgecarleitao)
• Simplified IPC stream writer / remove finish on drop from stream writer #575 (jorgecarleitao)
• Simplified trait in compute. #572 (jorgecarleitao)
• Compute: add partial option into CastOptions #561 (sundy-li)
• Introduced UnionMode enum #557 (simonvandel)
• Changed DataType::FixedSize*(i32) to DataType::FixedSize*(usize) #556 (simonvandel)

New features:

• Added support to write timestamps with timezones for CSV #623 (jorgecarleitao)
• Added support to read Avro files' metadata asynchronously #614 (jorgecarleitao)
• Added iterator for StructArray #613 (illumination-k)
• Added support to read snappy-compressed Avro #612 (jorgecarleitao)
• Added support to read decimal from csv #602 (jorgecarleitao)
• Added support to cast NullArray to all other types #589 (flaneur2020)
• Added support dictionaries in nested types over IPC #587 (jorgecarleitao)
• Added support to write Arrow IPC streams asynchronously #577 (jorgecarleitao)
• Added support to write compressed Arrow IPC (feather v2) #566 (jorgecarleitao)
• Added support for ffi for FixedSizeList and FixedSizeBinary #565 (jorgecarleitao)
• Added support for async csv reading. #562 (jorgecarleitao)
• Added support for bitwise operations #553 (1aguna)
• Added support to read StructArray from parquet #547 (jorgecarleitao)

Fixed bugs:

• Fixed error in reading nullable from Avro. #631 (jorgecarleitao)
• Fixed error in union FFI #625 (jorgecarleitao)
• Fixed error in computing projection in io::ipc::read::reader::FileReader #596 (illumination-k)
• Fixed error in compressing IPC LZ4 #593 (jorgecarleitao)
• Fixed growable of dictionaries negative keys #582 (ritchie46)
• Made substring kernel on utf8 take chars into account. #568 (ritchie46)
• Fixed error in passing sliced arrays via FFI #564 (jorgecarleitao)

Enhancements:

• Faster take with null values (2-3x) #633 (jorgecarleitao)
• Improved error message for missing feature in compressed parquet #632 (jorgecarleitao)
• Added to conversion to FixedSizeBinary #622 (ritchie46)
• Bumped confy-table #618 (jorgecarleitao)
• Made MutableArray Send + Sync #617 (jorgecarleitao)
• Removed most of allocations in IPC reading #611 (jorgecarleitao)
• Speed up boolean comparison kernels (~3x) #610 (Dandandan)
• Improved performance of decimal arithmetics #605 (jorgecarleitao)
• Simplified traits and added documentation #603 (jorgecarleitao)
• Improved performance of is_not_null. #600 (jorgecarleitao)
• Added len to every array #599 (jorgecarleitao)
• Added support for NullArray at FFI. #598 (jorgecarleitao)
• Optimized MutableBinaryArray #597 (jorgecarleitao)
• Speedup/simplify bitwise operations (avoid extra allocation) #586 (Dandandan)
• Improved performance of bitmap::from_trusted (3x) #578 (jorgecarleitao)
• Made bitmap not cache null count #563 (jorgecarleitao)
• Avoided redundant checks in creating an Utf8Array from MutableUtf8Array #560 (jorgecarleitao)
• Avoid unnecessary allocations #559 (simonvandel)
• Surfaced errors in reading from avro #558 (jorgecarleitao)

Documentation updates:

• Simplified example #619 (jorgecarleitao)
• Made example of parallel parquet write be over multiple batches #544 (jorgecarleitao)

Testing updates:

• Cleaned up benches #636 (jorgecarleitao)
• Ignored tests code in coverage report #615 (yjhmelody)
• Added more tests #601 (jorgecarleitao)
• Mitigated RUSTSEC-2020-0159 #595 (jorgecarleitao)
• Added more tests #591 (jorgecarleitao)