jorgecarleitao/arrow2 v0.6.1

v0.6.0

on GitHub

(in crates as 0.6.1: I made a mistake in publishing). Anyways, another big release is here!

There are just too many improvements for a 22 days release - let's try to capture important mentions:

• Buffer and MutableBuffer are now compatible with Rust's std::Vec with no strings attached: everything continues to work, including FFI with the rest of the ecosystem! You can recover the previous behavior (of using cached-aligned allocations), via feature cache_aligned
• Added broad support to timestamp with timezones. Kudos to @VasanthakumarV for all the help.
• Added read Decimal from parquet. Kudos to @potter420 for the contribution.
• More improvements to performance. Kudos to @Dandandan and @ritchie46.
• Support to read from the Avro via feature io_avro

Full Changelog

Breaking changes:

• Bring MutableFixedSizeListArray to the spec used by the rest of the Mutable API #475
• Removed ALIGNMENT invariant from [Mutable]Buffer #449
• Un-nested compute::arithemtics::basic #461 (jorgecarleitao)
• Added more serialization options for csv writer. #453 (ritchie46)
• Changed validity from &Option<Bitmap> to Option<&Bitmap>. #431 (jorgecarleitao)
• Bumped parquet2 #422 (jorgecarleitao)
• Changed IPC FileWriter to own the writer. #420 (yjshen)
• Made DynComparator Send+Sync #414 (yjshen)

New features:

• Read Decimal from Parquet File #444
• Add IO read for Avro #401
• Added support to read Avro logical types, List,Enum, Duration and Fixed. #493 (jorgecarleitao)
• Added read Decimal from parquet #489 (potter420)
• Implement BitXor trait for Bitmap #485 (houqp)
• Added extend/extend_unchecked for MutableBooleanArray #478 (VasanthakumarV)
• expose shrink_to_fit to mutable arrays #467 (ritchie46)
• Added support for DataType::Map and MapArray #464 (jorgecarleitao)
• Extract parts of datetime #433 (VasanthakumarV)
• Added support to add an interval to a timestamp #417 (jorgecarleitao)
• Added support to read Avro. #406 (jorgecarleitao)
• Replaced own allocator by std::Vec. #385 (jorgecarleitao)

Fixed bugs:

• crash in parquet read #459
• Made writing stream to parquet require a non-static lifetime #471 (GrandChaman)
• Made importing from FFI unsafe #458 (jorgecarleitao)
• Fixed panic in division using nulls. #438 (jorgecarleitao)
• Fixed error writing dictionary extension to IPC #397 (jorgecarleitao)
• Fixed error in extending MutableBitmap #393 (jorgecarleitao)

Enhancements:

• Some compare function are not exported #349
• Investigate how to add support for timezones in timestamp #23
• Made hash work for extension type #487 (jorgecarleitao)
• Added extend/extend_unchecked for MutableBinaryArray #486 (VasanthakumarV)
• Improved inference and deserialization of CSV #483 (jorgecarleitao)
• Added GrowableFixedSizeList and improved MutableFixedSizeListArray #470 (jorgecarleitao)
• Added MutableBitmap::shrink_to_fit #468 (jorgecarleitao)
• Added MutableArray::as_box #450 (sd2k)
• Improved performance of sum aggregation via aligned loads (-10%) #445 (ritchie46)
• Removed assert from MutableBuffer::set_len #443 (ritchie46)
• Optimized null_count #442 (ritchie46)
• Improved performance of list iterator (- 10-20%) #441 (ritchie46)
• Improved performance of PrimitiveGrowable for nulls (-10%) #434 (jorgecarleitao)
• Allowed accessing validity without importing Array #432 (jorgecarleitao)
• Optimize hashing using ahash and multiversion (-30%) #428 (Dandandan)
• Improved performance of iterator of Utf8Array and BinaryArray (3-4x) #427 (jorgecarleitao)
• Improved performance of utf8 validation of large strings via simdutf8 (-40%) #426 (Dandandan)
• Added reading of parquet required dictionary-encoded binary. #419 (jorgecarleitao)
• Add extend/extend_unchecked for MutableUtf8Array #413 (VasanthakumarV)
• Added support to extract hours and years from timestamps with timezone #412 (jorgecarleitao)
• Added io_csv_read and io_csv_write feature #408 (ritchie46)
• Improve comparison docs and re-export the array-comparing function #404 (HagaiHargil)
• Added support to read dict-encoded required primitive types from parquet #402 (Dandandan)
• Added Array::with_validity #399 (ritchie46)

Documentation updates:

• Improved documentation #491 (jorgecarleitao)
• Added more API docs. #479 (jorgecarleitao)
• Added more documentation #476 (jorgecarleitao)
• Improved documentation #462 (jorgecarleitao)
• Added example showing parallel writes to parquet (x num_cores) #436 (jorgecarleitao)
• Improved documentation #430 (jorgecarleitao)
• [0.5] The docs io module has no submodules #390
• Made docs be compiled with feature full #391 (jorgecarleitao)

Testing updates:

• DRY via macro. #477 (jorgecarleitao)
• DRY of type check and len check code in compute #474 (yjhmelody)
• Added property testing #460 (jorgecarleitao)
• Added fmt to CI. #455 (jorgecarleitao)
• Simplified CI #452 (jorgecarleitao)
• fix filter kernels bench #440 (ritchie46)
• Reduced number of combinations in feature tests. #429 (jorgecarleitao)
• Move tests from src/compute/ to tests/ #423 (VasanthakumarV)
• Skipped some feature permutations. #411 (jorgecarleitao)
• Added tests to some invariants of unsafe #403 (jorgecarleitao)
• Added support to read and write extension types to and from parquet #396 (jorgecarleitao)
• Fix testing of SIMD #394 (jorgecarleitao)