jorgecarleitao/arrow2 v0.3.0

on GitHub

This is the first Rust implementation to support reading parquet on wasm, which opens up a whole new range of possibilities by allowing reading parquet directly on a browser without having to communicate data to a server.

This is also the first time an implementation of arrow in Rust offers native async support to read parquet, enabling consumers to perform ranged queries against blob storage without blocking. Check out an example here

Finally, this implementation now offers a Scalar API that we can build upon to support arrow's more exotic types such as maps and unions.

Thanks a lot to @sundy-li , @PsiACE , @ritchie46, @ghuls and @Dandandan for the contributions and discussions and to @Darksonn for all the patience and help that unblocked me when working on the async parquet support.

Full Changelog

Breaking changes:

• Renamed sum to sum_primitive #273
• Moved trait Index from array::Index to types::Index #272
• Added optional projection to IPC FileReader #271
• Added optional page_filter to parquet's RecordReader and get_page_iterator #270
• Renamed parquets' CompressionCodec to Compression #269

New features:

• Added support for FFI of dictionary-encoded arrays #267 (jorgecarleitao)
• Added support for projection pushdown on IPC files #264 (jorgecarleitao)
• Added support to read parquet asynchronously #260 (jorgecarleitao)
• Added support to filter parquet pages. #256 (jorgecarleitao)
• Added wrapping_cast to cast kernels #254 (sundy-li)
• Added support to parquet IO on wasm32 #239 (jorgecarleitao)
• Added support to round-trip dictionary arrays on parquet #232 (jorgecarleitao)
• Added Scalar API #56 (jorgecarleitao)

Fixed bugs:

• Fixed error in computing remainder of chunk iterator #262 (jorgecarleitao)
• Fixed error in slicing bitmap. #250 (jorgecarleitao)

Enhancements:

• Improve the performance in cast kernel using AsPrimitive trait in generic dispatch #252
• Poor performance in sort::sort_to_indices with limit option in arrow2 #245
• Support loading Feather v2 (IPC) files with more than 1 million tables #231
• Migrated to parquet2 v0.3 #265 (jorgecarleitao)
• Added more tests to cast and min/max #253 (jorgecarleitao)
• Prettytable is unmaintained. Change to comfy-table #251 (PsiACE)
• Added IndexRange to remove checks in hot loops #247 (jorgecarleitao)
• Make merge_sort_slices MergeSortSlices public #243 (sundy-li)

Documentation updates:

• Added example and guide section on compute #242 (jorgecarleitao)

Closed issues:

• Allow projection pushdown to IPC files #261
• Add support to write dictionary-encoded pages #211
• Make IpcWriteOptions easier to find. #120