github apache/arrow-rs 60.0.0
arrow 60.0.0
on GitHub

8 hours ago

Changelog

60.0.0 - (2026-09-10)

Full Changelog

Breaking changes

  • minor : batches_to_flight_data() should not require ownership of arguments by @Rich-T-kid in #11010
  • Add PageIndexBuilder and PageIndexProvider for Parquet page indexes by @etseidl in #10842
  • [Parquet] ALP encoder/decoder support by @sdf-jkl in #9372
  • Remove force_validate feature from arrow-cast by @Jefffrey in #10950
  • parquet: introduce FILE logical type by @brkyvz in #10109
  • Add OffsetBuffer::first/last to always return offset & remove some unwraps by @emilk in #10759
  • fix(ffi): make FFI_ArrowSchema::with_metadata unsafe by @bit2swaz in #10764
  • Expose &Arc[Buffer] in GenericByteViewArray::data_buffers for cheaper cloning by @YimingQiao in #10708
  • parquet: Add new PageIndex struct to encapsulate column and offset indexes by @etseidl in #10719
  • seal ToByteSlice trait by @Rich-T-kid in #10701
  • fix(ffi): make FFI struct fields private to close Drop soundness hole by @bit2swaz in #10431
  • Upgrade rand to 0.10 by @Jefffrey in #10566
  • parquet: Remove deprecated functions by @etseidl in #10565
  • parquet: return an error instead of panicking when a pushed buffer's length does not match its range by @ranflarion in #10564
  • Implement Parquet GH-583 INT96 timestamp ColumnOrder by @etseidl in #10106
  • chore: bump object_store to 0.14.1 by @comphead in #10375
  • Deprecate parquet::basic::ColumnOrder::sort_order_for_type by @etseidl in #10104
  • Remove old deprecated items (ffi::export_array_into_raw and IPC StreamReader::try_new_unbuffered) by @Jefffrey in #10479
  • formalize the default map field names to match default arrow spec by @alamb in #10517
  • Bump MSRV to 1.88 by @alamb in #10254
  • Remove flight-sql-experimental feature from arrow-flight by @Harshalj2108 in #10335
  • Use Metadata struct instead of HashMap<String, String> for metadata: Ordered; cheap to clone by @emilk in #10075
  • Allow reading and writing more than 32k Parquet row groups by @etseidl in #10149
  • Implement PARQUET-2249: Introduce IEEE 754 total order by @etseidl in #9619

Enhancements

Bug fixes

  • fix(arrow-ipc): return an error for a DictionaryBatch without its data by @PlenoraETL in #11020
  • Fix interleave on zero-width FixedSizeListArray by @tpoterba in #11026
  • fix: validate union child bounds without truncating lengths by @sunchao in #10847
  • fix: Sparse UnionArray equality should ignore unselected children by @neilconway in #11009
  • fix(arrow-buffer): add a fallible collect_bool by @Cintu07 in #10984
  • arrow-data: remove unnecessary & incorrect null buffer check in ArrayData::validate by @alliasgher in #10709
  • fix(arrow-ipc): write 8-byte i64 length prefix for compressed IPC buffers on wasm32 by @Narendran-KT in #10989
  • fix(arrow-data): align struct null validation with parent offset by @linhongyu510 in #10970
  • fix(parquet): reject Thrift list sizes larger than remaining input by @M-Tesla in #10979
  • fix(parquet): cut data page byte-budget mini-batches on exact value counts by @adriangb in #10554
  • fix(arrow-string): reject FixedSizeBinary concat widths that overflow i32 by @Cintu07 in #10981
  • fix(arrow-array): align FFI buffers before validation under force_validate by @bit2swaz in #10798
  • fix(arrow-avro): reset union offsets after flush by @linhongyu510 in #10971
  • fix(arrow-buffer): prevent use-after-free in Buffer::shrink_to_fit when MemoryReservation::resize panics by @Rich-T-kid in #10932
  • fix(arrow-data): account for struct parent offsets by @jaideeppyne in #10934
  • arith: fallback to dividing decimals digitwise when regular scale into div overflows by @bharadwaj-pendyala in #10911
  • Fix pool claim race condition by @cetra3 in #10301
  • fix(arrow-avro): decode records with no fields by @sqd in #10771
  • fix: Consolidate string-to-decimal parsing into a single code path by @neilconway in #10850
  • fix(arrow-data): don't double-count offset when slicing struct ArrayData by @jaideeppyne in #10835
  • fix(arrow-select): treat null take indices as null for dense union and REE by @yongster in #10909
  • fix(arrow-avro): preserve object_store error source in the async reader by @ranflarion in #10496
  • Fix FFI import of zero-length Utf8/Binary arrays at a non-zero offset by @AndreaBozzo in #10916
  • Fix take on zero-width FixedSizeListArray by @1fanwang in #10915
  • fix: Correct two bugs when formatting decimal values by @neilconway in #10869
  • fix: Reject decimal arrays with different scales in make_comparator by @neilconway in #10864
  • fix: Avoid incorrect min/max stats for BYTE_ARRAY decimals of unequal byte lengths by @neilconway in #10861
  • fix: Correctly decode zero-valued Decimal256s in integration tests by @neilconway in #10873
  • fix: casting Decimal256 to signed integers returned wrong results by @neilconway in #10857
  • fix: Avoid spurious assert when Parquet decimal has scale == precision by @neilconway in #10875
  • fix(ffi): report the producer's C stream error message, don't unwrap it by @fornwall in #10844
  • Add proptest fuzzing to parquet-variant and implement fixes for findings by @cetra3 in #10352
  • fix(arrow-integration-test): preserve field metadata in JSON by @1fanwang in #10811
  • fix(arrow-select): handle all-empty run arrays in concat_run_arrays by @thorfour in #10782
  • fix(parquet): keep DELTA_BYTE_ARRAY dedup for values larger than the page size limit by @adriangb in #10505
  • fix(parquet): split row groups iteratively instead of recursively by @1fanwang in #10626
  • fix: stop try_ functions from panicking by @emilk in #10730
  • fix: try_binary dropped the logical nulls of run and dictionary arrays by @emilk in #10748
  • fix(arrow-json): validate REE nullability by @WaterWhisperer in #10749
  • fix(arrow-cast): do not truncate integers when casting to Decimal32/64 by @yongster in #10707
  • fix(arrow-data): don't panic on dictionary key overflow in interleave/concat by @okhsunrog in #10675
  • fix: don't panic in GenericByteArray::from_iter_values by @emilk in #10729
  • fix(arrow-ipc): return an error instead of panicking on IPC file with no footer schema by @bit2swaz in #10744
  • GH-10676: Fix IPC StreamReader projected schema by @codeAnqiang-ma in #10677
  • fix(arrow-arith): handle sliced boolean arrays in and_not by @yongster in #10699
  • Error instead of silent Variant::Null for unimplemented typed_value types by @raghav-reglobe in #10598
  • fix(arrow-ipc): return an error instead of panicking on malformed IPC schema by @bit2swaz in #10647
  • fix: Reject 0 for write_batch_size, data_page_row_count_limit by @neilconway in #10660
  • GH-10613: Fix IPC FileReader projected schema by @dhruvxvaishnav in #10627
  • GH-10382: Fix CSV RecordBatchReader projected schema by @Boulea7 in #10611
  • parquet-geospatial: fix wraparound bound width (#10534) by @subotac in #10572
  • [Variant] Unshredded VariantArray producers annotate the value field as nullable (spec requires required) by @sdf-jkl in #10320
  • [Variant] Fix the variant shred logic by @klion26 in #10157
  • fix: bound IPC message allocations by the bytes actually read by @ranflarion in #10522
  • Fix AsRef impl for Buffer and generically wrap Bytes::from_owner by @adamreichold in #10571
  • Parquet: tolerate mixed row-group ordinal metadata on read by @zhuqi-lucas in #10449
  • Fix Flight LargeList schema encoding by @yinli-systems in #10298
  • fix(arrow-data): account for view payload buffers in slice memory size by @wterrr in #10519
  • fix(arrow-cast): preserve null dictionary values when casting to a view by @Abhisheklearn12 in #10510
  • fix(arrow-avro): fix split sync marker assembly and validate per-block sync markers by @ranflarion in #10497
  • fix(arrow-buffer): preserve bits outside the requested range in in-place bitwise ops by @haohuaijin in #10444

Performance improvements

  • perf(ipc): use Vec instead of BufferBuilder for run-ends re-encoding (#10245) by @ax1s-x1zz in #11005
  • perf(nullbuf::expand) non aligned counts by @Rich-T-kid in #10980
  • perf: speed up FixedSizeBinary filter kernel by @Rich-T-kid in #10993
  • perf: Count digits to skip precision check when parsing decimals by @neilconway in #10998
  • perf(arrow-buffer): Format i256 values without num-bigint by @neilconway in #11000
  • perf(NullBuffer::Exapand) optimize 8 byte chunks by @Rich-T-kid in #10976
  • perf(arrow-cast): Scan integer and fractional decimal digits in separate loops by @neilconway in #10974
  • perf(arrow-array): speed up RunArray::try_new by avoiding the ArrayData roundtrip by @liamzwbao in #10807
  • perf: optimize take bool & take null buffers by @Rich-T-kid in #10813
  • perf: optimize rank for byte views via caching prefix keys by @YUZHEthefool in #10605
  • perf(parquet): resolve per-column writer properties once per column by @adriangb in #10880
  • [Variant] Skip UTF-8 revalidation when reading validated metadata dictionary entries by @adriangb in #10879
  • perf(arrow-array): Reduce binary size 0.5%: do PrimitiveArray → ArrayData conversion once, not per type by @alamb in #10893
  • perf(take): avoid bounds checks and speed up take on List by @Rich-T-kid in #10812
  • perf: Use interleave for fragmented zip masks by @Jo2234 in #10368
  • perf(arrow-array): compile array Debug formatting helpers once, not per primitive type (~2% binary size reduction) by @alamb in #10890
  • perf : optimize take(fixed_sized_list) by @Rich-T-kid in #10441
  • Improve arrow-avro decoding for one-record messages by @jordepic in #10713
  • Replace BufferBuilder with Vec in cast_byte_container by @zhangxinyao88 in #10867
  • perf: Write compressed directly to buffer by @EmilyMatt in #10833
  • (arrow-row) Replace BufferBuilder with Vec in decode_binary by @zhangxinyao88 in #10851
  • Use Vec for primitive array unary operations by @kowanietz in #10783
  • introduce GenericByteDictionaryBuilder::append_array() by @Rich-T-kid in #10765
  • perf(variant): build BinaryView arrays directly by @cakeni in #10640
  • Optimize PLAIN parquet string column to dictionary array by @Rich-T-kid in #10614
  • Replace BufferBuilder with Vec in fixed-size binary take by @kowanietz in #10773
  • perf: avoid re-allocation if buffer is not shared during BooleanArray::take_n_true by @Rich-T-kid in #10438
  • Replace concat_elements_utf8_many BufferBuilders with Vec by @cakeni in #10633
  • Replace concat_elements_bytes BufferBuilders with Vec by @cakeni in #10632
  • Replace take_run BufferBuilders with Vec by @cakeni in #10631
  • Replace BufferBuilder with Vec in sort_run_downcasted by @cakeni in #10630
  • Replace BufferBuilder with Vec in MapArrayDecoder by @cakeni in #10629
  • Replace substring BufferBuilders with Vec by @cakeni in #10634
  • perf(arrow-cmp): Speed up eq/neq of a byte-view array against a short scalar by @giladkl in #10689
  • perf(arrow-cast): gate Dictionary -> View fast path on cardinality by @Abhisheklearn12 in #10436
  • perf(arrow-cast): optimize parsing of decimals from strings by @neilconway in #10668
  • perf(parquet): avoid redundant copies in mask-backed intersection/union by @haohuaijin in #10446
  • perf(arrow-json): mark NullableEncoder with inline by @MassivePizza in #10612
  • perf(arrow-cast): improve custom temporal formatting performance in ArrayFormatter by @lyne7-sc in #10594
  • perf(parquet): scan DELTA_BYTE_ARRAY shared prefixes a block at a time by @adriangb in #10549
  • perf(parquet): reuse MaskSelection's cached selectors when converting to selectors by @haohuaijin in #10443

Documentation updates

  • chore: Fix typos by @neilconway in #10956
  • chore: fix typos in code comments by @MsfPablo in #10943
  • docs: update primitive docs to focus on vec by @Rich-T-kid in #10898
  • docs: provide performance guide for primitive by @Rich-T-kid in #10895
  • docs : add safety comments by @Rich-T-kid in #10843
  • docs : add saftey comments to unsafe callsites by @Rich-T-kid in #10839
  • docs(arrow-data): Document how ArrayData::offset applies to buffers, child_data and nulls by @alamb in #10838
  • docs: Revise Parquet metadata module documentation by @etseidl in #10796
  • Document how to customize JSON encoding via EncoderFactory by @alamb in #10741
  • minor/doc: document saftey for extend_bool_trusted_len by @Rich-T-kid in #10715
  • docs: add # Panics sections across the workspace by @emilk in #10656
  • doc(parquet): Distinguish difference between ColumnOrder and SortOrder by @etseidl in #10573
  • docs(arrow-select): document the InProgressArray copy methods by @alamb in #10057
  • chore: fix readmes by @Jefffrey in #10472
  • docs(arrow-select): document FilterSelection / FilterPredicate::selection (docs for #9755) by @alamb in #10056
  • Update release schedule on README by @alamb in #10528

Miscellaneous

  • fix(parquet): skip miniblocks wider than 64 values instead of erroring by @bharadwaj-pendyala in #11021
  • [Parquet] Populate bloom filters from the dictionary while a column is dictionary encoded by @ranflarion in #10966
  • [Parquet] Add writer option to skip bloom filters for column chunks whose data pages are all dictionary encoded by @ranflarion in #10963
  • Json decoder factory by @hareshkh in #10670
  • Return errors instead of panicking in fallible functions by @emilk in #10755
  • Parquet: Make page index decoders public by @etseidl in #10899
  • add const generic to take kernel by @Rich-T-kid in #10820
  • [Variant] Add nested object and array interoperability tests by @cakeni in #10637
  • fix(variant): improve malformed object field diagnostics by @cakeni in #10638
  • perf: skip checking if there is null if already materialized by @rluvaton in #10756
  • [Parquet] Implement num distinct values for parquet writer by @Rich-T-kid in #10654
  • fix(parquet): prevent cached Mask reads from crossing unloaded sparse pages by @hhhizzz in #10735
  • Fix: Error rather than panic on invalid dictionary index bit width in Parquet reader by @dhruvxvaishnav in #10725
  • bench: add binary/string to dict cast benchmarks in cast_kernels by @Rich-T-kid in #10763
  • bench(parquet): cover large dictionary values in arrow_reader by @AarryaSaraf in #10691
  • perf(parquet): compare before copying min/max in ByteArrayEncoder by @adriangb in #10745
  • feat: Add RowSelection::total_row_count by @haohuaijin in #10704
Check out latest releases or
releases around apache/arrow-rs 60.0.0

Don't miss a new arrow-rs release

NewReleases is sending notifications on new releases.

Get notifications