github apache/arrow-rs 60.2.0
arrow 60.2.0

one hour ago

Changelog

60.2.0 - (2026-11-18)

Full Changelog

Breaking changes

  • fix(parquet): error when writing bloom filters for encrypted columns by @alamb in #12165

Enhancements

  • fix(parquet): accept empty Thrift lists regardless of declared element type by @bharadwaj-pendyala in #12266
  • feat(arrow-array): add GenericByteArray::try_new_repeated (#12143) by @Ayush-CS-89112521 in #12149
  • [Variant] Remove TopLevelRowSink by @sdf-jkl in #12092
  • feat(parquet): allow updating cached file key-value metadata by @manuzhang in #12117
  • Add Sbbf::check_hashes and Sbbf::union, make Sbbf::insert_hash public by @adriangb in #11641
  • feat(arrow-string): support run-end encoded operands in like kernels by @asubiotto in #12154
  • feat(parquet): Improved back-end storage for PageIndex by @etseidl in #11475
  • feat: support casting strings to durations by @miguelcorderopamphile in #11572
  • Add ArrowSbbf to check Arrow arrays against Parquet bloom filters by @bhaskargurram-ai in #11549
  • feat(parquet): Add Bloom filter access to ParquetMetaDataReader by @etseidl in #12090
  • Optimize byte-array dictionary gathering in arrow-row by @JonSnow1807 in #12113
  • feat(arrow-cast): Distinguish unsupported casts from value conversion failures by @ZENOTME in #11507
  • Perf: Improve dictionary slice concatenation by @Rich-T-kid in #12008
  • Use fallible MutableBuffer methods in fallible APIs by @Rich-T-kid in #11912
  • feat(json): add opt-in non-finite float parsing to JSON reader by @dentiny in #11725
  • parquet: Enforce PackedDecoder invariant upon construction by @etseidl in #11981
  • Parquet: map the FILE logical type to the arrow.parquet.file extension type by @dejankrak-db in #11877
  • feat(parquet): Consolidate bloom filter reads with a push decoder by @etseidl in #11975
  • [Variant] Move VariantType into arrow-schema canonical extensions by @sdf-jkl in #12033
  • docs: add README to published crates missing one by @EaswarKilari in #12022
  • feat(parquet): Add async Sbbf::read_from_column_chunk_async Bloom filter API by @etseidl in #11953
  • feat(parquet): Make ParquetMetaData more easily clonable by @etseidl in #11297
  • [Variant] Move VariantType from parquet-variant-compute to parquet-variant by @sdf-jkl in #12005
  • feat(arrow-array): support Date32/Date64 in record_batch!/create_array! macros by @gabotechs in #11962
  • fix(arrow-array): make record_batch! usable without a direct arrow_schema dependency by @tz70s in #11734
  • fix(arrow-cast): parse ISO 8601 timestamps without seconds by @amitvijapur in #11179
  • docs(flight): document Flight SQL schema listing by @Rich-T-kid in #11897
  • feat(parquet): expose encryption key metadata by column name by @autoCAT6 in #11733
  • Re-export GetTableTypesBuilder, GetSqlInfoBuilder and GetXdbcTypeInfoBuilder from arrow_flight::sql::metadata by @morristai in #11846
  • fix(arrow-select): include column name in BatchCoalescer errors by @Rich-T-kid in #11870

Bug fixes

  • [Variant] Validate storage fields in VariantType by @sdf-jkl in #12245
  • fix(parquet-variant-compute): reject unsupported types in empty and all-null cast_to_variant input by @dayan02dev in #12246
  • fix(arrow-string): handle dictionaries without values by @Rich-T-kid in #12253
  • fix(arrow-ipc): do not write a validity bitmap for RunEndEncoded in V4 by @Mola-maker in #12267
  • Fix CSV schema inference for Unicode literals by @peterolkhov in #12144
  • fix(parquet): count only shared predicate cache hits in cache-read metrics by @dragonknightk in #12180
  • fix(arrow-buffer): return None from NullBufferBuilder when it has no nulls by @Mola-maker in #12251
  • fix(arrow-array): keep precision and scale in null_if_overflow_precision by @dayan02dev in #12242
  • fix(parquet): Harden Thrift decoding of binary lengths and collection counts by @etseidl in #12142
  • fix(variant): preserve nulls for missing shredded list indexes by @sdf-jkl in #12224
  • fix(parquet): check PLAIN byte array end offsets for overflow in read by @hsdfat in #12215
  • Fix escaping in DataType parsing and list field names by @timmyb32r in #11931
  • fix(parquet): honor requested dictionary value type by @Rich-T-kid in #12161
  • fix(parquet): return error instead of panic on corrupt PLAIN BYTE_ARRAY pages by @viirya in #12179
  • fix(arrow-array): avoid null bitmap allocation in non-nullable StructArray field validation by @llxgdtop in #12124
  • fix(arrow-cast): write timestamps whose zone offset has seconds in UTC by @hsdfat in #12102
  • fix(parquet): return error on negative DELTA_BYTE_ARRAY lengths in arrow decoder by @hsdfat in #12108
  • fix(arrow-arith): match DuckDB's rounding when multiplying an interval by a float by @EaswarKilari in #12029
  • fix(ffi): read byte array offsets unaligned in buffer_len by @AndreaBozzo in #12010
  • fix(variant): avoid overflow for minimum scale by @Phoenix500526 in #12104
  • fix(array): validate decimals with negative scale by @Phoenix500526 in #12101
  • fix(arrow-ipc): cap the decompression buffer reserved from the declared length by @geoHeil in #12077
  • fix(arrow-ipc): honor skip_validation for record batches in StreamDecoder by @comphead in #12094
  • fix(parquet): return error for out-of-range FLBA dictionary index by @linzch3 in #12065
  • fix(arrow-json): reject out-of-range float timestamps instead of saturating by @hsdfat in #12007
  • fix(arrow-string): saturate the length in Utf8View/BinaryView substring by @hsdfat in #12006
  • fix(arrow-array): return an error instead of wrapping on MapBuilder offset overflow by @llxgdtop in #12052
  • fix(parquet): cap byte-budget level windows by value count by @adriangb in #11996
  • fix(parquet): copy Decimal/Float16 column chunk statistics instead of aliasing the written values by @xudong963 in #12028
  • fix(parquet): do not write a dictionary page in empty row group by @jhorstmann in #12003
  • Resolve DST-ambiguous and nonexistent local times in string_to_datetime by @adriangb in #11054
  • Accept buffers with trailing bytes in ArrayData validation instead of panicking by @vntoinekaio in #11944
  • fix(parquet): return byte array overflow and map decoding errors by @hhhizzz in #11791
  • fix(parquet): write specification_version = 1 on the VARIANT logical type by @raghuvanshraj in #11961
  • fix(parquet): reject truncated definition level runs by @efegokdemir in #11700
  • Fix memory reservation leak in Buffer::into_vec by @timmyb32r in #11878
  • fix(parquet): skip empty CDC page flushes in add_data_page by @M-Tesla in #11003
  • fix(parquet): count levels, not rows, when marking null pages in the column index by @mightsleep in #11828
  • fix(arrow-select): merge dictionaries in zip instead of panicking on key overflow by @ump45nose in #11663
  • fix(arrow-select): merge dictionaries in merge and merge_n instead of panicking on key overflow by @ninglinLiu in #11925
  • fix(ipc): decode zero-length bodies without EOS by @Phoenix500526 in #11967
  • fix(parquet): validate Arrow writer shapes and map-key schemas by @HippoBaro in #11838
  • fix(arrow-ipc): return an error instead of panicking on out-of-bounds IPC buffers by @wleeaf in #11881
  • fix(arrow-avro): resolve named types declared with a full name by @sebxt in #11730
  • fix(parquet): Properly account for boolean lists in Parquet Thrift skipping code by @etseidl in #11923
  • Fix incorrect reversed DataType::contains check for union type by @timmyb32r in #11826
  • Reject merging Fields of unions with different modes by @timmyb32r in #11827
  • fix(arrow-array): return overflow error instead of panicking when negating i32::MIN months by @choksi2212 in #11611
  • fix(arrow-select): preserve nullif nulls for REE and Union by @yongster in #10991
  • fix(arrow-avro): keep nulls when reading strings as Utf8View by @ninglinLiu in #11915
  • fix(parquet): apply max_row_group_bytes to the first batch of a row group by @Tpt in #11798
  • fix(arrow-data): extend Interval(DayTime) as IntervalDayTime in MutableArrayData by @ninglinLiu in #11906
  • arrow-select: error on empty input and keep row count for column-less batches in interleave_record_batch by @princejain756 in #11845
  • fix(array-data): return an error on FixedSizeList validation overflow by @mvanhorn in #11859
  • fix(arrow-ipc): return an error instead of panicking on unions with more than 128 fields by @wleeaf in #11882
  • fix(arrow-ipc): reject negative FixedSizeBinary byte width in IPC schema by @wleeaf in #11883
  • fix(arrow-array): reset the lone offset of an empty FFI byte array by @ahlag in #11661

Performance improvements

  • perf(coalesce): hoist indexed ByteView buffer checks by @Rich-T-kid in #12212
  • perf(arrow-arith): inline decimal div and rem by a scalar into try_unary by @mbutrovich in #12080
  • perf(arrow-arith): divide integers by a scalar with a precomputed divisor by @mbutrovich in #12076
  • Use non-null value counts for Parquet dictionary caches by @JonSnow1807 in #12122
  • perf(coalesce): directly copy sparse UInt32 selections by @Rich-T-kid in #11174
  • perf(parquet): always inline OffsetBuffer::try_push by @viirya in #12177
  • perf(arrow): fuse Boolean filter copying by @Rich-T-kid in #11918
  • perf: Add a dictionary cache for parquet dict types by @EmilyMatt in #11982
  • perf(arrow-ipc): reuse lz4 frame encoder buffers across IPC writes by @gabotechs in #12084
  • perf(arrow-buffer): pack bools branchlessly in BooleanBufferBuilder::append_slice by @viirya in #12024
  • perf(arrow): coalesce dictionary keys without concat by @Rich-T-kid in #11743
  • perf(arrow-ipc): find dictionary fields without collecting the schema by @asubiotto in #12071
  • perf(arrow-select): zip a primitive array with a scalar without MutableArrayData by @mightsleep in #11559
  • perf(arrow-row): avoid encoding oversized dictionary values by @Rich-T-kid in #12043
  • perf(arrow-buffer): add precomputed divisors for 32-bit and 64-bit integers by @mbutrovich in #11467
  • Use ahash and a single lookup in PrimitiveDictionaryBuilder by @mhk197 in #12057
  • perf(parquet): avoid repeated page-index range discovery by @sunchao in #11276
  • perf(arrow-buffer): faster count_set_bits on aarch64 by summing byte popcounts before widening by @mbutrovich in #11818
  • perf(parquet): share Arrow write-schema plans across row groups by @HippoBaro in #11971
  • perf: optimize building out-of-line views in MutableArrayData by @neilconway in #11992
  • perf(parquet): optimize dictionary computation for byte arrays by @rluvaton in #11524
  • perf(arrow): specialize sparse list filtering by @Rich-T-kid in #11744
  • perf(parquet): convert FIXED_LEN_BYTE_ARRAY decimals with a width-specialized loop by @yihua in #11910
  • perf(coalesce): specialize sparse byte array filters by @Rich-T-kid in #11696
  • perf(variant): avoid bracket path string allocations by @DKER2 in #11232
  • perf: Optimize appending out-of-line views in concat, BatchCoalescer by @neilconway in #11770
  • perf(parquet-variant): Avoid intermediate string allocations in JSON serialization by @rizzabhdwaj in #11402
  • fix(parquet): release bloom filter capacity after folding by @starpact in #11701
  • perf(arrow): specialize Boolean filtering by @Rich-T-kid in #11732
  • perf(arrow-array): skip DataType drop glue in generic temporal converters by @mbutrovich in #11896
  • [Parquet] Create zstd contexts on first use by @azecevic000 in #11858
  • feat(buffer): add NullBufferBuilder::append_word by @Rich-T-kid in #11867
  • perf: Do not compute the entire byte view buffer size in set_source by @nuno-faria in #11830

Documentation updates

  • docs(parquet): add SAFETY comments for build_unchecked in array readers by @EaswarKilari in #12119
  • docs(arrow-select): document shorter filter predicates by @WaterWhisperer in #12116
  • docs: fix NaN ordering in min/max rustdoc by @EaswarKilari in #12021
  • docs: document RunArray logical offset/len and child array slicing by @alamb in #11710
  • docs: make fork CI setup harder to miss by @Rich-T-kid in #11914

Miscellaneous

  • Fix: prevent time64us and time64ns from wrapping out-of-bounds seconds to valid times (#11899) by @CodeMasterXZT

Don't miss a new arrow-rs release

NewReleases is sending notifications on new releases.