pola-rs/polars py-1.41.0

Python Polars 1.41.0

on GitHub

🏆 Highlights

• Add LazyFrame.gather (#27501)
• Nested common subplan elimination (#27340)
• Stabilize streaming engine (#27497)
• Speed up parquet metadata decode with hand-written Thrift (#27427)

⚠️ Deprecations

• Deprecate the StringCache (#27580)

🚀 Performance improvements

• Dispatch {list,arr}.{unique,n_unique,reverse} to group_by engine (#27278)
• Improve in-memory grouped non-null count (#27702)
• Factor shared conjuncts out of OR-of-ANDs predicates (#27627)
• Skip downloading IPC batches exceeding slice bounds (#27683)
• Avoid materializing broadcast list in list.shift (#27628)
• Optimise json_decode Datetime string parsing (#27559)
• Speed up to_numpy C-order via cache-blocked transpose (#27522)
• Optimize select(len()) for non-strict horizontal concat (#27516)
• Pushdown slices to inputs on left/right/full join (#27508)
• Don't infer CSV schema if schema is set (#27507)
• Nested common subplan elimination (#27340)
• Make is_in row-group pruning precise on null-containing haystacks (#27495)
• Don't do fused-multiply-add on scalars (#27479)
• List full fast path (#27477)
• Make is_in row-group pruning precise on multi-value lists (#27475)
• Add streaming GatherNode (#27465)
• Lower non-elementwise FunctionExprIR to ColumnarFunctionNode (#27462)
• Speed up parquet metadata decode with hand-written Thrift (#27427)

✨ Enhancements

• Use true division for the / operator in Polars SQL (#27391)
• Add Rust backend for Expr.has_nulls (#27590)
• Stabilize float16 (#27607)
• Add Expr.is_empty (#27583)
• Add support for the SQL FILTER clause for aggregate functions, and STRING_AGG (#27564)
• Make parquet FileMetadata prunable for IR-plan dispatch (#27535)
• Broadcast scalar input for list.slice (#27487)
• Add LazyFrame.gather (#27501)
• Add null_on_oob in {Expr/Series}.gather (#27327)
• Stabilize streaming engine (#27497)
• Process batched arr.eval on overflow boundaries (#27496)
• Process batched list.eval on overflow boundaries (#27483)
• Print SLICED UNION in LazyFrame explain (#27467)

🐞 Bug fixes

• Panic in scan of empty IPC with slice (#27708)
• Persist object_store rebuild state in cache (#27707)
• Sort flag on GroupsType only applies to first element (#27684)
• Invalid unwrap_unchecked when length isn't exact (#27685)
• Don't unwrap channel send in streaming join_asof (#27688)
• Fix merge_sorted panic when List in frame (#27568)
• Put AsOf join buffered Morsels back the front of the deque if we cannot process them rn (#27658)
• Fix skip_batches logic for NaN (#27673)
• Raise TypeError when calling next() directly on GroupBy objects (#27562)
• Data type comparison for extension types (#27632)
• Share last-morsel split budget across files in streaming multi-scan (#27630)
• Bytes scalars were not being broadcast in dataframe constructor (#27621)
• Reset the sort-options in Series::is_sorted() after row-encoding columns (#27614)
• Rayon deadlock with re-entrant io sources (#27600)
• Don't push negative-offset slices through HConcat (#27570)
• Logic error in streaming is_empty (#27602)
• Fix incorrect CSE with large is_in literal (#27575)
• AnonymousFunction can qualify as SQL aggregator (#26986)
• Fix CSPE panic in cloud (#27594)
• Set merge-join streaming node to Finished if its sending port is Done (#27572)
• Widen decimal precision on sum aggregation at runtime (#27579)
• Fix str.to_time was raising unnecessarily when input was all nulls (#27574)
• Prevent panic when switching from one extension dtype to another (#27566)
• Fix DataFrame.write_database(..., if_table_exists="append", engine="adbc") not handling missing tables correctly (#26913)
• Ensure json_decode doesn't fail for Date and Time string deserialization (#27554)
• Incorrect RUSTFLAGS passing in Makefile (#27555)
• Fix panic on reading IPC with 0-row compressed bitmap (#27551)
• Set HEAD_RESPONSE_SIZE_ESTIMATE to 0 (#27548)
• Fix lazy concat horizontal didn't raise on mismatching heights after projection pushdown (#27506)
• Prevent join panic when suffix="" and coalesce=True (#27376)
• Do not make a FastCount for csv if pre_slice is set (#27536)
• Support duplicate names in over (#27544)
• Reassign sequence numbers when distributing input morsels in streaming AsOf join node (#27538)
• Do not reverse dataframes when sorting with all-null key columns (#27517)
• Incorrect length check on streaming zip (#27505)
• Remove invalid type annotation Sequence[int] from DataFrame.\_\_setitem\_\_ key (#27355)
• Respect nulls_last for descending over(order_by) in group_by().agg() (#27486)
• Fix perf regression in scan_csv select(len()) when collected on streaming engine (#27504)
• Harden extend strictness (#27476)
• Prevent deadlock when using to_arrow() in a multithreaded context (#27472)
• Do not flatten sliced union (#27466)
• Prevent deadlock when using to_pandas() in multithreaded context (#27451)
• Struct rechunk bug and add Series::with_validity (#27446)
• Handle column indexing in read_parquet/read_csv with pyarrow reader (#27397)
• Export enum as ordered dictionary to arrow (#27432)
• Ensure sample() respects shuffle=False (#27248)
• Return empty DataFrame from concat_list with lit and empty column (#27305)
• Read parquet MAP columns without LogicalType annotation (#27404)
• Raise DuplicateError on parquet files with duplicate column names (#27399)

📖 Documentation

• Document Expr.list.__getitem__ (#27689)
• Add cloudpickle requirement (#27703)
• Clarify from_arrow schema ordering (#27493)
• Fix a typo in join_asof docstring (#27682)
• Clarify schema column order (#27681)
• Document horizontal string concatenation (#27542)
• Document all valid engine options on LazyFrame collect/sink/explain methods (#27374)
• Orchestration docs check (#27605)
• Drop redundant Pattern 2 from Dagster integration page (#27581)
• Update to remove Dockerhub PAT references (#27582)
• Modernize Dagster integration example for Polars Cloud (#27560)
• Use Polars random seed in sample example (#27537)
• Clarify full join description (#27530)
• Make expressions operations RNG deterministic (#27494)
• Document struct field order (#27492)
• Improve over:order_by description (#27520)
• Clarify join output columns (#27449)
• Document null propagation in pl.format (#27447)
• Document gzip support in read_csv (#27434)
• Add See Also sections for datetime docstrings (#27316)
• Polars On-Prem release (#27439)
• Rename to Polars On-Prem (#27435)
• Clarify null handling in unique operations (#27431)
• Document write_ipc buffer behavior with file=None (#27430)

📦 Build system

• Also split debug info in debug-release (#27609)
• Use split-debuginfo on linux (#27608)
• Bump deltalake to 1.5.1 in CI (#27387)

🛠️ Other improvements

• Remove redundant DSL::AGG::Unique (#27718)
• Harden against async blocking deadlocks (#27653)
• Print Python traceback when POLARS_TIMEOUT_MS is exceeded (#27657)
• Format missed in previous PR (#27700)
• Bump pytest and remove codspeed (#27686)
• Remove client-side allow_local_scans option for prepare_cloud_plan (#27663)
• Remove superfluous test (#27676)
• Cleanup streaming flags (#27671)
• Expose unordered concatenation in python visitor (#27666)
• Bump deltalake and fix CI (#27660)
• Add impl IntoAExprBuilder for ExprIR (#27656)
• Split _expand_selector_dicts into multiple functions so return type is simple and accurate (#27618)
• Update object_store patch repo (#27650)
• Match NumPy signature in DataFrame.__array__ and Series.__array__ (#27634)
• Add ImageVersion to rust-cache key (#27626)
• Run Pyrefly on tests (#27459)
• Fix is_empty test (#27597)
• Fix tz type difference pandas assert, take 2 (#27596)
• Fix CSPE panic in cloud (#27594)
• Fix tz type difference pandas assert (#27593)
• Add contributing note about conventional comments (#27543)
• Nested common subplan elimination (#27340)
• Deduplicate interns (#27470)
• Fix merge conflict in ColumnarFunction (#27464)
• Keep the schema ordered in scan projection pushdown (#27429)
• Remove unused type: ignore statements (#27360)
• Remove redundant PhysNodeKind::AsOfJoin::{left_right}_by fields (#27400)
• Resolve type-ignores in udfs.py (#27341)
• Bump rustls-webpki (#27382)

Thank you to all our contributors for making this release possible!
@0guban0v, @EndPositive, @JakubValtar, @Jesse-Bakker, @Kevin-Patyk, @Liyixin95, @MarcoGorelli, @NedJWestern, @Shoeboxam, @SuryaSunil1326, @TNieuwdorp, @alexander-beedie, @aryansri05, @ashler-herrick, @azimafroozeh, @carnarez, @coastalwhite, @dependabot[bot], @dsprenkels, @gab23r, @gautamvarmadatla, @ilya-pevzner, @jonathansergio, @junnythemarksman, @kdn36, @lun3x, @nameexhaustion, @orlp, @pablogsal, @ritchie46, @uurl, @waamm, @wence-, @wmoss, @xronocode and dependabot[bot]