pola-rs/polars py-1.44.0

Python Polars 1.44.0

on GitHub

⚠️ Deprecations

• Deprecate rechunk parameter for all read/scan functions (#28063)
• Deprecate Expr.rechunk() (#28692)
• Deprecate struct.rename_fields() with an incorrect number of fields (#28672)

🚀 Performance improvements

• Add private env var toggle for HTTP rate limit (#28882)
• Relax default CloudRetryConfig for rate limit stability (#28885)
• Elide generic cross join on subquery decorrelation in equality predicates (#28876)
• cache CTE's in SQL layer (#28864)
• Allow CSPE if manual caches are set (#28859)
• Lower unqualified join predicates to inner joins in SQL (#28854)
• Rechunk mask for DataFrame filter (#28762)
• Improve performance of when/then/otherwise by masking out unevaluated elements (#28498)
• Drop unused projections in filter on streaming engine (#28713)
• Retain Parquet Partial metadata on filter (#28737)

✨ Enhancements

• Support integer fixed-array dot products (#28829)
• SQL correlated in (#28927)
• Support more join_types in join_where (#28880)
• Support Iceberg schema evolution (#28794)
• SQL conformance (#28494)
• Introduce RemoteEngine and a common base class for all engines (#28800)
• Support Iceberg table with V3 deletion vectors in native scan_iceberg (#28772)
• Support Iceberg snapshot properties (#28793)
• Expose sinked_paths_callback on sink_parquet as unstable parameter (#28814)
• Add infer_schema_files to CSV inference hint (#28809)
• Add native fixed-array dot (#28504)
• Export query metrics to polars_cloud (#28757)
• Support Iceberg object storage paths (#28634)
• More CSEE simplification (#28731)
• Improve error in Series constructor with name in context (#28743)
• Add adaptive HTTP rate-limiter for cloud IO (#28591)
• Add struct.drop() (#28666)
• Improve error message when CSV name de-duplication fails (#28658)
• Always keep first metadata per source for Parquet (#28661)
• Improve plan-time row estimates for multi-file parquet scans (#28380)
• Expose array and plugin function views in Python visitor (#28635)

🐞 Bug fixes

• SQL handle quantified comparisons against a subquery (#28929)
• Fix SQL subquery qualified lowering and cache row-index (#28926)
• Clippy-stable failures (#28919)
• Use uuidv7 for cloud observed queries (#28910)
• Unsoundness in AnyValue::into_static for structs (#28889)
• Invalidate suborders on projection (#28866)
• Ensure SQL function registry is available in isolated subqueries (#28881)
• Handle null in Expr.is_not_nan (#28883)
• Ensure eager unpivot column selection aligns with the lazy engine (#28846)
• Fix assertion error / incorrect data load from parquet written by fastparquet (#28656)
• Fix projection pushdown incorrect select() height (#28751)
• Restore monitoring as an engine-level parameter (#28821)
• Bincode serialization of IR hints (#28837)
• Retain datetime inference across CSV fallback (#28663)
• Avoid chunk fragmentation on bool agg min and max (#28789)
• Merge join with coalesce and empty suffix (#28799)
• Wrong function name call in test (#28801)
• Preserve row mapping in nested over expressions (#28767)
• Restore CI compatibility with NumPy 2.5.2 (#28782)
• Do not deduplicate inherently nondeterministic plan nodes (#28768)
• Fix OOC spilling of categoricals (#28761)
• Correct partitionable filter handling for pivot (#28749)
• Small typo in docstring (#28763)
• Avoid chunk fragmentation on bool agg any and all (#28754)
• Ignore nans in (min|max)_horizontal (#28710)
• Correct function display for str.replace_all (#28742)
• Exclude debug tests from nodebug streaming run (#28747)
• Convert predicate series dtype during prefiltered parquet reading (#28711)
• Do not remove nested caches when performing predicate pushdown past a cache node (#28740)
• Correct handling of nulls at the start of business day functions (#28703)
• Do not panic when exceeding Parquet metadata scan cache (#28721)
• Remove cryptographic hashing from CSPE (#28716)
• Unsoundness in rayon block_on (#28709)
• Check for (shallow) IR node equality on top of hashing in CSPE (#28506)
• Don't hand IO plugins a predicate they cannot evaluate (#28643)
• Raise on overflow in Decimal sum instead of silently wrapping (#28688)
• Preserve nulls when importing Arrow maps (#28680)
• Propagate null by column values in rolling_*_by (#27367)
• Fix self-referencing field in struct.with_fields with over (#28678)
• Fix Arrow buffer offset for Utf8 and Binary (#28662)
• Clamp group-by slice offset (#28579)
• Release the GIL while resolving the schema in __arrow_c_stream__ (#28642)
• Fix data corruption on Series from nested Arrow LargeList (#28632)

📖 Documentation

• Interlink join docstrings (#28907)
• Migration guide for 2.0 release (#28788)
• Clarify read_database_uri requirements for SQLAlchemy (#28366)
• Update engine tags (#28852)
• Clarify "that that side" comment in asof_join (#28847)
• Update mage.ai documentation link (#28813)
• Add engine tags (#28708)
• Document that arg_extremum functions return any index on ties (#28691)
• Fix Polars Cloud API reference link (#28693)

🛠️ Other improvements

• Bump runtime toolchains (#28924)
• Add min-publish-age for cargo (#28906)
• Tighten join invariants DSL (#28913)
• Bump Rust toolchain (#28900)
• Ensure that predicates are pushed down to the scans (#28857)
• Clarify "that that side" comment in asof_join (#28847)
• Restore monitoring as an engine-level parameter (#28821)
• Remove workaround for inconsistent CSE (#28834)
• Introduce RemoteEngine and a common base class for all engines (#28800)
• Add row count and file byte size to sinked paths callback (#28522)
• Use canonical expression IDs in CSEE (#28786)
• Enforce 100% type coverage in CI with Pyrefly (#28795)
• Attribute annotations for CredentialProviderAWS and CredentialProviderAzure (#28790)
• Attribute annotations for ScanCastOptions (#27906)
• Disable credentials in upload chunk size tests (#28745)
• Improve backtrace path formatting (#28732)
• Exclude debug tests from nodebug streaming run (#28747)
• Mark test_grouped_agg_parametric as slow (and add param ids) (#28715)
• Fix first expect in test_group_by_arg_max_boolean_26978 (#28690)
• Fix the name of struct.drop() in rename_fields() deprecation message (#28687)
• Add dist=loadgroup to the pytest config (#28676)
• Enable a lot of may_fail_auto_streaming tests (#28675)
• Re-enable test_extension() for streaming engine (#28611)
• Update test_hive_join_rewrite_semi_join test to work with streaming engine (#28610)

Thank you to all our contributors for making this release possible!
@0guban0v, @JakubValtar, @Joosboy, @Kevin-Patyk, @MatteoPossamai, @NicoOhR, @TNieuwdorp, @VedantMadane, @aarushkandukoori, @alexander-beedie, @azimafroozeh, @borchero, @carnarez, @dancsi, @dependabot[bot], @dsprenkels, @gautamvarmadatla, @jonasdedden, @jorenham, @kdn36, @lzcmian, @malhotrashivam, @matthewbayer, @mikhail5555, @nameexhaustion, @orlp, @ritchie46, @subotac and dependabot[bot]