pola-rs/polars py-2.0.0-rc.2

Python Polars 2.0.0-rc.2

on GitHub

💥 Breaking changes

• Read Parquet ENUM type as pl.String (#29331)
• More map operations (#29296)
• Deprecate cut/qcut (#29329)
• Remove legacy POLARS_STREAMING_CHUNK_SIZE (#29046)
• Introduce the Map dtype (#28984)
• Update behavior of DataFrame/LazyFrame functions on 0-width input (#29099)

🚀 Performance improvements

• Order pushed parquet predicate columns by measured selectivity (#29397)
• Coerce float literals to decimal instead of casting the column (#29395)
• Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
• Restrict correlated SQL aggregates to requested keys (#29383)
• Prepare Parquet scans for row-group splitting in Polars Cloud (#29295)
• Skip row groups by join runtime ranges without a statistics frame (#29370)
• Reduce copy in scan_lines (#29310)
• Increase HTTP read rate-limit default (#29363)
• Evaluate a pushed parquet predicate sequentially per conjuct (#29352)
• Disable system certificates for CloudScheme::Http sources (#29284)
• Reduce rechunk in sort_in_place (#29343)
• Dynamic predicates for hash joins (#29312)
• Use HTTP suffix range for Parquet size and footer (#29308)
• Derive predicates from join conditions (#29304)
• Lower uncorrelated subqueries to semi joins and push semi/anti joins below inner joins (#29289)
• Make leaf name iterator unique (#29291)
• Don't clone the full frame per arm in when/then/otherwise (#29258)
• Push inner joins before outer joins and rewrite left-join-is-null to anti join (#29277)
• Use stats to decide cross join buffering side (#29270)
• Improve cache-removal and join-order cost estimates (#29263)
• Improve CSPE cost evaluation (#29250)
• Fuse group-by pre-select into node after partition (#29251)
• Inline hot small functions (#29244)
• Fix plan-time regressions in projection pushdown for wide frames (#28724)
• Rechunk before selecting in group-by pre-select (#29219)
• Reuse Iceberg data file sizes (#29063)
• Avoid gather for >= 75% cold keys in streaming group-by (#29217)
• Elide memset on pread for FileByteSource (#29162)
• Add prefetching to OOC (#29081)
• Push more predicates past caches (#29182)
• Prevent cache trashing in row encoding for fixed width types (#29175)
• Improve group-by cardinality estimates (#29174)
• Split join clusters across filters and bound key domains by value range (#29166)
• Make file read concurrency semaphore global (#29163)
• Skip redundant output validity construction in all-valid arr.dot (#29115)
• Default posix_fadv to Normal (#29157)
• Avoid a full fsync per file when writing Parquet (#29108)
• Replace mmap with file I/O for parquet scan (#29091)
• Use adaptive concurrency for head requests (#29131)
• Improve join-reordering (#29092)

✨ Enhancements

• Expose the registered source as scan_fn.io_source (#28897)
• Support selectors in join keys (#29233)
• Honor Iceberg sort orders in native sinks (#29318)
• Add APPROX_QUANTILE to the SQL frontend (#29288)
• Add support for approx_quantile in the streaming engine (#29237)
• Read Parquet ENUM type as pl.String (#29331)
• Expose more scan_iceberg/delta-related attributes in the visitor for cudf_polars (#29297)
• More map operations (#29296)
• Binning functions (#28888)
• Support collect and collect_batches using RemoteEngine (#28914)
• Support GROUP BY GROUPING SETS, ROLLUP, CUBE and GROUPING() (#29278)
• Fix tpch SQL issues (#29269)
• Fuse filters in (inner) join operation (#29218)
• Add in-memory support for approximate quantile (#29206)
• Disable casts from String to Time (#29215)
• Add organization to Config.enable_monitoring (#29221)
• Add a workspace argument to Config.enable_monitoring (#29196)
• Remove legacy POLARS_STREAMING_CHUNK_SIZE (#29046)
• Add LazyFrame resolver API (#29003)
• Support incremental append Iceberg scans (#28820)
• Support date/integer arithmetic and decimal non-equi join comparisons (#29156)
• Update documentation analytics implementation (#29127)
• Introduce the Map dtype (#28984)
• Update behavior of DataFrame/LazyFrame functions on 0-width input (#29099)

🐞 Bug fixes

• Fix OOM on TPCH SQL and fix fuzzing errors (#29389)
• Size row-index table statistics from the statistics frame (#29381)
• Preserve categories object in pyo3-polars (#29385)
• Strip the leading slash from Windows Delta table roots (#29386)
• Isolate expanded Python dataset scans (#29378)
• Make Iceberg bucket sort keys serializable (#29376)
• Apply "schema_overrides" in read_database for Arrow-based drivers (#29273)
• Remove duplicated word in rate-limit comment (#29372)
• Check whether Datetime is monotonically increasing dynamically (#29293)
• Keep input order of unmatched build rows in ordered streaming equi join (#29371)
• Unaliased constants in SQL SELECT with GROUP BY (#29367)
• Consistent Date and Decimal means between the streaming and in-memory engines (#29359)
• Respect string statistics for enum columns during parquet scanning (#29366)
• Support selectors properly in DataFrame n_unique (#29360)
• Decimal Parquet statistics for decimal/i128/f16 (#29350)
• Early check for converting integer map keys (#29348)
• Lowering for input-independent filter (#29340)
• Correlation of constant column returning non-NaN for larger inputs (#29319)
• Incorrect height in multi-input GroupBy (#29332)
• Fix panic in scan_iceberg for snapshot_id before a schema change (#28895)
• Resolve arithmetic Struct supertypes per-field (#29261)
• Fix comparison expression method comment (#29257)
• Don't panic on an empty or null quantile expression input (#29240)
• Detect list-valued quantile literals in approx_quantile auto (#29236)
• Various map issues (#29147)
• Rolling quantile should respect window trimming (#29200)
• Ensure ignored columns are excluded from dtype Wildcard selector (#29220)
• Include the library path in the plugin dlopen error message (#29213)
• Respect Parquet schema when reading Int96 timestamps (#29208)
• Wrap IoError in PathIoError such that callers can extract untruncated path (#29128)
• Slice on Series with negative index beyond length (#29191)
• Fix OOB access due to empty chunks (#29177)
• Compare struct AnyValues through their field dtypes (#29185)
• Improve group-by cardinality estimates (#29174)
• Preserve scalar arrays in arr.dot dispatch (#29000)
• Empty array collect missing dtype in join (#29095)
• Apply the OVER clause to multi-argument aggregates (#29160)
• Respect NULLS FIRST/LAST in a window's ORDER BY (#29159)
• Resolve relation aliases declared inside a parenthesized JOIN (#29158)
• Fix a DataFrame construction race (and generator data loss) (#29132)
• Properly compute null count in parquet statistics for list of categorical/enum (#29140)
• Address read_database Oracle regression, following introduction of Arrow fast-path (#29142)
• Synchronize PR label matchers (#29144)
• Unblock Python thread during collect_all_async (#29097)
• Match engine output dtype in planner for Decimal true division with integers (#29122)
• Error instead of segfault for exp/log1p on non-numeric dtypes (#29112)
• Prevent panic in offset_by when target date is out-of-range (#29059)
• Incorrect broadcast height min/max_horizontal for scalar inputs (#29083)
• Proper seq id relabeling before distributing (#29060)

📖 Documentation

• Clarify that ambiguous parameter refers to DST transitions (#28873)
• Document gzip/zstd compression for read_csv and scan_csv (#29387)
• Add LazyFrameResolver to reference guide (#29379)
• Add user-guide for new enable_monitoring feature (#29358)
• Improve join_where engine tag (#29234)
• Add 2 to the version switcher dropdown (#29194)
• Show more information in engine tags (#29154)
• Engine tags for is_empty/ has_nulls (#29152)
• More engine tags (#29148)
• Fix misleading reinterpret() docstring (#29075)

📦 Build system

• Add a fast-release profile without LTO (#29168)

🛠️ Other improvements

• Add dymanicpredicates to preferred build sides (#29335)
• Deprecate cut/qcut (#29329)
• Mark test as slow (#29336)
• Bump maturin (#29282)
• Bump object_store crate to 0.14.2 (#29317)
• Update rustls dependency to version 0.23.45 (#29303)
• Bump build deps used in ARM64 Windows release pipeline (#29280)
• Fix the stalling test_fused_many_morsels_and_skew test (#29290)
• Use SpillFrames in DataFrameSearchBuffers (#29207)
• More obvious MapChunked storage handling (#29248)
• Centralize hoisted aggregate bookkeeping and clarify grouping predicates (#29287)
• Update analytics endpoint config (#29222)
• Add a blanket lf.collect().schema == lf.collect_schema() check (#29224)
• Add expand_paths parameter to toggle path expansion (#29210)
• Remove auto-streaming jobs (#29214)
• Bump object_store for new DnsResolver trait (#29145)
• The in-memory engine is no longer the default in the docstrings (#29137)
• Test list-of-structs in map_elements (#24925)
• Update existing comment instead of new comment for file size on PR (#29134)
• Output stdout/stderr of crashed pytest-xdist workers (#29094)
• Remove batched_reader.py (#29098)
• Update test for latest pyiceberg (#29090)

Thank you to all our contributors for making this release possible!
@0guban0v, @ATL2001, @Aidavdw, @AlessandroKuz, @BitWeaverDev, @DeflateAwning, @EndPositive, @Kevin-Patyk, @MarcoGorelli, @SatvikMishra08, @aarushkandukoori, @abokhalill, @alexander-beedie, @atharva7905k, @ayushh0110, @borchero, @carnarez, @dancsi, @dsprenkels, @freundTech, @fsimkovic, @jonasdedden, @kafka1991, @kdn36, @lun3x, @madsbk, @matthewbayer, @mikhail5555, @mroeschke, @nameexhaustion, @orlp, @r-brink and @ritchie46