pola-rs/polars py-2.1.0

Python Polars 2.1.0

on GitHub

💥 Breaking changes

• Disallow mixed string <=> numeric arithmetic in eager (#30071)

🚀 Performance improvements

• Limit the memory of buckets sorted ahead in the sort flush (#30297)
• Make file readers wait for spills while over the memory budget (#30296)
• Count unspilled frames in the memory estimate right away (#30295)
• Simplify more boolean expressions and drop redundant null checks (#30293)
• Wait for spills while over the out-of-core memory budget (#30294)
• Lower the default out-of-core prefetch fraction to 0.5 (#30292)
• Aggregate unique and n_unique in the streaming group-by itself (#30281)
• Return freed memory to the OS right away while out-of-core data is spilled (#30280)
• Gather List(Array) and Array columns without per-row slices (#30278)
• Evaluate the streaming window node out of core (#30282)
• Avoid deep copies when building a DataFrame from flat dataclasses (#30275)
• Sort the window partitions by each order key instead of their row encoding (#30277)
• Scan the haystack for a single needle in is_in (#30276)
• Prefer fixed-width columns when only the row count is needed (#30274)
• Native gather for List arrays (#30019)
• Evaluate windows in filter predicates in the streaming window node (#30272)
• Don't busy-wait in RAYON.block_on (#30259)
• Encode nested Parquet levels directly from the level buffers (#30251)
• Reduce memory of CSV scans on wide rows (#30165)
• Avoid quadratic JSON serialization of nested lists and maps (#30203)
• Skip the gather when exploding lists without nulls (#30234)
• Speed up writing nested Parquet columns (#30245)
• Don't keep whole parquet string dictionaries alive for a few selected rows (#30248)
• Set default spill compression level to 1 (#30237)
• Reuse zstd contexts in compression and avoid memcpys (#30233)
• Remove repeated and scalar ordering keys before the sort rewrites (#30231)
• Remove repeated keys from by's (#30095)
• Keep only one guessed unique key per join order leaf (#30191)
• Don't let extra join keys shrink an estimate once one key is unique (#30190)
• Reduce binary size of asof join with by keys (#30192)
• Reduce binary size of SQL visitors and parallel group-by collect (#30171)
• Push as-of join key filters to the right input (#30123)
• Drop null as-of right keys with a parallel filter before the join node (#30125)
• Keep as-of nearest join state small (#30126)
• Explode lists in group_by aggregations without a per-group apply (#30020)
• Streaming OOC group-by with shuffles (#29911)
• Drop scalar ordering keys in the optimizer (#30063)
• Merge union inputs that scan files of the same kind into one multi-file scan (#30056)
• Use the streaming scalar window for std, var and combined reductions in over (#30066)
• Limit the bytes of local CSV and NDJSON files read at once in multi-file scans (#30055)
• Check the matched flag of join keys in the table entry before the bitset (#30067)
• Track matched build rows of all join tables in a bitset (#30050)
• Add dedicated list gather kernels (#30049)
• Emit unmatched build rows of single-key joins in row order (#30037)
• Use TopK for sorts sliced at a nonzero offset (#29989)
• Add constant folding for more datetime ops (#30016)
• Rewrite the sum of a column plus a literal to a sum and a count of the column (#30011)
• Push predicates through order-keeping columns (#30007)
• Fix dead early exit in coalesce (#30013)
• Count distinct non-null values in SQL with one group-by in the streaming engine (#30009)
• Skip parquet row groups with the running top-k threshold (#30006)
• Keep decompressed plain string pages as the string data buffer (#29984)
• Fix some in-memory filter regressions (#29988)
• Decompress snappy and zstd parquet pages without zeroing the buffer (#29980)
• Speed up str.len_chars (#29972)
• Speed up str.replace for patterns that end in .*$ (#29971)
• Use the Parquet string prefilter for literal str.contains and SQL LIKE (#29953)
• Keep single-chunk output when filtering wide frames in parallel (#29847)
• Add dedicated list slice kernels (#29938)
• Principled execution order of accumulating streaming nodes (#29933)
• Huge speedup creating wide Struct data (#29813)
• Speed up arg_min and arg_max (especially on data with nulls) (#29849)
• Huge reduction in from_rows and from_dicts memory usage during nested value conversion (#29739)
• Safely push down nested struct field filters to Iceberg (#29665)
• Compare Parquet dictionaries of Categorical and Enum columns as strings for == and is_in (#29885)
• Filter Categorical and Enum Parquet columns while decoding (#29882)
• Rewrite sort().first() to min() and single-key sort_by().first() to min_by() (#29860)
• Decode fixed-length binary Parquet decimals directly into i128 (#29866)
• Look up each dictionary value once when importing Arrow dictionaries as Categorical (#29865)
• Reduce requests and raise concurrency for cloud CSV scans (#29863)
• Reduce requests and raise concurrency for cloud NDJSON scans (#29857)
• Track sortedness through joins (#29852)
• Keep sortedness through widening numeric casts (#29851)
• Rewrite sort_by().first() to min_by and limit sorts read with head() (#29831)
• Cache parsed Iceberg metadata for metadata-path scans (#29791)
• Drop memory when Done in streaming asof_join/is_first_distinct (#29783)
• Drop scalar sort_by keys in the optimizer (#29812)
• Parallel CSV chunk analysis and pread default (#29818)
• Hash inline binview values directly from their view (#29820)
• Use iceberg statistics (#29798)
• Reuse Iceberg catalog and cache REST catalog scans (#29790)
• Preserve sortedness of computed join keys (#29782)
• TopK was holding onto too much data (#29784)
• Write CSV temporal values straight into the buffer (#29778)
• Parse strftime format once and reuse buffer in regex replace (#29777)
• Reduce needless allocations in various kernels (#29775)
• Cache http object store (#29757)
• Fetch each record batch group in its own task (#29755)

✨ Enhancements

• Support SQL CREATE TABLE with column definitions in attached Iceberg catalogs (#30260)
• Support SQL CREATE [OR REPLACE] TEMP TABLE (#30256)
• Support recursive SQL CTEs (#30188)
• Writing encrypted Parquet files (#29905)
• Support SQL DROP TABLE [PURGE] on tables of attached catalogs (#30247)
• Add conversion to/from masked numpy arrays (#30232)
• Add overwrite_filter and dynamic partition overwrite to sink_iceberg (#29909)
• Support SQL JOIN ... ON subqueries that read only one input (#30206)
• Support full joins in join_where (#30169)
• Expose fused and nested loop join predicates in the IR visitor (#30176)
• Support LIMIT 1 in correlated SQL scalar subqueries (#30168)
• Support SQL WITH RECURSIVE when no CTE reads itself (#30166)
• Support correlated IN and EXISTS that read both inputs of a SQL join in ON (#30159)
• Support CURRENT_DATE, CURRENT_TIME and CURRENT_TIMESTAMP in SQL (#30144)
• Allow reinterpret between Int128 and Decimal (#30133)
• Add Expr.truncdiv for division rounding toward zero (#30138)
• Support uncorrelated subqueries in SQL JOIN ... ON (#30141)
• Support correlated subqueries in SQL JOIN ... ON and correlated scalar subqueries without an aggregate (#30131)
• Support sinking to Iceberg table with bucket partitioned field (#30091)
• Add create_namespace_if_not_exists to sink_iceberg (#30087)
• Attach Iceberg catalogs to SQLContext via register_catalog (#30113)
• Support OOC kill fraction (#30104)
• Support qualified table names in SQL (#29963)
• Reverse node for streaming engine (#29259)
• Add pl.delete_iceberg (#30032)
• Support named options in SQL read_ipc table function (#30054)
• Support named options in SQL read_parquet table function (#29998)
• Add experimental geoparquet read/write (#29987)
• Accept a list of paths in SQL read_* table functions (#30047)
• Add SQL read_ndjson table function with named options (#29954)
• Support STRUCT and MAP type names in SQL (#29951)
• Support unstable column/schema options in SQL read_csv (#29975)
• Add sampling to KLL approx-quantile sketch (#29424)
• Rust resolver for scan_iceberg (#29753)
• Add REGEXP_REPLACE and DATE_TRUNC to SQL (#29967)
• Add generate_series and range to SQL (#29912)
• Add create_if_not_exists to sink_iceberg (#29908)
• Support named options in SQL read_csv table function (#29917)
• Add query-level IO active time and thread count metrics (#29792)
• Support time-zone-aware timestamps in SQL (#29899)
• Add support for reading encrypted Parquet files (#29606)
• Add SQL ASOF join (#29834)

🐞 Bug fixes

• Compare Array(Struct) rows field by field (#30290)
• Don't revive a dropped out-of-core spill context through a weak reference (#30291)
• Do not read the values stored under null Array rows (#30286)
• Return freed memory to the OS right away while out-of-core data is spilled (#30280)
• Propagate nulls through all nesting levels before row encoding (#30269)
• Add a method to merge finalized approx_quantile sketches deterministically (#30263)
• Use the supertype of both inputs for the cov/corr output dtype (#30262)
• Keep buffered frames spillable while a streaming sort partitions (#30268)
• Respect memory limits of nested cgroups (#30266)
• Correct schema for integer literal divided by Float16/Float32 (#30158)
• Correct str.replace for nested repetitions before .*$ (#30258)
• Allow write_iceberg into Iceberg tables with required columns (#30253)
• Correct operator overloading for reflect ops (#29946)
• Raise on unsupported Avro codecs in read_avro (#30240)
• Support several inputs to anonymous aggregations in the in-memory engine (#30152)
• Stop SQL FROM item aliases from acting as tables (#30207)
• Correct unsigned parquet stats merging (#30199)
• Read Parquet schema in lockstep with arrow metadata hints (#30202)
• Raise on Delta tables with column mapping (#30204)
• Correct comparison with numpy array (#30241)
• Don't keep whole parquet string dictionaries alive for a few selected rows (#30248)
• Raise on SQL CREATE TABLE of a name in the default catalog (#30235)
• Widen truncated timestamp max statistics in scan_delta file skipping (#30194)
• Fix wrong JSON output for Struct and Array inside list (#30200)
• Correct schema for Float16/Float32 divided by Decimal (#30093)
• Pass every input of an anonymous aggregation to its streaming reduction (#30151)
• Support Array, nested List and Null in is_first_distinct and is_last_distinct (#30162)
• Refine register_catalog for IF EXISTS, re-registration and scopes (#30160)
• Support duckdb 2.0 in CI (#30161)
• Fix reading uncompressed buffers in compressed IPC files (#30084)
• Track SQL column names of every relation in a joined FROM item (#30153)
• Evaluate uncorrelated SQL EXISTS once and don't push len() into sliced unions (#30150)
• Treat a qualifier as the outer relation unless the subquery has a struct column of that name (#30149)
• Revert dropped offset check for bitmap into_mut (#30132)
• SQL EXISTS over aggregates and scope after semi/anti joins (#30146)
• Resolve SQL RIGHT SEMI and RIGHT ANTI join conditions against the right tables (#30135)
• Assign SQL join keys to the input holding the relation they name (#30137)
• Give correlated SQL aggregate expressions their value over no rows (#30130)
• Clear sorted flag on temporal expression inserting nulls (#30110)
• Ignore null right keys in asof join (#30122)
• Preserve order of sliced maintain-order group_by (#30096)
• Make ternary schema agree with type coercion (#29940)
• Raise error instead of panic when a sink is used as input of another operation (#29945)
• Reject String operands before float arms in true division type resolution (#30018)
• Ensure we don't allow scan+sink on same file (#30065)
• Correct rolling window when offset is not negation of period (#30060)
• Disallow mixed string <=> numeric arithmetic in eager (#30071)
• Preserve temporal sorted flag only when allowed (#30058)
• Decimal cast to int should truncate (#30069)
• Casting far future dates shouldn't silently wrap (#30068)
• Correctly parse timestamps with offsets in JSON (#30064)
• Use the correct input schema for join expressions in the expression optimizer (#30053)
• Sliced list writes in Avro (#30061)
• Raise on String operands in eager Series true division (#30021)
• Only infer CSV/NDJSON schema from complete rows on progressive download (#29961)
• Check SQL casts to STRUCT[]/MAP[] and specify cast failure (#30026)
• Allow empty groups in multi key sort_by (#29825)
• Fix reading Parquet files with empty row groups (#30001)
• Check the last 4 bytes of inlined strings for invalid UTF-8 in parquet (#29981)
• Fix shift with fill_value reporting the wrong lazy schema (#29978)
• Correct schema supertype for mixed bool <> unknown float ops (#29947)
• Sane output for full-key rows_by_key (#29943)
• More numerically stable ewm_std/var (#29942)
• Expand $ references in str.replace for patterns without punctuation (#29966)
• Fix wrong results and name clashes for computed SQL GROUP BY keys (#29965)
• Clear null propagated flag when setting outer validity on struct (#29949)
• Accept Mapping types not just dicts where advertised (#29952)
• Check moment agg dtype support during planning (#29955)
• Fix panic in when/then that masks a multi-chunk struct column (#29804)
• Don't extract scalar common subexpressions in group-by aggregations (#29941)
• Keep key dtypes when moving predicates across join keys (#29936)
• Use upcast key dtype in schema of coalescing full join (#29931)
• Return null instead of raising for a null scalar offset in list.slice (#29932)
• Raise error instead of panicking on out-of-range datetimes in to_list (#29918)
• Don't drop filter bounds whose literal has another dtype than the column (#29916)
• Handle CSPE in in-mem projection (#29914)
• Don't wait for ooc cleanup in forked child (#29900)
• Resolve WITH clauses inside SQL derived tables and nested queries (#29897)
• Fix incorrect filter expression rewrites (#29887)
• Lost wakeup in async executor when waking an already-scheduled task (#29824)
• Resolve SQL subquery columns in their own scope first (#29889)
• Don't push filters below a select of only scalars (#29837)
• Write valid Parquet statistics and column index for NaN and nested values (#29868)
• Clamp the mask in shift_and_fill for inputs shorter than n (#29840)
• Cast the inner values by their type in Array and List casts (#29864)
• Support empty NDJSON files read through file:// paths (#29862)
• Support SQL IN with a date against datetimes and with nested values of other types (#29861)
• Support a string literal on the left of SQL BETWEEN and minus with a temporal value (#29856)
• Track sortedness through joins (#29852)
• Apply SQL SELECT * options to the wildcard columns only (#29854)
• Fix row-estimation panic for aggregations over empty scan (#29850)
• Give Iceberg scans statistics before the cache and SQL subquery decisions (#29836)
• Infer Float16 from Parquet FLOAT16 logical type without Arrow schema (#29832)
• Support SQL HAVING without GROUP BY and check columns outside aggregates (#29833)
• Raise on oob gather indices (#29827)
• Call statx via raw syscall so wheels load on glibc < 2.28 (#29828)
• Broadcast scalar inputs in min_by/max_by and sort_by (#29808)
• Fix wrong results in SQL REPLACE and ARRAY_AGG, join_where on decimals and sorted group-by keys (#29823)
• Support a string literal on the left of a SQL comparison with a temporal column (#29814)
• Livelock in TaskParker on very low thread counts (#29819)
• Compare floats with integers and decimals in SQL IN (#29810)
• Fix SQL ANY/ALL comparisons with a subquery (#29802)
• Fix SQL FILTER clauses that contain a subquery (#29794)
• Don't push filters into a cache past a projection (#29793)
• Phase logic error in merge_sorted (#29781)
• Incorrect length error in streaming group-by item() (#29787)
• NegativeSlice could hold onto more data than necessary (#29779)
• Fix small bugs in SQL aggregates and in group contexts (#29754)
• Fix clip not broadcasting length-1 input against full-length bounds (#29666)
• Fix scan_lance CancelledError (#29774)
• Fix cross join filters on Decimals of different scales (#29776)
• Use correct bucket for CSV and NDJSON cloud scans (#29772)
• Re-entrance in PyO3 PolarsAllocator (#29763)

📖 Documentation

• List extra required deps for make pre-commit (#30145)
• Target the stub with loc.stub in the styling example (#29996)
• Migrate to Zensical (#29426)

📦 Build system

• Allow choosing the Python interpreter for the dev venv with PYTHON (#30142)
• Reduce binary size of SQL visitors and parallel group-by collect (#30171)
• Drop support for Python 3.10 and test on Python 3.15 (#29888)

🛠️ Other improvements

• Fix allocator-dependent failure in masked zero-copy to_numpy test (#30298)
• Simplify the out-of-core decay state and tests (#30283)
• Bump the clipboard-win dependency (#30264)
• Rename polars_cloud package (#30099)
• Fix Excel pyrefly lint (#30201)
• Reduce binary size of SQL visitors and parallel group-by collect (#30171)
• Sort unordered QUALIFY test query (#30167)
• Support duckdb 2.0 in CI (#30161)
• Mark large multi-column key streaming tests as slow (#30143)
• Bump webpki-roots and webpki-root-certs to 1.0.10 (RUSTSEC-2026-0383) (#30129)
• Don't make your private keys XKCD-221 compliant (#30115)
• Mark test as slow (#30086)
• Remove xdist_group "streaming" to speed up test suite by 50s (#30057)
• Replace the home crate with std::env::home_dir (#30039)
• Add polars-iceberg to Cargo.lock in Rust lint CI (#30033)
• Remove stale polars-io-ext entry from Cargo.lock (#30025)
• Move SQL read_csv option tests from Rust to Python (#29983)
• Run polars-stream unit tests in the Rust test workflow (#29703)
• Fail fast on dead cloud endpoints and re-enable Windows (#29621)
• Make ExprIR::is_order_insensitive public (#29994)
• Update rustls to 0.23.46 (#29944)
• A few additions to .gitignore (#29633)
• Fix flaky HTTP store cache tests (#29929)
• bump simd-json (#29838)
• Improve OOC out-of-disk error (#29761)

Thank you to all our contributors for making this release possible!
@Aidavdw, @EndPositive, @FalkWoldmann, @Hoeze, @JasonLi314, @Jesse-Bakker, @Kevin-Patyk, @TiansuYu, @adamreeve, @alexander-beedie, @borchero, @carnarez, @dancsi, @dominikandreasseitz, @dsprenkels, @fsimkovic, @jqnatividad, @jvdd, @kdn36, @lun3x, @mroeschke, @nameexhaustion, @orlp, @ritchie46, @tahakotil and @useredsa