pola-rs/polars py-1.0.0

Python Polars 1.0.0

on GitHub

This is the first major release for Python Polars. Please check out the upgrade guide for help navigating the breaking changes when upgrading to this version.

💥 Breaking changes

• Change default engine for read_excel to "calamine" (#17263)
• Implement binary serialization of LazyFrame/DataFrame/Expr and set it as the default format (#17223)
• Streamline optional dependency definitions in pyproject.toml (#17168)
• Update read/scan_parquet to disable Hive partitioning by default for file inputs (#17106)
• Split replace functionality into two separate methods (#16921)
• Default to writing binview data to IPC, mark compression argument as keyword-only (#17084)
• Remove re-export of type aliases (#17032)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Properly apply strict parameter in Series constructor (#16939)
• Remove supertype definition of List and non-List types (#16918)
• Consistently convert to given time zone in Series constructor (#16828)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Update set_sorted to only accept a single column (#16800)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Update some error types to more appropriate variants (#15030)
• Scheduled removal of deprecated functionality (#16715)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)

⚠️ Deprecations

• Deprecate LazyFrame.fetch (#17278)
• Deprecate size parameter in parametric testing strategies in favor of min_size/max_size (#17128)
• Split replace functionality into two separate methods (#16921)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Remove re-export of exceptions at top-level (#17059)
• Deprecate dt.mean/dt.median in favor of mean/median (#16888)
• Deprecate LazyFrame.with_context in favor of horizontal concatenation (#16860)
• Rename parameter descending to reverse in top_k methods (#16817)
• Rename str.concat to str.join and update default delimiter (#16790)
• Deprecate arctan2d in favor of arctan2(...).degrees() (#16786)

🚀 Performance improvements

• Rechunk before group_by `iteration (#17302)
• Improve unique performance by adding RangedUniqueKernel for primitive arrays (#17166)
• Improve unique performance by creating UniqueKernel and improve bool implementation (#17160)
• Default to writing binview data to IPC, mark compression argument as keyword-only (#17084)
• Parallelize arrow conversion if binview -> large_bin (#17083)
• Garbage collect buffers in if-then-else view kernel (#16993)
• Desugar AND filter into multiple nodes (#16992)
• Optimize generic arg_sort of row-encoding (#16894)
• Improve rle_id iteration performance and set sorted flags (#16893)
• Optimize sort for String and Binary types (#16871)
• Use split_at in split (#16865)
• Use split_at instead of double slice in chunk splits. (#16856)
• Don't rechunk in align_ if arrays are aligned (#16850)
• Don't create small chunks in parallel collect. (#16845)
• Add dedicated no-null branch in arg_sort (#16808)
• Speed up dt.offset_by 2x for constant durations (#16728)
• Toggle coalesce in join if non-coalesced key isn't projected (#16677)
• Make dt.truncate 1.5x faster when every is just a single duration (and not an expression) (#16666)
• Always prune unused columns in semi/anti join (#16665)

✨ Enhancements

• Add SQL support for NATURAL joins and the COLUMNS function (#17295)
• Add str.extract_many expression (#17304)
• Change default engine for read_excel to "calamine" (#17263)
• Deprecate LazyFrame.fetch (#17278)
• Support '%' in pathnames for async scan (#17271)
• Support SQL Struct/JSON field access operators (#17226)
• Exclude directories from glob expansion result (#17174)
• Support SQL ORDER BY ALL syntax (#17212)
• Support PostgreSQL ^@ ("starts with"), and ~~,~~*,!~~,!~~* ("like", "ilike") string-matching operators (#17251)
• Support SQL SELECT * ILIKE wildcard syntax (#17169)
• Support SQL temporal functions STRFTIME and STRPTIME, and typed literal syntax (#17245)
• Support date/datetime for hive parts (#17256)
• Implement binary serialization of LazyFrame/DataFrame/Expr and set it as the default format (#17223)
• Allow no-op round/ceil/floor on integer types (#17241)
• Support loading from datasets where the hive columns are also stored in the file (#17203)
• Implement serde for Null columns (#17218)
• Support Decimal types in write_csv/write_json (#14209)
• Add optional "default" to get_column DataFrame method (#17176)
• Improve SQL support for array indexing, increase test coverage (#16972)
• Support reading byte stream split encoded floats and doubles in parquet (#17099)
• Add float_scientific option to write_csv/sink_csv (#17111)
• Support Struct field selection in the SQL engine, RENAME and REPLACE select wildcard options (#17109)
• Update DataFrame.pivot to allow index=None when values is set (#17126)
• Update read/scan_parquet to disable Hive partitioning by default for file inputs (#17106)
• Improve ipython autocomplete for LazyFrame and DataFrame (#17091)
• Split replace functionality into two separate methods (#16921)
• Improve schema inference for hive partitions (#17079)
• Rename DataFrame.melt to unpivot and make parameters consistent with pivot (#17095)
• Print row index in explain and show_graph (#17074)
• Support top-level pl.col autocompletion for iPython (#17080)
• Remove re-export of exceptions at top-level (#17059)
• Implement predicate and projection pushdown for read_ndjson (#17068)
• Allow (non-)coalescing in join_asof (#17066)
• Turn of coalescing and fix mutation of join on expressions (#17061)
• Expand NDJson glob into one SCAN (#17063)
• Do not parse hive partitions from user provided base directory path (#17055)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Implement general array equality checks (#17043)
• Add strict parameter to DataFrame/LazyFrame.drop and fix behavior to default to True (#17044)
• Rename ModuleUpgradeRequired and PolarsPanicError error, remove InvalidAssert error (#17033)
• Add rechunk parameter to read_delta (#16991)
• allow experimental metadata use on release (#17005)
• Add simple version of json_normalize (#17015)
• Change data orientation inference logic for DataFrame construction and warn when row orientation is inferred (#16976)
• Desugar AND filter into multiple nodes (#16992)
• Handle textio even if not correct (#16971)
• Properly apply strict parameter in Series constructor (#16939)
• Add SQL support for INTERSECT and EXCEPT ops (#16960)
• Add PerformanceWarning to LazyFrame properties (#16964)
• Add collect_schema method to LazyFrame and DataFrame (#16929)
• Allow setting file cache TTL on a per-file basis (#16891)
• Support Decimal inputs for lit (#16950)
• Implement multiply and division for lhs duration (#16948)
• Raise on invalid temporal arithmetic (#16934)
• Always end with a in-memory sink on collect (#16928)
• Add DataFrame.style namespace (#16809)
• Add Schema class (#16873)
• Normalize value_counts (#16917)
• Implement equality for more Array types (#16902)
• Set up some of the infrastructure for new streaming engine (#16900)
• Cache downloaded cloud IPC files (#16892)
• Consistently convert to given time zone in Series constructor (#16828)
• Improve read_csv SQL table reading function defaults (better handle dates) (#16866)
• Support SQL VALUES clause and inline renaming of columns in CTE & derived table definitions (#16851)
• Support Python Enum values in lit (#16858)
• Convert to given time zone in .str.to_datetime when values are offset-aware (#16742)
• Update reshape to return Array types instead of List types (#16825)
• Default to raising on out-of-bounds indices in all get/gather operations (#16841)
• Support SQL "SELECT" with no tables, optimise registration of globals (#16836)
• Native selector XOR set operation, guarantee consistent selector column-order (#16833)
• Extend recognised EXTRACT and DATE_PART SQL part abbreviations (#16767)
• Improve error message when raising integers to negative integers, improve docs (#16827)
• Return datetime for mean/median of Date colum (#16795)
• Update set_sorted to only accept a single column (#16800)
• Expose overflowing cast (#16805)
• Update group_by iteration and partition_by to always return tuple keys (#16793)
• Support array arithmetic for equally sized shapes (#16791)
• Expedited removal of certain deprecated functionality (2) (#16779)
• Removal of read_database_uri passthrough from read_database (#16783)
• Remove pyxlsb engine from read_excel (#16784)
• Add check_order parameter to assert_series_equal (#16778)
• Enforce deprecation of keyword arguments as positional (#16755)
• Support cloud storage in scan_csv (#16674)
• Streamline SQL INTERVAL handling and improve related error messages, update sqlparser-rs lib (#16744)
• Support use of ordinal values in SQL ORDER BY clause (#16745)
• Support executing polars SQL against pandas and pyarrow objects (#16746)
• Remove deprecated parameters in Series.cut/qcut and update struct field names (#16741)
• Expedited removal of certain deprecated functionality (#16754)
• Remove deprecated functionality from rolling methods (#16750)
• Update date_range to no longer produce datetime ranges (#16734)
• Mark min_periods as keyword-only for rolling methods (#16738)
• Remove deprecated top_k parameters nulls_last, maintain_order, and multithreaded (#16599)
• Support order-by in window functions (#16743)
• Add SQL support for NULLS FIRST/LAST ordering (#16711)
• Update some error types to more appropriate variants (#15030)
• Initial SQL support for INTERVAL strings (#16732)
• Scheduled removal of deprecated functionality (2) (#16724)
• Scheduled removal of deprecated functionality (#16715)
• Enforce deprecation of offset arg in truncate and round (#16655)
• Change default offset in group_by_dynamic from 'negative every' to 'zero' (#16658)
• Constrain access to globals from DataFrame.sql in favor of top-level pl.sql (#16598)
• Read 2D NumPy arrays as Array type instead of List (#16710)
• Update clip to no longer propagate nulls in the given bounds (#14413)
• Change str.to_datetime to default to microsecond precision for format specifiers "%f" and "%.f" (#13597)
• Update resulting column names in pivot when pivoting by multiple values (#16439)
• Preserve nulls in ewm_mean, ewm_std, and ewm_var (#15503)
• Restrict casting for temporal data types (#14142)
• Add many more auto-inferable datetime formats for str.to_datetime (#16634)
• Support Decimal types by default when converting from Arrow (#15324)
• Remove serde functionality from pl.read_json and DataFrame.write_json (#16550)
• Update function signature of nth to allow positional input of indices, remove columns parameter (#16510)
• Rename struct fields of rle output to len/value and update data type of len field (#15249)
• Remove class variables from some DataTypes (#16524)
• Add check_names parameter to Series.equals and default to False (#16610)
• Dedicated SQLInterface and SQLSyntax errors (#16635)
• Add DIV function support to the SQL interface (#16678)
• Support non-coalescing streaming left join (#16672)
• Allow wildcard and exclude before struct expansions (#16671)

🐞 Bug fixes

• Raise on invalid shape dataframe arithmetic (#17322)
• Fix panic in window case (#17320)
• Raise errors instead of panicking when sink_csv fails (#17313)
• Raise if join keys are passed to cross join (#17305)
• Ensure we don't close extant adbc connections in write_database (#17298)
• Don't null on oob in list.get for column index (#17276)
• Fix issue where sliced PyArrow record batches were not handled correctly (#17058)
• Don't oob on nulls in list.get (#17262)
• Fix list getter with nulls (#17261)
• Respect nulls_last parameter in aggregate sort_by (#17249)
• Fix literal slice in group by (#17242)
• Fix DataFrame.top_k not handling nulls correctly (#17239)
• Update implementation of Enum support in lit to address spurious test failure (#17187)
• Use explicit turbofish to help rustc (#17159)
• Raise on invalid set dtypes (#17157)
• Fix corrupted reads for hive parts from cloud and projection pushdown failure on hive parts (#17152)
• Set intersection supertype (#17154)
• ChainedWhen should not inherit Expr (#17142)
• Fix decompress_impl for csv with n_rows set (#17118)
• Fix incorrect window std for chunked series (#17110)
• Fix panic when using fold in certain situations (#17114)
• Fix melt panic (#17088)
• Fix expression autocomplete in IPython (#17072)
• Exclude index from expansion in rolling/group_by_dynamic (#17086)
• Update some Series dunder method type signatures (#17053)
• Fix oob of join with literals and empty table (#17047)
• Don't silently accept multi-table FROM clauses (implicit JOIN syntax) (#17028)
• Don't split up ANDed filters that are group-aware (#17031)
• Harden "async" check for users with out-of-date sqlalchemy libraries (#17029)
• Error when sort_by of unequal length (#17026)
• Properly catch not found explode cols (#17020)
• Correctly convert data frames to NumPy for C index order (#17000)
• Raise on invalid arithmetic shapes (#16986)
• Don't pushdown predicates in cross join if the refer to both tables (#16983)
• Fix projection pushdown with literal joins (#16981)
• Fix edge case in DataFrame constructor data orientation inference (#16975)
• Raise on list of objects (#16959)
• Handle strictness for Decimal Series construction (#15309)
• Don't panic in object to anyvalue (#16957)
• Properly set FAST_EXPLODE_LIST metadata (#16951)
• Raise informative error when writing object to file (#16954)
• Remove supertype definition of List and non-List types (#16918)
• Remove unwrap in extend() (#16890)
• Fix should_rechunk check (#16852)
• Ensure read_excel and read_ods return identical frames across all engines when given empty spreadsheet tables (#16802)
• Consistent behaviour when "infer_schema_length=0" for read_excel (#16840)
• Standardised additional SQL interface errors (#16829)
• Ensure that splitted ChunkedArray also flattens chunks (#16837)
• Reduce needless panics in comparisons (#16831)
• Reset if next caller clones inner series (#16812)
• Raise on non-positive json schema inference (#16770)
• Rewrite implementation of top_k/bottom_k and fix a variety of bugs (#16804)
• Fix comparison of UInt64 with zero (#16799)
• Fix incorrect parquet statistics written for UInt64 values > Int64::MAX (#16766)
• Fix boolean distinct (#16765)
• DATE_PART SQL syntax/parsing, improve some error messages (#16761)
• Include pl. qualifier for inner dtypes in to_init_repr (#16235)
• Column selection wasn't applied when reading CSV with no rows (#16739)
• Panic on empty df / null List(Categorical) (#16730)
• Only flush if operator can flush in streaming outer join (#16723)
• Raise unsupported cat array (#16717)
• Assert SQLInterfaceError is raised (#16713)
• Restrict casting for temporal data types (#14142)
• Handle nested categoricals in assert_series_equal when categorical_as_str=True (#16700)
• Improve read_database check for SQLAlchemy async Session objects (#16680)
• Reduce scope of multi-threaded numpy conversion (#16686)
• Full null on dyn int (#16679)
• Fix filter shape on empty null (#16670)

📖 Documentation

• Update version switcher for 1.0.0 final release (#16848)
• Finish upgrade guide for 1.0.0 (#17257)
• Minor layout/terminology improvement for selector set ops (#17299)
• Mark hypothesis testing functionality as unstable (#17258)
• Add SQL docs for the CAST and TRY_CAST functions (#17214)
• Mark plot namespace as unstable (#17205)
• Bump docs dependencies (#17199)
• More accurate and helpful docs for user defined functions (#15194)
• Add doc examples to concat_list (#17127)
• Add "coming from pandas" note to DataFrame.unique docstring (#17119)
• Fix some warnings during doc build (#17077)
• Properly expose InProcessQuery in docs, mark as unstable (#17097)
• Add upgrade guide for Python Polars 1.0.0 (#16914)
• Lots of additions to the SQL reference docs (#16990)
• Minor doctest fixes (#17002)
• Include a doc entry for every exception type (#17001)
• Fixup bullet points in write_parquet docstring (#16909)
• Update version switcher for 1.0.0 prereleases (#16847)
• Update link from Python API reference to user guide (#16849)
• Update docstring/test/etc usage of select and with_columns to idiomatic form (#16801)
• Update versioning docs for 1.0.0 (#16757)
• Add docstring example for DataFrame.limit (#16753)
• Fix incorrect stated value of include_nulls in DataFrame.update docstring (#16701)
• Update deprecation docs in the user guide (#14315)
• Add example for index count in DataFrame.rolling (#16600)
• Improve docstring of Expr/Series.map_elements (#16079)
• Add missing polars.sql docs entry and small docstring update (#16656)

📦 Build system

• Update Cargo.lock (#17284)
• Streamline optional dependency definitions in pyproject.toml (#17168)
• Update rustc 2024-06-23 (#17135)
• Do not set environment variable on import (#17101)
• Fix config flag for Tracemalloc (#17098)
• Pin optional NumPy dependency to < 2.0.0 for now (#17060)

🛠️ Other improvements

• Fix typo in join validation error message (#17296)
• Fix linting issue in docs (#17292)
• Use typed iter in list.get (#17286)
• Rename type_aliases module to _typing (#17282)
• add ability to have pipeline blockers in new streaming engine (#17247)
• Support date/datetime for hive parts (#17256)
• Refactor serde tests, add hypothesis tests (#17216)
• Refactor parsing of data type inputs to Polars data types (#17164)
• Skip all moto AWS tests for now (#17178)
• Add missing spaces in cargo.toml (#17145)
• Minor test refactor for concat_list (#17120)
• Remove re-export of data type groups (#17073)
• Add pivot test #17081 (#17090)
• Minor cleanup to better define boundaries of public API (#17051)
• Support directory paths in scans for Parquet, IPC and CSV (#17017)
• Remove re-export of type aliases (#17032)
• Remove file cache test (#17038)
• Update exception imports in test suite (#17035)
• Point polars-stream to crates/ again (#17024)
• Fix failing file cache test in CI (#17014)
• Add some parametric tests for sort functionality (#17008)
• Pin NumPy to <2.0 for now (#16999)
• Use proper join type in test (#16994)
• Fix file cache verbose logging leakage during pytest (#16984)
• Skip another intermitently failing AWS test (#16980)
• Update test suite to explicitly use orient="row" in DataFrame constructor when applicable (#16977)
• Remove redundant projection attribute in IR::DataFrameScan (#16952)
• Factor out some apply calls in duration namespace (#16941)
• Skip intermittently failing AWS test (#16908)
• Refactor expression parsing utils (#16906)
• Set up some of the infrastructure for new streaming engine (#16900)
• Refactor parts of IR. (#16899)
• Add fundamentals for new async-based streaming execution engine (#16884)
• Move around some existing tests (#16877)
• Remove inner Arc from FileCacheEntry (#16870)
• Do not update stable API reference on prerelease (#16846)
• Update links to API references (#16843)
• Prepare update of API reference URLs (#16816)
• Rename allow_overflow to wrap_numerical (#16807)
• Set infer_schema_length as keyword-only argument in str.json_decode (#16835)
• Don't enter streaming engine for groupby-> agg mean/median … (#16810)
• Improve safety of amortized_iter (#16820)
• Remove needless inner type clone (#16718)
• Fix incorrect debug assertion in ChunkedArray::from_chunks_and_dtype (#16697)
• Update version resolver for 1.0.0 release (#16705)
• Avoid AWS pinning to outdated crc32c version (#16681)

Thank you to all our contributors for making this release possible!
@IvanIsCoding, @JamesCE2001, @JulianCologne, @KDruzhkin, @Kylea650, @MarcoGorelli, @Mottl, @Object905, @SeanTater, @adamreeve, @alexander-beedie, @bertiewooster, @borchero, @c-peters, @coastalwhite, @datapythonista, @datenzauberai, @dependabot, @dependabot[bot], @eitsupi, @flisky, @henryharbeck, @itamarst, @jqnatividad, @lukeshingles, @machow, @marenwestermann, @mcrumiller, @montanarograziano, @nameexhaustion, @orlp, @p3i0t, @ritchie46, @sherlockbeard, @stinodego, @tkellogg, @universalmind303 and @wence-