explorer 0.9.0

v0.9.0

on Hex

Added

• Add initial support for SQL queries.

  The Explorer.DataFrame.sql/3 is a function that accepts a dataframe and a SQL query. The SQL is not validated by Explorer, so the queries will be backend dependent. Right now we have only Polars as the backend.

• Add support for remote series and dataframes.

  Automatically transfer data between nodes for remote series and dataframes and perform distributed garbage collection.

  The functions in Explorer.DataFrame and Explorer.Series will automatically move operations on remote dataframes to the nodes they belong to.
  The Explorer.Remote module provides additional conveniences for manual placement.

• Add FLAME integration, so we automatically track remote series and dataframes returned from FLAME calls when the :track_resources option is enabled.
  See FLAME for more.

• Add Explorer.DataFrame.transform/3 that applies an Elixir function to each row. This function is similar to Explorer.Series.transform/2, and as such, it's considered an expensive operation. So it's recommended only if there is no similar dataframe or series operation available.

• Improve performance of Explorer.Series.from_list/2 for most of the cases where the :dtype option is given. This is specially true for when the dtype is :binary.

Changed

• Stop inference of dtypes if the :dtype option is given by the user.
  The main goal of this change is to improve performance. We are now delegating the job of decoding the terms as the given :dtype to the backend.

• Explorer.Series.pow/2 no longer casts to float when the exponent is a signed integer. We are following the way Polars works now, which is to try to execute the operation or raise an exception in case the exponent is negative.

• Explorer.Series.pivot_wider/4 no longer includes the names_from column name in the new columns when values_from is a list of columns. This is more consistent with its behaviour when values_from is a single column.

• Explorer.Series.substring/3 no longer cycles to the end of the string if the negative offset surpasses the beginning of that string. In that case, an empty string is returned.

• The Explorer.Series.ewm_* functions no longer replace nil values with the value at the previous index. They now propogate nil values through to the result series.

• Saving a dataframe as a Parquet file to S3 services no longer works when streaming is enabled. This is temporary due to a bug in Polars. An exception should be raised instead.

Pull requests

• Update workflows to remove warnings by @philss in #925
• Add Artifact Attestations by @philss in #926
• Stop inference of type if dtype is given and manually decode terms where needed by @philss in #928
• Improve "from_list/2" decoding of terms for binary series by @philss in #929
• Initial SQL support by @billylanchantin in #918
• Remote series and dataframes and distributed GC by @josevalim in #932
• Add datetime literals to expressions by @billylanchantin in #935
• Fix DateTime -> ExDateTime lifetime issue by @elbow-jason in #909
• Improve performance of "concat_columns" by @philss in #938
• Add DF.transform by @billylanchantin in #912
• Implement message-on-gc NIFs in Rust by @filmor in #936
• Docs: replace bad link for nsjson by @zhzy0077 in #940
• Update Rust toolchain to nightly-2024-06-23 by @philss in #942
• Update MacOS version in the precompilation GH workflow by @philss in #943
• Improve the docs for from_query/4 by @cigrainger in #946
• Bump Polars to v0.41.3 by @lkarthee in #917
• Add FLAME integration by @josevalim in #945
• Add changes for v0.9 by @philss in #950
• Release v0.9 by @philss in #951

New Contributors

• @elbow-jason made their first contribution in #909
• @zhzy0077 made their first contribution in #940

Full Changelog: v0.8.3...v0.9.0

SHA256 of artifacts

aeed3719479b9bbe1e342af272927a75b5ad4f38bd89dd971e739561d1923172  explorer-v0.9.0-nif-2.15-x86_64-pc-windows-gnu--legacy_cpu.dll.tar.gz
720914c3e85a0869174cd43c26b65400e6ac0131993ea5796689b2d19cf364e2  explorer-v0.9.0-nif-2.15-x86_64-pc-windows-gnu.dll.tar.gz
ddd1f1f70d2791fc662fd477c34f9c4aa18a7d4d1a80bf953e39ac3d49924de7  explorer-v0.9.0-nif-2.15-x86_64-pc-windows-msvc--legacy_cpu.dll.tar.gz
52138d2657f8af5c85b75d22129b107e0547d0ef4b3abca966de48a1b18be6d6  explorer-v0.9.0-nif-2.15-x86_64-pc-windows-msvc.dll.tar.gz
6210497158c3479bdf9f8ab8661fca6bc02addde2cce085d5b134b4cc43ad5d4  libexplorer-v0.9.0-nif-2.15-aarch64-apple-darwin.so.tar.gz
78d54509a7a37e8e174cff4cce06329e75740d2cab8936834f8df06ed6a4eaea  libexplorer-v0.9.0-nif-2.15-aarch64-unknown-linux-gnu.so.tar.gz
3acd48fc82d89eeb74b54db70e3a4f44404f8764f5f63d767f0281a93f91ab35  libexplorer-v0.9.0-nif-2.15-aarch64-unknown-linux-musl.so.tar.gz
2c2390d37a0171c0e096e96620959d78b8d66c73907d219d9f205562df933983  libexplorer-v0.9.0-nif-2.15-x86_64-apple-darwin.so.tar.gz
0dca014ba38a6be5705607bdf6560bd9c485dcd1596e8f0c7dde4e69001a2c93  libexplorer-v0.9.0-nif-2.15-x86_64-unknown-freebsd--legacy_cpu.so.tar.gz
96ec9de1e472a504bd101cd05273873930337089286103911a2d43db72ae48a8  libexplorer-v0.9.0-nif-2.15-x86_64-unknown-freebsd.so.tar.gz
1baee4332cc0f6e5c2e1f505ce3ca98f6350cdcda73a3cde476d2f2690ab2094  libexplorer-v0.9.0-nif-2.15-x86_64-unknown-linux-gnu--legacy_cpu.so.tar.gz
320b2fc65700b09f20c58d0674631b67714dffe665df34dd39e9b9984051ee79  libexplorer-v0.9.0-nif-2.15-x86_64-unknown-linux-gnu.so.tar.gz
7a93e9a34e7dfac721d96dca49c57405b40da5330dfa9ac78add561a234c4040  libexplorer-v0.9.0-nif-2.15-x86_64-unknown-linux-musl.so.tar.gz