explorer 0.4.0

v0.4.0

on Hex

This is an exciting release with a lot of features, a bunch of changes and some fixes.
Happy Hacking!

Added

• Add Series.quotient/2 and Series.remainder/2 to work with integer division.

• Add Series.bintype/1 to return the underlying representation type.

• Allow series on both sides of binary operations, like: add(series, 1)
  and add(1, series).

• Allow comparison, concat and coalesce operations on "(series, lazy series)".

• Add lazy version of Series.sample/3 and Series.size/1.

• Add support for Arrow IPC Stream files.

• Add Explorer.Query and the macros that allow a simplified query API.
  This is a huge improvement to some of the main functions, and allow refering to
  columns as they were variables.

  Before this change we would need to write a filter like this:

  Explorer.DataFrame.filter_with(df, &Explorer.Series.greater(&1["col1"], 42))

  But now it's also possible to write this operation like this:

  Explorer.DataFrame.filter(df, col1 > 42)

  This operation is going to use filter_with/2 underneath, which means that
  is going to use lazy series and compute the results at once.
  Notice that is mandatory to "require" the DataFrame module, since these operations
  are implemented as macros.

  The following new macros were added:

  • filter/2
  • mutate/2
  • summarise/2
  • arrange/2

  They substitute older versions that did not accept the new query syntax.

• Add DataFrame.put/3 to enable adding or replacing columns in a eager manner.
  This works similar to the previous version of mutate/2.

• Add Series.select/3 operation that enables selecting a value
  from two series based on a predicate.

• Add "dump" and "load" functions to IO operations. They are useful to load
  or dump dataframes from/to memory.

• Add Series.to_iovec/2 and Series.to_binary/1. They return the underlying
  representation of series as binary. The first one returns a list of binaries,
  possibly with one element if the series is contiguous in memory. The second one
  returns a single binary representing the series.

• Add Series.shift/2 that shifts the series by an offset with nil values.

• Rename Series.fetch!/2 and Series.take_every/2 to Series.at/2
  and Series.at_every/2.

• Add DataFrame.discard/2 to drop columns. This is the opposite of select/2.

• Implement Nx.LazyContainer for Explorer.DataFrame and Explorer.Series
  so data can be passed into Nx.

• Add Series.not/1 that negates values in a boolean series.

• Add the :binary dtype for Series. This enables the usage of arbitrary binaries.

Changed

• Change DataFrame's to_* functions to return only :ok.
• Change series inspect to resamble the dataframe inspect with the backend name.
• Rename Series.var/1 to Series.variance/1
• Rename Series.std/1 to Series.standard_deviation/1
• Rename Series.count/2 to Series.frequencies/1 and add a new Series.count/1
  that returns the size of an "eager" series, or the count of members in a group
  for a lazy series.
  In case there is no groups, it calculates the size of the dataframe.
• Change the option to control direction in Series.sort/2 and Series.argsort/2.
  Instead of a boolean, now we have a new option called :direction that accepts
  :asc or :desc.

Fixed

• Fix the following DataFrame functions to work with groups:
  • filter_with/2
  • head/2
  • tail/2
  • slice/2
  • slice/3
  • pivot_longer/3
  • pivot_wider/4
  • concat_rows/1
  • concat_columns/1
• Improve the documentation of functions that behave differently with groups.
• Fix arrange_with/2 to use "group by" stable, making results more predictable.
• Add nil as a possible return value of aggregations.
• Fix the behaviour of Series.sort/2 and Series.argsort/2 to add nils at the
  front when direction is descending, or at the back when the direction is ascending.
  This also adds an option to control this behaviour.

Removed

• Remove support for NDJSON read and write for ARM 32 bits targets.
  This is due to a limitation of a dependency of Polars.

New Contributors

• @Ian-GL made their first contribution in #352
• @pgeraghty made their first contribution in #361
• @thbar made their first contribution in #362
• @nallwhy made their first contribution in #366
• @xaviRodri made their first contribution in #365
• @tmr08c made their first contribution in #388
• @linusdm made their first contribution in #412
• @thmsmlr made their first contribution in #418
• @sasikumar87 made their first contribution in #433

We also had @joshuataylor and @Benjamin-Philip contributing to this release.

Full Changelog: v0.3.1...v0.4.0
Official Changelog: https://hexdocs.pm/explorer/changelog.html

Checksums

Above is the checksum of the precompiled NIFs.

5c806f61d76e161b03372d7607704a98a50e0a0687c2d763041edcc265a7e1ed  explorer-v0.4.0-nif-2.15-x86_64-pc-windows-gnu.dll.tar.gz
017304769d8ad46cdfa44520ceba6431b7f5c8934c8dd62613317126bfaf4447  explorer-v0.4.0-nif-2.15-x86_64-pc-windows-msvc.dll.tar.gz
97680f8a446c2de04add8a66d63db44085d4b7008fb135fc8a3c62df3bdb6857  explorer-v0.4.0-nif-2.16-x86_64-pc-windows-gnu.dll.tar.gz
b4339bad6faabad736a698fc989cf0f24640ea98f46d4ec3a79330b7ef273673  explorer-v0.4.0-nif-2.16-x86_64-pc-windows-msvc.dll.tar.gz
01678187108b7dc472748e9163bfbad15f74aeca5da96c285ad22ce1f4398994  libexplorer-v0.4.0-nif-2.15-aarch64-apple-darwin.so.tar.gz
22b5d6ac3a0123f0b3d4c84129d2730c583d59dfa846b77524d13ee34baceb96  libexplorer-v0.4.0-nif-2.15-aarch64-unknown-linux-gnu.so.tar.gz
21d9c5c81a86aa23d6d0e630307bf86732e0e7f31b35d2aad093248ec747ba65  libexplorer-v0.4.0-nif-2.15-arm-unknown-linux-gnueabihf.so.tar.gz
aed562d906f16cecca3fc12e77a8a03825f0aa52ef3421a8db0df9f258fd7e29  libexplorer-v0.4.0-nif-2.15-x86_64-apple-darwin.so.tar.gz
60f66e0a3ec24b323b7996b5c30d0b9fbb714674fa0fa2dcbe48b1b7af9e85f4  libexplorer-v0.4.0-nif-2.15-x86_64-unknown-linux-gnu.so.tar.gz
24387916a67f93d2a4dd2e2b595451ad3ed052dddf91956bd9dc9ec1a39cb32c  libexplorer-v0.4.0-nif-2.15-x86_64-unknown-linux-musl.so.tar.gz
3fabdc92ba66c33f0a6b220a06dccfa3738d231d90e1824c47c0a4a838d2bee8  libexplorer-v0.4.0-nif-2.16-aarch64-apple-darwin.so.tar.gz
9213e107fb29a416c9db40040b4b657199bd5e9e95990e1b6fcf6e090a006810  libexplorer-v0.4.0-nif-2.16-aarch64-unknown-linux-gnu.so.tar.gz
025fb32c9b337d3bcc50be321c9a26bc6ea7ecbef0813df7729a53e6826c0fe3  libexplorer-v0.4.0-nif-2.16-arm-unknown-linux-gnueabihf.so.tar.gz
7ad59e6e041267a78d0671ef11b60e8edca66ae2cd58e052731ab38f3c1af8d2  libexplorer-v0.4.0-nif-2.16-x86_64-apple-darwin.so.tar.gz
0bdddb296b7be4957fe40cdcbfc1fa92802fab1fed0d6fd40122acbcc90f6b61  libexplorer-v0.4.0-nif-2.16-x86_64-unknown-linux-gnu.so.tar.gz
de8dd8313d7fff2fbd8d70bb26f163b60af9ac41bc26f799ca94aae78570d45f  libexplorer-v0.4.0-nif-2.16-x86_64-unknown-linux-musl.so.tar.gz