numpy-groupies 0.16.3

Release 0.16.3

on Python PyPI

Release 0.16.3

Highlights

The topk: the k largest (or smallest) values of every group (#189)

The package keeps the k largest values of every group now, and the k smallest ones as well:

npg.topk(group_idx, a, k=5)                 # the 5 largest values of every group
npg.topk(group_idx, a, k=5, largest=False)  # the 5 smallest ones
npg.aggregate(group_idx, a, func="topk", k=3)

The answer is a matrix with one row per group and k columns, the values in the order of their rank, the largest one first. A group with fewer than k values fills the places it has none with the fill value (nan for floating output, 0 where the output is an integer type and cannot hold a nan), and a group that holds no value at all is filled in every one of its columns.

The missing values are in no selection: the nans of the inexact dtypes and the NaT of the temporal ones are neither the largest nor the smallest value of a group, the way the sort of a group puts them last - so there is no nan-variant of the topk, and func="nantopk" refuses itself with the same words func="nansort" does. The temporal values are aggregated on their int64 ticks, where the NaT is the smallest tick of all.

The numpy implementation partitions the values of the group and takes the k wanted ones out of the block, the numba one selects them with the quickselect that the median and the quantile use - the same counting sort, the same parallel path over the groups. Measured with 500,000 values in 1,000 groups: 13.9 ms for the numpy implementation and 2.9 ms for the numba one, about the time of the median.

The internals of the numpy path, untangled (#190)

The shared modules of the numpy based backends were reworked, with no change of behaviour pinned by an equivalence suite over 61,920 temporal cases:

• The temporal policy lives in one shared engine now, and each backend hands the operations it built to its own dispatch - the numba backend feeds its compiled kernels into the same policy instead of carrying its own decisions, and every backend states the way it calls its operations explicitly.
• The one nptools module of the numpy path and the common module of the package are resolved into modules of their own: numpy_groupies.driver (the shared aggregate and the out-of-place out handling), numpy_groupies.dtypes (the dtype ladders and the classification), numpy_groupies.fills (the fill value defaults), numpy_groupies.functions (the properties of the functions), numpy_groupies.groups (the group tools) and numpy_groupies.inputs (the input validation). The import graph of the numpy-free part of the package is acyclic, and the pure python implementation still imports without numpy.

Bug fix

• aggregate_numpy and the benchmark backends swallowed an out= argument silently - it fell into their **kwargs and the call answered a fresh array as if none had been given. Only the numba implementation supports the array of the caller, and every other path refuses it with a NotImplementedError now instead of ignoring it.

Other changes

• The marker that asks for the default of a fill value is named DefaultFillMarker, and it lives with the fill values; the module of the types names it only where the type checker looks.
• The isnan kernels that only the pandas benchmark used live in the benchmark now.