numpy-groupies 0.16.2

v0.16.2

on Python PyPI

Release 0.16.2

Highlights

Datetime and timedelta values are aggregated (#186, #115, #183)

The package aggregates datetime64 and timedelta64 values now, and it does so without a new backend: handing a temporal array to aggregate() is all it takes, and the numpy and the numba backend answer it.

npg.aggregate(group_idx, timestamps, func="mean")   # a datetime64, exactly

The rules are the ones of numpy, with two exceptions that are named below:

• the plain reductions carry a NaT through, the nan variants leave it out - the NaT takes the place of the nan.
• the unit of the input is the unit of the answer, and NaT is the default fill.
• what numpy refuses, the package refuses with the same TypeError: no sum, prod, cumsum, cumprod, median, var, std, skew, kurt, trapezoid or sumofsquares of a datetime64.
• the quantile and the percentile of a temporal input follow numpy tick for tick: the interpolation runs on the ticks of the values, with the same rule of truncation that numpy applies.

The mean of a temporal input is exact: the ticks of a nanosecond timestamp are around 1.6e18, far beyond the window a float64 holds exactly, so the mean is computed on the ticks, with the rounding half to even. The same holds for the median and the quantile of the temporal input.

Two deliberate deviations from numpy are pinned by the suite and documented:

• the nan argreductions of a temporal input skip the NaT. numpy does not - np.nanargmin and np.nanargmax answer the position of the NaT of a datetime64 (reported upstream as numpy gh-33207). The contract of the nan forms says skip, and the package keeps it for every dtype.
• the mean of a temporal input exists where numpy has none: numpy cannot add datetime64 values, so it has no mean of them. The package computes it on the ticks.

The crash of flox gh-520 is gone: the mean of the int64 view of a datetime is exact now, and the datetime64 itself can be handed over directly.

The bare kernels of the numba backend take the temporal values as they are, as many as their arithmetic allows - the selections, the positions, the counts, the truth of the values, and the sums and the cumulative sums of the durations. The kernels that cannot take them say so by name.

The mean of integers is exact, and the gate that refused it is gone (#115, #186)

The mean of integer values refused to answer when the sum of the group could leave the +/-2**53 window that a float64 holds exactly, and it suggested to use sum and len instead - which is no advice at all, since neither of them is exact for wide integers.

The mean is now computed exactly, with the sum of the group carried in the limbs of a Python integer: the result does not depend on the backend any more, and no group is refused any more. The variance and the standard deviation centre every group on its first value, so that a large common offset cannot erase the small deviations around it - and a group whose values span the whole signed range is refused with an OverflowError, because the centring would itself wrap.

The wide integer sums keep their dtype as well: the sum of a uint64 is a uint64 and is exact (#185).

Other changes

• The dtype ladder that guesses the dtype of an integer sum keeps the sign of the input now (uint8 -> uint32 -> uint64, int8 -> int32 -> int64, where it turned every unsigned input into a signed one), and its last step is float64 instead of float32 - a mantissa of 24 bits is no place for a sum (#188).
• The helper modules of the package lost their utils_ prefix: they are numpy_groupies.cache, numpy_groupies.nptools, numpy_groupies.temporal and numpy_groupies.typing now (#187). The types the dispatchers carry the values in moved into a module of their own, so that a module which only needs a type does not pull in the whole of numpy.
• minimum_dtype could die with an OverflowError instead of saying that the value does not fit: the comparison of a numpy scalar with a python integer that does not fit into it raises in numpy 2. It is caught now, and the ladder climbs on (#188).
• The cache of the compiled kernels is purged from the whole tree of the package again - the module that walks the tree found the root of the package through its __path__ (#187).

Benchmarks and documentation

• docs/benchmarks.md has a table of its own for the temporal functions, measured with 500,000 values in 1,000 groups: the quantile of a datetime64 takes 41.9 ms in the numpy implementation and 4.1 ms in the numba one, where the median of the same values takes 2.9 ms - the quantile of the ticks runs in a compiled kernel of its own, with all five methods of numpy in it.
• docs/functions.md describes the rules of the temporal input, the two deviations from numpy and the quantile of a temporal input.

Issues

• #115, the exactness of the mean of the integers, and #183, the overflows with datetime64 types, are closed by #186.
• #184, the uint64 sum that answered with a float32, is closed: the first part by #185, the second one by the decision that the overflows of a fixed width sum stay in, the way numpy leaves them.