numpy-groupies 0.16.1

v0.16.1

on Python PyPI

Release 0.16.1

Highlights

The second wave of the parallelization (#180)

The operations of the Numba backend that were still running on one thread now run on the parallel paths that the cost model decides:

• skew and kurt - the Welford state is folded with the merge of Chan et al. The result differs from the sequential one in the last bits, the way the sum of the floats does.
• first and last - the same scan as the sequential path, the chunks folded in the order of their index. The result is bit for bit identical.
• argmin and argmax - the same scan, the fold compares the states of the chunks. The result is bit for bit identical.
• The callable of the caller - the groups are cut into ranges the threads work on, like the order statistics. The result is bit for bit identical.

Measured with 500,000 values in 1,000 groups on six cores: skew -61 %, kurt -62 %, first -33 %, last -46 %, argmax -37 %, argmin -47 %, the callable of the caller -53 %.

One scan for both paths

The sequential driver of the argmax, the argmin, the first and the last is now a thin wrapper around the very same scan that the parallel path runs over every chunk of them. The tie-break (the first of two equal values wins) and the poisoning of a group by a nan are written down once and cannot drift apart between the two paths.

The nan variants of the argmax and the argmin got about 30 % faster by it - the select-form wrote the value of every group at every value, the branchy scan writes it once per group - and the last about 30 % faster in the sequential path, for the same reason.

The counting sort of the order statistics, the gather loop and the end of an aggregation (the exactness gate, the finishing pass and the fill of the untouched groups) were written out several times each; now they are written once each.

The cache of the compiled kernels leaves the source tree (#181)

The cache of the compiled kernels lived next to the source - in the installation directory of an installed package. Numba never cleans its cache up, so every renamed kernel left its files behind forever, and the cache of an installed package survived the uninstallation and left a broken installation behind (numba #9312).

The cache now lives in $NUMBA_CACHE_DIR if the environment names one, and in ~/.cache/numba/numpy_groupies otherwise - a directory of the package, so that the cache of this package can be removed without touching the one of another.

npg.purge_cache() removes it, both the directory and the leftovers of the versions that cached next to their source, and returns the number of the files it removed:

npg.purge_cache()   # the compiled kernels are gone, the next call compiles them anew

Other changes

• The dead branch in the fold of the moments is fixed: the check read the count that was already folded.
• The weight of the state of the moments in the cost model is calibrated (5 instead of 6). With the values 3 and 4 the model would send shapes to the parallel path in which it is measurably slower (0.70x with 8,000 groups).
• The number of the threads of the pool is asked for a callable of the caller only when the call is big enough for the parallel path at all.
• The annotations of the index arrays of the argmax and the argmin say int64, as the rule of the kernel and the NumPy implementation have it.

Benchmarks and documentation

• The benchmark tables of the readme and of docs/benchmarks.md are refreshed. Within the measurement uncertainty no operation is slower than before; the differences that the first comparison showed were the load of a second benchmark process on the same machine - the operations concerned are the ones that run on the parallel path, and they are the load-sensitive ones.
• The readme has a section on the cache of the compiled kernels, and docs/functions.md describes purge_cache().