Cyan4973/xxHash v0.7.4

xxHash v0.7.4 - Finalizing XXH3 and XXH128

on GitHub

xxHash v0.7.4 is the last evolution of xxh3 and xxh128, primarily designed to finalize the algorithm.
It is considered release candidate for v0.8.0, which means that if all goes right, this version will rebranded v0.8.0, almost "as is", within the next few weeks, after receiving sufficient feedback.
v0.8.0 is the official version after which XXH3 and XXH128 are considered "stabilized", meaning that return values will never change given the same input and seed, making the hash suitable for long-term storage and transmission.

Beyond these "final touches", the new version also brings a few notable improvements.

Automatic vector detection

x86/x64 systems can enjoy a new unit, xxh_x86dispatch, which can detect at runtime the best vector instruction set present on host system (none, sse2, avx2 or avx512), thanks to a cpu feature detector designed by @easyaspi314. It then automatically runs the appropriate vector code.
This makes it safer to deploy a single binary with advanced vector instruction sets, such as AVX2, since there is no hard requirement for all target systems to actually support it : the binary can automatically switch to SSE2 instead.
As a proof of concept, the windows builds provided alongside this release are compiled with this new capability.

AVX512 support

A new vector instruction set is supported, thanks to @gzm55 : AVX512. It can be applied on XXH3 and XXH128, using some of the most recent Intel cpus, such as IceLake on laptop. It typically offers +50% more performance compared to AVX2.

Secret Generator

Advanced users can be interested in the highly customizable variant _withSecret(), which makes it possible to run XXH3 and XXH128 algorithms using one's own secret.
However, the quality of the hash depends on the high entropy (randomness) of the secret. And sometimes, it can be difficult to ensure that the candidate secret is "random enough".
In order to produce a secret of high quality, a new function XXH3_generateSecret() is proposed in the advanced API section. It will convert any blob of bytes, named customSeed, into a high quality secret which respects all conditions expected by XXH3 and XXH128. This is true even if customSeed itself is of poor quality, such as a bunch of \0 bytes or some short or repeated common sequence.

No API modification

The existing API present in 0.7.3 has remained unchanged in 0.7.4. Any programs linking with 0.7.3 should continue to work as-is.
Note however that xxh3/xxh128 return values are not comparable across these versions.
0.7.x are labelled development versions, and should only be used for ephemeral data (hash produced and consume in the same local session).
(note : this limitation does not extend to XXH32 and XXH64, which are considered fully stable and specified).

Changelist

There are multiple smaller bug fixes and minor improvements that have been brought to this repository by great contributors. Here is a summarized list:

• perf: automatic vector detection and selection at runtime (xxh_x86dispatch.h), initiated by @easyaspi314
• perf: added AVX512 support, by @gzm55
• api : new: secret generator XXH_generateSecret(), suggested by @koraa
• api : fix: XXH3_state_t is movable, identified by @koraa
• api : fix: state is correctly aligned in AVX mode (unlike malloc()), by @easyaspi314
• api : fix: streaming generated wrong values in some combination of random ingestion lengths, reported by @WayneD
• cli : fix unicode print on Windows, by @easyaspi314
• cli : can -c check file generated by sfv
• build: make DISPATCH=1 generates xxhsum and libxxhash with runtime vector detection (x86/x64 only)
• install: cygwin installation support
• doc : Cryptol specification of XXH32 and XXH64, by @weaversa