Release Notes

v0.14.0 is a decode-speed release built on a format break.

This release boosts decompression speed by up to 19% over v0.13.x, up to doubling our performance lead over LZ4 on ARM64 (+85% to +105%) and up to tripling it on x86_64 (+50% to +175%).

Three independent changes get there: blocks stop describing what the decoder can derive (format v8), short back-references are copied 32 bytes at a time instead of 16, and at levels 1-5 the encoder keeps its matches far enough back that the widest SIMD copy path stays on. The PivCo Huffman kernels also picked up AVX-512 VBMI, 256-bit AVX2 and NEON work.

Nothing in the public API moved: no symbol added, none removed, ABI unchanged (libzxc.so.4). The break is in the container format.

⚠️ Breaking change (format). The container format is now v8. v0.14.0 writes v8 archives that v0.13.x and earlier decoders reject, and it cannot read v7 or older: the header version byte is checked for exact equality (ZXC_ERROR_BAD_VERSION). The ABI is unchanged (soname stays libzxc.so.4), so no re-link is required — but upgrading the library alone makes newly written archives v8.

Migrating v7 -> v8 is a one-time transcode — decompress with your old build, recompress with the new one:

zxc-old -dc old.zxc | zxc-new -z -c > new.zxc

Keep a v7 build around until your data at rest is converted; if the old archive used a dictionary, pass it to the decompress side (zxc-old -dc -D dict.zxd). Full guide (bulk migration, dictionaries, verification): docs/MIGRATION.md.

ZXC format v8

Compression is untouched — same parser, same match finder, same entropy coders. What changed is how a block describes itself, and every change pulls the same way: stop writing down what the decoder can work out. (#374)

• Block sub-header: 16 -> 12 bytes. Four reserved bytes nobody used.
• Descriptors only when they carry information. v7 wrote a fixed 32-byte table of section sizes; most of them follow from the header. v8 keeps the two that don't: the literal section's compressed size when it is RLE- or entropy-coded, and the token section's at level 7. GLO now carries 0, 4 or 8 descriptor bytes, GHI none. The extras section is the leftover payload.
• enc_litlen -> enc_tok. The field always described the token section, never literal lengths.
• 32 readable bytes must follow the literal section. Literals are copied with a wild copy that overshoots, and those bytes keep the last copy inside the payload. Real blocks clear the bar without trying; the encoder pads into the extras when a tiny block doesn't. Always relied on, never written down — now normative, and decoders must reject blocks that break it.
• enc_lit, enc_tok and enc_off are closed value sets. Decoders reject unknown values instead of defaulting, which keeps the spare values free for a future version.

Per block this saves 25 to 34 bytes: invisible at the 512 KB default (levels 6 and 7 land 0.02 % smaller), worth about 2 % at the 4 KB blocks where dictionaries live.

Performance

Decompression on Silesia (202 MB), single-threaded, v0.13.3 -> v0.14.0 on the same reference hardware:

In absolute terms, level 3 decodes at 8.4 GB/s on Apple M2 and 6.7 GB/s on Zen 5; level 6 holds 6.7 GB/s on M2 at a 36.28 % ratio, ahead of lz4hc -9 on both axes on every machine measured. Full tables, cycles/byte and effective throughput: README and WHITEPAPER.

Three levers get there, all independent.

Overlap copies run 32 bytes at a time (#374)

Short back-references took one of two paths, and one re-read bytes it had just written at a distance straddling two stores: no store forwarding, a stall every iteration. A 32-byte pattern covers distances 2 to 31 in a single arm — the stall goes, four distance tests become three, and the old 16-byte kernel (four SIMD variants, two lookup tables) is deleted. Short GLO sequences also get their own copy paths, specialized on branches the decoder was already taking.

A minimum match distance at levels 1-5 (#376)

Levels 1 to 5 now keep matches at least 32 bytes back, so the decoder stays on its widest SIMD copy path instead of dropping into overlap handling. On periodic data that floor would cost ratio and speed, so the call is made per block: a probe samples about one position per KB (16 to 64 samples) for 4-byte repeats inside the forbidden window, and past 20 % hits the block keeps its short matches. Levels 6 and 7 are untouched.

The trade is paid in ratio:

Levels 1-5 buy 10-19 % of decode speed for under 1 % of size. Levels 6 and 7 keep their ratio and get slightly smaller, from the v8 header savings.

Wider PivCo Huffman kernels (#373, #377)

• AVX2 merge widened to 256 bits. The AVX2 tier was still running the SSSE3 kernel: two 128-bit selects, an OR and two stores per 32 outputs. Now one pair of 256-bit selects, a blend and a single store — 16 vector instructions instead of 24. (#377)
• AVX-512. _mm512_mask_expandloadu_epi8 folds the masked load and the byte expand into one fault-safe instruction. With VBMI, vpmultishiftqb does the unaligned D-bit read in a single instruction, 64 codes per step, and leaf emission uses the control word directly as its blend mask. The dispatch gate now checks VBMI, which the variant is already built with.
• AVX2 unpack. A 32-codes-per-step path for D = 2..6, plus a 32-byte leaf emitter.
• NEON. vld1q_u8_x2 / vld1q_u8_x4 for the c2s tables, instead of filling the lanes one load at a time.
• Encoder. A symbol's bit-reversed residual never changes, so it is computed once per symbol instead of once per literal.

Reliability & Security

In-place decompression: forged footers and buffer separation (#375)

• Forged footer sizes. The decompressed size from the footer is checked for plausibility before anything is sized from it, so untrusted input can't drive an outsized allocation. The check is centralized and shared by every path that reads the footer.
• Buffer separation floor. Attacker-controlled padding could shrink the read/write separation in-place decoding depends on. A minimum floor is enforced, so the margin holds whatever the archive claims.
• A forged footer caught during in-place decompression returns ZXC_ERROR_CORRUPT_DATA rather than a vaguer code.

Stricter block validation (#374)

Blocks that break the 32-readable-bytes rule are rejected. Unknown enc_lit / enc_tok / enc_off values are rejected instead of silently defaulting. The stream layer reports distinct error codes where it used to return one generic I/O error.

Library internals: hardening and consolidation (#391)

What changes behaviour:

• Destination capacity below a block sub-header. The size check wrapped around and the encoder wrote the sub-header out of bounds. Rejected up front now, on the block and frame APIs.
• Arithmetic that could wrap. Block-header validation moved to 64 bits; the seekable entry count is checked against its 32-bit field; the block count uses division-and-remainder instead of a ceiling addition.
• A truncated stream with an agreeing footer decompressed as a clean short read. An unreadable block header is corruption now, and the stream layer stops swallowing allocation failures.
• Measuring runs are exact. Compressing with a NULL output wrote the file header and the seek table without counting them, under-reporting by 16 bytes, 52 with a seek table.

The rest is consolidation: one threading shim instead of two, one RLE lookahead instead of one per SIMD backend, centralized cleanup on the engine and dictionary-training failure paths, and a read buffer held on the seekable handle instead of reallocated per range request.

Format testing: annotated vectors, frozen corpus (#407)

The format moved this release under a test suite that could only say the bytes differ.

• Golden archives ship an annotated dump. Each tests/format/golden/*.zxc has a .zxc.txt beside it: every header field decoded and named, plus a payload hash. A format change reviews as a readable diff instead of Binary files differ, and the hash catches an encoder change that leaves the fields intact.
• The conformance corpus is versioned and frozen. Vectors live in conformance/v8/, one directory per format version, each with a vectors.sha256 manifest CI verifies byte for byte. v8 ships 15 valid and 28 invalid vectors.
• Valid vectors have a written recipe. valid_cases.h and zxc_valid_gen make the construction parameters explicit and the corpus regenerable, and the suite is one entry per decoder-visible trait rather than one per level, so a failure points at a single format rule.

The golden and conformance workflows merged into one, Test Vector Stability.

Release Integrity & Packaging

Signed, verifiable releases (#383)

Releases ship a canonical source tarball, zxc-<version>.tar.gz, reproducible with git archive --format=tar --prefix=zxc-<version>/ v<version> | gzip -n -9, plus the same tar compressed with zxc -7 (.tar.zxc). Every asset is listed in checksums.sha256, signed with minisign — verify the manifest before trusting it:

minisign -Vm checksums.sha256 -P 'RWQV0cpiyJYPkxF5iIysJzKNtzcGphqeyyFkiFErLMo5UZkWisGBxkNB'
sha256sum -c checksums.sha256 --ignore-missing
gh attestation verify zxc-<version>-linux-x86_64.tar.gz --repo hellobertrand/zxc

SLSA provenance is generated for the binaries, release tags stay PGP-signed, and asset upload plus wrapper publishing are centralized in the workflows. SECURITY.md and the README document the full verification path. The SBOM file name drops the tag's v.

glibc floor back at 2.34 (#385)

_GNU_SOURCE had quietly pulled the Linux binaries up to GLIBC_2.38, through __isoc23_strtol. Requesting POSIX.1-2008 and _XOPEN_SOURCE instead brings the floor back to 2.34 on x86_64 and arm64, in both the CMake and Meson builds.

libzxc.pc is relocatable (#385)

The pkg-config file baked a /usr/local prefix at configure time, which broke anyone extracting a pre-built archive somewhere else. The prefix comes from ${pcfiledir} now.

Developer Certificate of Origin (#388)

Contributions now carry the same provenance guarantee as the release artifacts. ZXC adopts the Developer Certificate of Origin 1.1 — reproduced verbatim in CONTRIBUTING.md — and every commit must carry a Signed-off-by trailer:

git commit -s -m "feat: your commit message"

The trailer is taken from your configured user.name and user.email. It certifies that you wrote the contribution or otherwise have the right to submit it under BSD-3-Clause, and that it stays in the public project history with that name and address.

Documentation

• FORMAT: the v8 block layout — 12-byte sub-header, descriptor rules, extras residue, 32-byte tail invariant. Hexdump examples corrected.
• MIGRATION: the v7 -> v8 transcode guide (bulk migration, dictionaries, verification).
• Doxygen: header documentation deduplicated — one copy of each block (#378), and all warnings cleaned up (#390).
• Terminology: the header integrity field is a checksum, not a CRC, everywhere — docs, error constants, comments and wrappers. (#389)

Acknowledgements

ZXC's Huffman bitstream is built on PivCo-Huffman. Sincere thanks to @MarcinZukowski for the technique and its paper, and to @rygorous, whose breakdown of the PivCo merge operations made the SIMD decoder practical — this release widens that merge to 256 bits on AVX2 and adds the AVX-512 VBMI unpack.

Thanks also to @inikep (lzbench) and @powturbo (TurboBench) for keeping ZXC in the independent benchmark suites, to @guglovich for packaging ZXC for the Arch Linux AUR, and to @jeanga for continued testing and packaging help.

Changelog

• api: Pin the glibc floor at 2.34, make libzxc.pc relocatable (#385)
• api: Wire format v8, faster overlap copies, stricter block validation (#374)
• perf: Optimizes PivCo Huffman with advanced SIMD (#373)
• perf: Make the short match distance floor a default policy (#376)
• perf: Widen the PivCo merge to 256 bits on AVX2 (#377)
• perf: Improve code and enhances robustness (#391)
• fix: Validates in-place footer size and buffer separation (#375)
• build: bump setuptools from 83.0.0 to 84.0.0 in /wrappers/python (#397)
• build: bump build from 1.5.0 to 1.6.0 in /wrappers/python (#394)
• build: bump cibuildwheel from 4.1.1 to 4.2.0 in /wrappers/python (#398)
• build: bump vitest from 4.1.10 to 4.1.11 in /wrappers/nodejs (#395)
• build: bump thiserror from 2.0.19 to 2.0.20 in /wrappers/rust (#393)
• build: bump cc from 1.4.0 to 1.4.4 in /wrappers/rust (#392)
• build: bump node-addon-api from 8.9.1 to 8.9.2 in /wrappers/nodejs (#396)
• build: bump github/codeql-action/analyze from 4.36.0 to 4.37.9 (#400)
• build: bump pypa/gh-action-pypi-publish from 1.14.0 to 1.14.2 (#402)
• build: bump cross-platform-actions/action from 1.2.0 to 1.5.0 (#401)
• build: bump meson from 1.11.2 to 1.12.0 in /.github/requirements (#399)
• build: bump actions/setup-go from 6.4.0 to 7.0.0 (#404)
• build: Rename release checksum manifest to checksums.sha256
• build: Improve release verifiability and automation (#383)
• doc: Fix hexdump examples
• doc: clean up Doxygen warnings and improve API documentation structure (#390) (#378)
• doc: rename CRC to checksum in documentation and code comments (#389)
• misc: Enhances format testing and maintainability (#407)
• misc: formalize Developer Certificate of Origin requirements (#388)
• misc: Enforce 10-minute timeout for CI jobs (#387)
• misc: Drop the tag's "v" from the SBOM file name

Full Changelog: v0.13.3...v0.14.0