Changes from 3.3.5 to 3.4.0
This is a minor release introducing an opt-in Blosc1-compatible chunk writer,
bounded-memory APIs for variable-length block decompression, as well as
important bug fixes and robustness improvements in DELTA filter decoding, blocksize
tuning, ZFP codec handling, b2nd slicing, and thread management.
- New opt-in writer for Blosc1 (C-Blosc 1.x compatible) chunks:
blosc2_ctx_set_blosc1_format()makes a compression context write chunks in
format version 2 with the 16-byte header, which C-Blosc 1.x can decode (stock
C-Blosc2 chunks, even withBLOSC_BLOSC1_COMPAT, cannot be). Besides the
header, such chunks follow the version 2 bitshuffle rule (a block is
bit-shuffled only if its number of elements is a multiple of 8) and only split
blocks that a C-Blosc 1.x reader also splits. Only plain shuffle/bitshuffle with
the blosclz, lz4, lz4hc, zlib and zstd codecs are supported; anything else
(other filters including delta, other codecs, dictionaries, variable-length
blocks) fails with an error. Off by default, no ABI change. The
BLOSC_BLOSC1_COMPATenvironment variable now enables the same mode for
blosc1_compress()(it previously only dropped the extended header, still
writing a format version 5 chunk), so delta there is now an error.
blosc2_ctx_get_blosc1_format()reports whether a context is in this mode. The
filter may be in any slot of the pipeline; a prefilter is applied as usual.
Thanks to Mark Kittisopikul (@mkitti). - Added
blosc2_vlblock_get_nbytes_ctx()to query one block's uncompressed size
without payload, dictionary, or scratch allocation. Lazy queries read only
the selected four-byte prefix. - Added
blosc2_vldecompress_block_into_ctx()for single-block decompression
into a caller-owned buffer. Capacity is checked before dictionary/scratch
allocation, and scratch is sized from the admitted block. Both APIs support
ordinary and lazy VL chunks. The existing allocating API is unchanged. - Fixed DELTA filter decoding on partial chunks, maskout, and getitem:
When an array compressed with the DELTA filter was sliced or read partially,
non-zero blocks could be corrupted because DELTA encodes blocks > 0 relative
to block 0. If block 0 was masked out or omitted (such as inb2nd's compact
read path for small block fractions, or in block-level getitem), subsequent
blocks decoded against uninitialized memory. b2nd now disables the compact
path for DELTA and always unmasks block 0, and_blosc_getitemand
blosc2_decompress_block_ctxfall back to full chunk decompression when DELTA
is active. Destination pointers and sizes are reset after fallback decompression
to avoid leaving dangling references on reused contexts. Fixes Blosc/python-blosc2#740.
Thanks to Karol Gotkowski (@Karol-G) for reporting this issue. - Fixed bit-shuffled Blosc1 chunks with leftover bytes being decoded incorrectly:
When reading a chunk with format version 2 whose bit-shuffled blocks have a
number of elements that is a multiple of 8, the bytes left over after the last
whole element were not copied to the output, so the end of the buffer was
garbage while the call reported success. They are now copied, as C-Blosc 1.x
does. See #812. Thanks to Mark Kittisopikul (@mkitti). - Fixed chunks that could not be read back when the requested block size was not a
multiple of the typesize: the tuner rounds a requested block size down to a multiple
of the typesize, but the chunk header recorded the size that had been asked for, so a
reader cut the chunk into blocks of a different size and failed (error -3, or wrong
data) with every filter and codec. The header now records the block size that is used.
Thanks to Mark Kittisopikul (@mkitti). - Fixed the block size of buffers with a typesize above
BLOSC_MAX_TYPESIZE: such
buffers are stored as a stream of bytes with a typesize of 1, but the tuner chose the
block size with the large typesize and only then was it capped. A buffer shorter than
the typesize was split into blocks of one byte (200 bytes of typesize 300 took 1831
bytes) and a longer one got a block size that was a multiple of the large typesize.
The typesize is now capped before the tuner runs, so that it is blocked exactly like a
typesize of 1. Thanks to Mark Kittisopikul (@mkitti). blosc2_compress_ctx()with a typesize of 0 stopped with SIGFPE (a division by
zero when choosing the block size). It now returnsBLOSC2_ERROR_INVALID_PARAM.
Thanks to Mark Kittisopikul (@mkitti).- Hardened ZFP codec against bitstream over-read and invalid metadata:
ZFP bitstream decompression now operates from a zero-padded buffer sized to
zfp_stream_maximum_size(), preventing out-of-bounds heap reads when 64-bit
word reads extend past unaligned payloads (blocks * bitsnot a multiple of 64).
Also guarded against word-alignment over-read inzfp_getcell(), removed
redundant stream opens, validated metalayer bounds viab2nd_deserialize_meta_inline,
and rejected non-positive blockshapes and invalidndim.
Thanks to @kalt2212 and Farkhalit Rida (@fr-manvi). - Hardened
b2nd_get_slice_nchunksbounds handling and allocation:
Clamped slice bounds against negative values andextshape, sizedchunks_idx
from slice bounding boxes to prevent mismatch overflows, protected against
multiplication overflow on 32-bit platforms, and handled empty slices and
shrinking reallocations without memory leaks.
Thanks to Farkhalit Rida (@fr-manvi). - Rejected split blocks where streams do not cover blocksize:
Inblosc_d, decompression verifies thatneblock * nstreams == bsizeand returns
BLOSC2_ERROR_DATAif split streams fail to cover the full block, avoiding leaving
bytes unwritten in destination buffers (backported from C-Blosc #407).
Thanks to @suraiya-ctrl. - Avoided unused parallel work for small block counts:
Single-block compression and decompression jobs remain serial whennthreads > 1,
avoiding worker thread pool startup and overhead when threads cannot be used.
Parallel dispatch also caps worker entries tomin(nthreads, nblocks).
Fixed thread backend initialization on Windows. Thanks to
Johnny Kao (@Johnny-Kao). - Added
examples/vlchunks.cillustrating variable-length chunk usage, updates,
compression, and verification. Thanks to Lance Uymatiao (@lanceu0128).