Changes from 3.3.3 to 3.3.4
This release introduces bounded-memory tail compaction for contiguous on-disk
frames and clean file eviction for payload-free chunks in sparse frames.
-
Bounded-buffer tail compaction in contiguous frames:
When updating a chunk in place within an on-disk contiguous frame (cframe),
a change in compressed size requires shifting all subsequent chunks in the
payload tail. Previously, Blosc2 allocated a single heap buffer sized to the
entire remaining tail (malloc(tail_nbytes)). For multi-gigabyte or terabyte
frames, updating an early chunk could require gigabytes of RAM, risking
out-of-memory aborts.Tail compaction now uses an internal bounded-buffer movement helper
(frame_move_range) capped at 1 MiB (FRAME_TAIL_COPY_BUFFER_CAP), decoupling
RAM consumption from frame size:- Shifts forward when shrinking (
dst < src) and backward when expanding
(dst > src) to guarantee overlap safety across any displacement. - Reads each segment fully into owned scratch memory before writing, avoiding
pointer invalidation when memory-mapped files are remapped during expansion. - Adds checked arithmetic to prevent integer overflow and negative buffer
allocations. - Consolidates error cleanup paths in
frame_update_chunk()and
frame_update_trailer(), and accounts forframe->file_offseton truncation.
- Shifts forward when shrinking (
-
Sparse frame special chunk file removal:
Updating a chunk to a payload-free special value (such as ZERO, UNINIT, or NAN)
in a sparse frame (sframe) previously truncated the chunk file to 0 bytes
rather than deleting it. Repeated cache eviction and refill cycles could leave
thousands of empty chunk files (0000000X.chunk), cluttering directories and
exhausting filesystem inodes.Updating to a payload-free special value now unlinks and deletes the old
physical chunk file (sframe_delete_chunk()), matching the representation of
never-materialized special chunks:BLOSC2_SPECIAL_VALUEchunks (which store repeated payload data) retain
their physical chunk file.- Adopts recoverable-orphan ordering: the chunk file is deleted only after
publishing the updated offsets index and trailer to disk. - Properly tracks the preceding physical chunk ID when logical chunks have been
reordered. - Custom I/O backends that map file paths are handled cleanly: standard backends
useremove()(treatingENOENTas idempotent success), while custom backends
use their registered callbacks (open,truncateto 0,close), preventing
unintended file operations on host paths. - Eviction failures occurring after the metadata update has been committed are
logged as warnings rather than returning an error, preventing committed
updates from being reported as failures.
-
Memory mapping (
mmap) fixes and external file growth preservation:- Memory remapping on non-Linux POSIX systems (macOS, BSD) in
blosc2_stdio_mmap_write()now maps cleanly to a new address before
unmapping the old one, avoidingMAP_FIXEDpitfalls. - In
blosc2_stdio_mmap_destroy(), file size truncation respects external
growth. On Windows, file padding tomapping_sizeis only trimmed if the
file has not grown externally, avoiding truncating appends made by other
handles. On POSIX, avoids truncating away external writes or appends.
- Memory remapping on non-Linux POSIX systems (macOS, BSD) in
-
New benchmarks and tests:
- Added
bench/bench_compact_tail.cto benchmark tail updates across
different buffer caps and chunk positions (with portable timing support
across POSIX and Windows). - Added comprehensive test suites in
tests/test_compact_tail_update.cand
tests/test_sframe_special_eviction.c.
- Added
There are no API or format changes in this release.