Changes from 3.4.0 to 3.4.1
This is a bugfix release that brings substantial performance improvements for
in-place writes to on-disk contiguous frames (especially on NFS), security and
robustness fixes for sparse frames, fixes for filter pipelines with 3 or more
filters alongside multithreaded DELTA decoding, and an example for sparse
coordinate handling.
- Faster in-place writes to on-disk contiguous frames, especially on NFS:
appending, inserting, updating or deleting a chunk (and so b2nd
set_slice) opened and closed the frame file four times and truncated it on
every trailer update. On NFS each close is a synchronous flush to the
server, which made such writes cost about 20 ms per chunk whatever its size.
With the default filesystem backend on POSIX, a frame now keeps one writable
handle open, truncates the file only when the frame got shorter, and checks
for a replaced file withstatx(STATX_INO)on Linux, which does not flush
pending writes. A chunk write on NFS now costs about 1 ms of I/O; local
disks gain about 1.4x. Other NFS clients now see the data when the kernel
writes it back, or at the latest when the frame is freed, rather than after
every update; withBLOSC_LOCKINGit is still flushed when the lock is
released. The writable handle comes out of the same budget as the cached
read handle (BLOSC_MAX_CACHED_READERS), so a schunk that has been written
to can hold two file descriptors. Thanks to Fabian Isensee (@FabianIsensee). - Security fix for sparse (directory) frame chunk reading:
sframe_get_chunk()andframe_decompress_chunk()now validate that the
chunk's compressed length (cbytes) recorded in the chunk header does not
exceed the actual size of the chunk file on disk. Previously, a truncated or
crafted chunk file could cause an out-of-bounds heap read during decompression
(#822). Thanks to Mar Dominguez (@mardgez). - Fix filter pipeline buffer cycling and multithreaded DELTA decoding:
- In
pipeline_forward(), cycling buffers now strictly alternates between
thread-local scratch buffers so the caller's read-only input buffer is never
recycled as a destination buffer when 3 or more filters are active.
Previously, the 3rd active filter overwrote block 0 of the input buffer,
corrupting the reference block used by subsequent blocks in the DELTA filter
(#820). Thanks to Gert Hulselmans (@ghuls) for reporting. - In
pipeline_backward(), prevented destination buffer aliasing across
multiple out-of-place backward filters, ensured intermediate buffers are
copied before in-place DELTA decoding, and postponed signalingdref
readiness until block 0 completes its entire backward pipeline.
- In
- Added a new example demonstrating sparse coordinates storage and retrieval
(examples/sparse_coords.c). Thanks to Lance Uymatiao (@lanceu0128).