github Blosc/c-blosc2 v3.4.1
Release 3.4.1

5 hours ago

Changes from 3.4.0 to 3.4.1

This is a bugfix release that brings substantial performance improvements for
in-place writes to on-disk contiguous frames (especially on NFS), security and
robustness fixes for sparse frames, fixes for filter pipelines with 3 or more
filters alongside multithreaded DELTA decoding, and an example for sparse
coordinate handling.

  • Faster in-place writes to on-disk contiguous frames, especially on NFS:
    appending, inserting, updating or deleting a chunk (and so b2nd
    set_slice) opened and closed the frame file four times and truncated it on
    every trailer update. On NFS each close is a synchronous flush to the
    server, which made such writes cost about 20 ms per chunk whatever its size.
    With the default filesystem backend on POSIX, a frame now keeps one writable
    handle open, truncates the file only when the frame got shorter, and checks
    for a replaced file with statx(STATX_INO) on Linux, which does not flush
    pending writes. A chunk write on NFS now costs about 1 ms of I/O; local
    disks gain about 1.4x. Other NFS clients now see the data when the kernel
    writes it back, or at the latest when the frame is freed, rather than after
    every update; with BLOSC_LOCKING it is still flushed when the lock is
    released. The writable handle comes out of the same budget as the cached
    read handle (BLOSC_MAX_CACHED_READERS), so a schunk that has been written
    to can hold two file descriptors. Thanks to Fabian Isensee (@FabianIsensee).
  • Security fix for sparse (directory) frame chunk reading:
    sframe_get_chunk() and frame_decompress_chunk() now validate that the
    chunk's compressed length (cbytes) recorded in the chunk header does not
    exceed the actual size of the chunk file on disk. Previously, a truncated or
    crafted chunk file could cause an out-of-bounds heap read during decompression
    (#822). Thanks to Mar Dominguez (@mardgez).
  • Fix filter pipeline buffer cycling and multithreaded DELTA decoding:
    • In pipeline_forward(), cycling buffers now strictly alternates between
      thread-local scratch buffers so the caller's read-only input buffer is never
      recycled as a destination buffer when 3 or more filters are active.
      Previously, the 3rd active filter overwrote block 0 of the input buffer,
      corrupting the reference block used by subsequent blocks in the DELTA filter
      (#820). Thanks to Gert Hulselmans (@ghuls) for reporting.
    • In pipeline_backward(), prevented destination buffer aliasing across
      multiple out-of-place backward filters, ensured intermediate buffers are
      copied before in-place DELTA decoding, and postponed signaling dref
      readiness until block 0 completes its entire backward pipeline.
  • Added a new example demonstrating sparse coordinates storage and retrieval
    (examples/sparse_coords.c). Thanks to Lance Uymatiao (@lanceu0128).

Don't miss a new c-blosc2 release

NewReleases is sending notifications on new releases.