Byron/gitoxide gix-pack-v0.36.0

gix-pack v0.36.0

on GitHub

New Features

• improve performance by reducing the trashing of zlib::Inflate.
  Previously, for every zlib inflate operation, we would allocate memory
  due to the creation of a decompressor.

  This is now avoided by reusing thread-local copies of such a decompressor,
  which is reused and reset instead.

• improve performance by avoiding zeroying buffers.
  Previously we would use resize(new_len, 0) to resize buffers, even though
  these values would then be overwritten (or the buffer isn't available).

  Now we use set_len(new_len) after calling reserve to do the same, but safe
  a memset.

Bug Fixes

• static linked list delta cache with memory cap
  Previously, the 64 slot big LRU cache for pack deltas didn't
  use any memory limit which could lead to memory exhaustion in the
  face of untypical, large objects.

  Now we add a generous default limit to do better in such situations.
  It's worth noting though that that even without any cache, the working
  set of buffers to do delta resolution takes considerable memory, despite
  trying to keep it minimal.

  Note that for bigger objects, the cache is now not used at all, which probably
  leads to terrible performance as not even the base object can be cached.

• check for interrupt more often
  Previously when traversing a pack it could appear to hang as checks
  were only performed on chunk or base (of a delta-tree) level.

  Now interrupt checks are performed more often to stop all work much quicker.

• memory capped hashmap as pack delta cache won't trash memory as much.
  Previously it would take a buffer from the free-list, copy data into it, and
  when exceeding the capacity loose it entirely.
  Now the freelist is handled correctly.

New Features (BREAKING)

• index-backed tree traversal with a form of work-stealing.
  When delta-trees are unbalanced, in pathological cases it's possible that that one thread
  ends up with more than half of the work. In this case it's required that it manages to
  spawn its own threads to parallelize the work it has.

• make current thread-count accessible in slice-workers.
  Threads started for working on an entry in a slice can now see the amount
  of threads left for use (and manipulate that variable) which effectively
  allows them to implement their own parallelization on top of the current one.

  This is useful if there is there is very imbalanced work within the slice itself.

  While at it, we also make consumer functions mutable as they exsit per thread.

Commit Statistics

• 28 commits contributed to the release over the course of 25 calendar days.
• 40 days passed between releases.
• 7 commits were understood as conventional.
• 1 unique issue was worked on: #851

Thanks Clippy

Clippy helped 1 time to make code idiomatic.

Commit Details