GitoxideLabs/gitoxide gix-pack-v0.74.0

gix-pack v0.74.0

on GitHub

Refactor (BREAKING)

• remove object_hash from Options
  This forces callers to explicitly choose a hash, eliminating the risk
  that Sha1 is implicitly chosen for them in SHA-1/SHA-256 builds
  through Default::default() without them being aware of it or noticing.
  This comes at the cost of callers having to always pass the hash, but
  that seems like a reasonable trade-off.

New Features

• expose index-less parallel pack traversal (cache Tree, Context, Options)
  Re-export cache::delta::{Tree, traverse::{Context, Options}} at cache so
  external callers can build a delta Tree directly from a pack header scan and
  resolve all of its objects in parallel via Tree::traverse -- without a
  pre-built .idx. This is the index-less companion to the already-public,
  idx-verified index::File::traverse_with_index.

  Minimal surface by design: the delta module stays pub(crate), so only the
  three already-documented items a caller names become public -- pub mod delta
  would instead expose the module's internal error types and constructors and fail
  #![deny(missing_docs)]. Tree::from_offsets_in_pack (already public) builds the
  tree straight from the pack; with_capacity/add_root/add_child remain for
  callers who assemble it by ascending pack offset themselves.

  Additive and non-breaking; cargo doc is clean under -D warnings, and a
  doc-test anchors the index-less path (Tree::from_offsets_in_pack). Shape
  approved in #2922.

Bug Fixes

• validate delta base sizes during entry decoding

  Reject delta instruction streams whose declared base size differs from the
  resolved base object. The regression demonstrates that a REF_DELTA could
  previously reconstruct successfully despite lying about its base size.

  Git reference: patch_delta() compares the decoded source size with src_size
  before applying instructions, observed at cf5497b14c5a.

• resolve delta trees lazily in a lock-free pool

  --- Summary of findings

  The lazy, stealable resolver fixes the pathological phpstan pack without
  regressing the Linux best case. At 16 threads, phpstan falls from the
  98.38-second baseline resolver time to 22.70 seconds, while the Linux
  best-case resolver improves slightly from 11.81 to 11.33 seconds. Charged
  peak memory on phpstan falls from 13.14 GB to 2.16 GB, and the Linux best
  case falls from 1.96 GB to 1.76 GB.

  Across the measured packs, gix resolver speedup at 16 threads ranges from
  10.80x to 12.48x over its new serial path. On phpstan, gix is 1.07x faster
  than Git at one thread and 2.31x faster at 16 threads. On the identical
  10.9M-object Linux payload, gix's SHA-256 resolver is 2.72x to 2.99x faster
  than its SHA-1 resolver, confirming that the slow SHA-1 implementation
  dominates much of the remaining absolute cost.

  The Git SHA-256 comparison was initially unfair: the local Git build used
  the portable SHA256_BLK backend while RustCrypto selected ARMv8 SHA-2
  instructions. Rebuilding Git with OpenSSL reduced its eight-thread wall
  time from 144.15 to 40.45 seconds and user CPU time from 471.94 to
  99.81 seconds. All final Git SHA-256 comparisons use that faster backend.

  --- Pathological phpstan pack

  The pack contains 100.7k objects but expands to 174.6 GB because one root
  ends in a deep, expensive delta tree. The previous resolver takes
  98.53 seconds wall clock and has a 13.14 GB peak memory footprint.

  The new resolver takes 283.41, 71.94, 36.42, and 22.83 seconds wall clock
  at 1, 4, 8, and 16 threads. Its resolver speedups are 3.94x, 7.81x, and
  12.48x relative to one thread, remaining nearly linear through eight
  threads. Peak memory footprint is 0.68, 0.75, 1.17, and 2.16 GB
  respectively. Thus the fastest run is 4.31x faster than the old wall time
  while using 84% less charged peak memory.

  Git was measured at 1, 2, 4, 8, and 16 threads while keeping its aggregate
  delta-base cache allowance at least as large as the default eight-thread
  allowance of 8 times 96 MiB. Git takes 304.51, 168.06, 98.80, 65.41, and
  52.79 seconds, for 5.77x one-to-sixteen-thread scaling. At equal thread
  counts, gix is 1.07x, 1.37x, 1.80x, and 2.31x faster at 1, 4, 8, and
  16 threads.

  At the nearest wall-time operating points, gix at four threads takes
  71.94 seconds with a 0.75 GB peak footprint, while Git at eight threads
  takes 65.41 seconds with a 1.90 GB peak footprint. gix therefore uses
  61% less charged working memory near Git-matching throughput.

  --- Linux best-case pack

  The 7.6M-object fixture expands to 95.6 GB and guards the already-friendly
  case. The old implementation takes 13.68 seconds wall clock and
  11.81 seconds in the resolver. The new implementation takes 137.48,
  36.75, 19.81, and 13.16 seconds wall clock at 1, 4, 8, and 16 threads;
  resolver scaling reaches 11.97x at 16 threads.

  The 16-thread result is slightly faster than the old implementation, with
  11.33 seconds in the resolver, and lowers peak footprint from 1.96 to
  1.76 GB. Peak footprint remains effectively constant from one through
  16 threads. Git takes 156.65, 60.48, 45.48, and 42.97 seconds at the same
  thread counts, so gix's advantage grows from 1.14x to 3.27x.

  --- SHA-1 and SHA-256 sibling packs

  The sibling packs contain 10.9M objects, expand to 146.7 GB, and preserve
  the same compressed payload and delta topology. This isolates hashing from
  work distribution.

  For SHA-1, gix takes 212.57, 56.58, 30.60, and 20.74 seconds wall clock at
  1, 4, 8, and 16 threads. Resolver scaling reaches 11.36x at 16 threads.
  The default eight-thread Git run takes 125.11 seconds, while gix takes
  30.60 seconds at eight threads and 20.74 seconds at 16.

  For SHA-256, gix takes 72.47, 21.83, 12.58, and 8.67 seconds wall clock.
  Resolver scaling reaches 10.80x at 16 threads. Against OpenSSL-backed Git,
  gix is 1.21x, 2.02x, 3.22x, and 5.61x faster at equal thread counts.
  Git improves from 87.85 seconds at one thread to 40.45 seconds at eight,
  then regresses to 48.68 seconds at 16 with 204.81 seconds of system CPU.

  --- Hasher and scheduler interpretation

  On identical pack data, gix SHA-256 resolver time is 70.31 seconds at one
  thread versus 210.35 seconds for SHA-1, and 6.51 seconds at 16 threads
  versus 18.51 seconds for SHA-1. SHA-256 is therefore 2.99x faster
  serially and 2.84x faster at 16 threads, with a 2.72x to 2.99x advantage
  throughout the measured range.

  The similar 10.80x to 12.48x gix scaling across pathological, best-case,
  SHA-1, and SHA-256 packs argues against lock contention being the main
  high-thread limitation. The remaining flattening is consistent with
  finite parallel work, scheduling overhead, memory bandwidth, and hashing
  cost.

  --- Memory interpretation

  The resolver no longer materializes the unresolved internal-node frontier.
  It resolves a child only when a worker starts it, shares immutable bases
  between sibling tasks, and recycles the final base reference. A linear
  chain therefore needs roughly two object buffers per active worker instead
  of retaining every intermediate base.

  macOS max RSS includes clean file-backed pages. gix maps the pack, so its
  reported RSS can include most of a multi-gigabyte pack even when those
  pages are reclaimable; Git reads through bounded pread buffers. Peak memory
  footprint better represents charged working memory here. On phpstan, gix
  max RSS grows from 7.20 to 8.90 GB across 1 to 16 threads, but charged
  footprint grows from only 0.68 to 2.16 GB. On the Linux best case,
  footprint stays at 1.76 GB across the same range.

  --- Implementation

  Use lock-free local deques and a shared root injector so idle workers can
  steal branches from the last expensive root without every worker hoarding
  roots. Enable this resolver through the gix parallel feature.

  Regression tests prove that internal siblings are not all materialized
  before descent and that two workers can concurrently resolve children from
  one remaining root. The scheduler follows the proven Git index-pack
  principles of bounded live delta bases and keeping independent delta work
  available, informed by builtin/index-pack.c at cf5497b14c.

• resolve in-pack ref deltas while indexing
  Valid packs may encode a base inside the same pack with REF_DELTA and may
  place the delta before that base. The streaming lookup previously treated an
  object-database miss as fatal, while index construction rejected every remaining
  ref delta.

  Keep lookup misses in the stream and park ref-delta children by base object ID
  until traversal resolves the matching object. This preserves external thin-pack
  injection and reuses the existing delta tree for forward and chained references.

  Git reference: a23bace963d508bd96983cc637131392d3face18, builtin/index-pack.c and t/t5300-pack-object.sh.

Commit Statistics

• 19 commits contributed to the release over the course of 30 calendar days.
• 30 days passed between releases.
• 5 commits were understood as conventional.
• 3 unique issues were worked on: #1025, #2424, #2868

Commit Details