Byron/dua-cli dua-core-v4.1.0

dua-core v4.1.0

on GitHub

New Features

• stream roots and support cancellable walks
  Accept newly discovered roots in a running fixed-size worker pool. Keep
  per-root predicates and completion accounting, and prioritize submitted
  roots so existing directory trees cannot starve them.

  • Add RootSender, stream_roots(), and Walk::next_cancellable().
  • Disconnect bounded output before joining workers during teardown.
  • Preserve native enumeration and bounded parallel metadata processing.
  • Cover independent completion, submission priority, input closure,
    full-channel shutdown, and cancellation.

Performance

• parallelize I/O-bound macOS directory scans
  This was an issue I encountered on APFS, a directory created by rustc with
  more than 1m files in it. In that case, bulk reading is slow (while efficient),
  and it turned out to be better to detect this and switch over to stat-based traversal.

  While looking at the filesystem probing code more closely (and how it's not dependent
  on CPU performance by differntiating wall time from kernel time), I basically
  rubber-stamped all the other code. Too much to look at, too foreign by now.
  But it did look cleaned up, so 👍.

  The reported target-directory scan left one worker waiting inside
  getattrlistbulk at about 15% of a CPU core. Native directory collection also
  held an entire parent directory before publishing entries and child jobs.

  Probe at most two initial bulk buffers before publishing them. Compare
  calling-thread user and kernel CPU time with elapsed time, and select the
  existing parallel stat queue when both refills spend more time waiting than
  executing. Reopen through ordinary enumeration only after that decision,
  discarding the unpublished probe to avoid mixed cursors or duplicate entries.
  Keep bulk reads when the probe is CPU-bound. The initial probe deliberately
  does not adapt to later cache changes.

  Share streaming native traversal between ordering modes, preserve parent
  ordering and APFS metadata, and bound queued metadata jobs by processing
  new batches inline when the queue fills.

  Regression coverage checks scale-independent timing decisions, bounded
  probing and metadata backlogs, streamed child jobs, parent ordering, and
  stat/native parity for clones, resource forks, hard links and symlinks.
  The timing test failed against the provisional fixed latency policy; the
  queue-bound test failed without backpressure.

Commit Statistics

• 5 commits contributed to the release over the course of 1 calendar day.
• 5 days passed between releases.
• 2 commits were understood as conventional.
• 0 issues like '(#ID)' were seen in commit messages

Commit Details