treeverse/dvc 3.54.0

on GitHub

What's Changed

⚡ Optimizations

• Improves performance of building objects, especially for directories with large no. of files by as much as 70% - 90%. This affects most of the commands. For example,

  • checkout dropped by as much as 80% (see #10491).
  • dvc add performance improved by more than 70%.

  See https://bench.dvc.org/latest.html for more.

• Avoids relinking on dvc add and dvc commit if possible (iterative/dvc-data#548)

• Improves performance for dvc diff rename detection (iterative/dvc-data#550).

• Reintroduces support for hashing files concurrently. This is only done for files larger than 1MB to avoid Python's thread overhead.

• Avoids stat calls (or, info() calls) during import-url and import. This is now done as part of walk().