jbruchon/jdupes fdupes-jody-2.0

fdupes-jody 2.0

on GitHub

WARNING: A bug in commit f51a163 causes this release to miss some duplicates. Use v1.2.1 or newer instead!

The hyphenated revisions of fdupes-jody have gotten annoying and the program has been changed significantly since the fork from fdupes-1.51 began. It seems that there will never be a perfect time to release a new major version for continuously improving software, but this fork of fdupes has made so much progress that it's time to bless it with its own version independent from its ancestor.

fdupes-jody 2.0 represents a huge set of performance improvements and feature additions. The -O/--paramorder option finally grants some control over the automatic deletion process of the -dN switch combination which was long overdue. Hard links are supported on Windows natively (though Unicode remains unsupported; perhaps in a future release.) Progress indicators have been enhanced to be much more meaningful. Support for more than 2 million files was added to keep the program future-proof. Internal changes to memory allocation, buffer sizes, data structure order, hashing code, and file comparison behavior have improved performance greatly.

Changes from 1.51-jody5 to 2.0

• Bumped major version to 2.0 due to the number of behavioral changes
  and improvements to the code, plus it looks less messy than the
  hyphenated versioning
• Increased "chunk size" for better performance and to solve the disk
  thrashing problem when checking two large files for a match
• When using the -H option, hard links now automatically match each
  other without performing any file reads
• Changed primary memory allocator to string_alloc by Jody Bruchon
  to improve performance over generic malloc/calloc
• Progress indicator now lists the number of duplicate pairs found in
  addition to the usual file progress count and completion percentage
• Progress is updated more rapidly when full file comparisons happen
  so users are less likely to think fdupes is "frozen"
• Floating point code was made optional and is removed by default
• A comparison script was added to check built fdupes behavior against
  whatever fdupes is currently installed on the system
• Added "undocumented" -Q / --quick option which is not fully safe but can
  be used to drastically reduce run time for large data sets if some risk
  of data loss is acceptable to the user
• Added -O/--paramorder option to sort files by the order their parent
  directory set was specified on the command line first. This makes it
  possible to choose what directories' files get preserved over others
  while using -d and -N together
• The file list loading progress indicator was revamped. Rather than a
  simple "spinning pipe" indicator, it now shows the number of files and
  directories scanned as well as which command line specified set the
  scanning is currently happening in
• fdupes was enhanced to support more than 2 million files total by
  changing from int internal sizes to the maximum supported by the
  platform being compiled for
• Hard link code was modified to be much safer; now any file is only
  permanently deleted after a hard link succeeds
• Hard links on Windows (on supporting filesystems) are now supported
• Hashing code was optimized for a benchmarked 8.4% improvement in file
  processing overhead (and much more under typical real-world conditions)
• Hard linking checks for more error conditions and output is much
  clearer about what action was taken on any given file