github jbruchon/jdupes fdupes-jody-2.0
fdupes-jody 2.0

WARNING: A bug in commit f51a16314be9ab6d654330055fade57ec4998bea causes this release to miss some duplicates. Use v1.2.1 or newer instead!

The hyphenated revisions of fdupes-jody have gotten annoying and the program has been changed significantly since the fork from fdupes-1.51 began. It seems that there will never be a perfect time to release a new major version for continuously improving software, but this fork of fdupes has made so much progress that it's time to bless it with its own version independent from its ancestor.

fdupes-jody 2.0 represents a huge set of performance improvements and feature additions. The -O/--paramorder option finally grants some control over the automatic deletion process of the -dN switch combination which was long overdue. Hard links are supported on Windows natively (though Unicode remains unsupported; perhaps in a future release.) Progress indicators have been enhanced to be much more meaningful. Support for more than 2 million files was added to keep the program future-proof. Internal changes to memory allocation, buffer sizes, data structure order, hashing code, and file comparison behavior have improved performance greatly.

Changes from 1.51-jody5 to 2.0

  • Bumped major version to 2.0 due to the number of behavioral changes
    and improvements to the code, plus it looks less messy than the
    hyphenated versioning
  • Increased "chunk size" for better performance and to solve the disk
    thrashing problem when checking two large files for a match
  • When using the -H option, hard links now automatically match each
    other without performing any file reads
  • Changed primary memory allocator to string_alloc by Jody Bruchon
    to improve performance over generic malloc/calloc
  • Progress indicator now lists the number of duplicate pairs found in
    addition to the usual file progress count and completion percentage
  • Progress is updated more rapidly when full file comparisons happen
    so users are less likely to think fdupes is "frozen"
  • Floating point code was made optional and is removed by default
  • A comparison script was added to check built fdupes behavior against
    whatever fdupes is currently installed on the system
  • Added "undocumented" -Q / --quick option which is not fully safe but can
    be used to drastically reduce run time for large data sets if some risk
    of data loss is acceptable to the user
  • Added -O/--paramorder option to sort files by the order their parent
    directory set was specified on the command line first. This makes it
    possible to choose what directories' files get preserved over others
    while using -d and -N together
  • The file list loading progress indicator was revamped. Rather than a
    simple "spinning pipe" indicator, it now shows the number of files and
    directories scanned as well as which command line specified set the
    scanning is currently happening in
  • fdupes was enhanced to support more than 2 million files total by
    changing from int internal sizes to the maximum supported by the
    platform being compiled for
  • Hard link code was modified to be much safer; now any file is only
    permanently deleted after a hard link succeeds
  • Hard links on Windows (on supporting filesystems) are now supported
  • Hashing code was optimized for a benchmarked 8.4% improvement in file
    processing overhead (and much more under typical real-world conditions)
  • Hard linking checks for more error conditions and output is much
    clearer about what action was taken on any given file
latest releases: v1.20.0, v1.19.2, v1.19.1...
5 years ago