hydrusnetwork/hydrus v615

Version 615

on GitHub

duplicates auto-resolution brief

• the system is ready for advanced users to try! there is one simple static rule available. check out the help https://hydrusnetwork.github.io/hydrus/advanced_duplicates_auto_resolution.html , go into the UI, and try out the suggested pixel-perfect jpg & png rule. I want to know if--
  • it all makes sense to you
  • where it runs smooth
  • where it runs garbage
  • any errors?
  • out of interest, what do you get? Of ~800k potential pairs, I had ~6,000 jpg/png pixel pairs, resulting in ~4,700 total actual processed pairs (numbers shrink because multiple good files can share the same bad file). speed was bleh in the preview viewer (about 30 seconds to load the preview numbers) but nice when doing work: only a second or two to save the rule and then ~20k files/s in the search stage and 10 files/s in the processing stage. about 7 mins to ditch 7.5GB of Clipboard.png, hooray

duplicates auto-resolution

• fleshed out the help page here: https://hydrusnetwork.github.io/hydrus/advanced_duplicates_auto_resolution.html it is linked in the main help directory, too, under 'advanced'
• made it so you can double-click or enter/return any pass/fail test row in the auto-resolution preview panel to open that pair in a normal media viewer
• added work on these rules during idle/normal time to the cog button on the auto-resolution sidebar tab
• wrote up 'work hard' functionality and wired up the button--however I think I might remove this, since the system works well enough on its own. let me know what you think
• reworked the preview panel to have a two-stage search. it spends a whack of CPU time fetching the total count of the search, and then the sample part works faster afterwards and can be hit over and over. it still sucks and I have another idea to speed things up, we'll see
• updated the main duplicates auto-resolution maintenance routine to do the search step in blocks of 8192 rather than trying to do everything at once as soon as a rule is added. I hesitated doing this earlier, since the large one-time init search is more efficient, but we are going to be doing incremental searches in normal operation, so making small search chunks work well is not optional. I think the biggest potential weakness of this whole system is going to be incremental searches on new potential pairs when the potential duplicate pairs search of the auto-resolution rule has significant per-search overhead, but which searches will those be? sounds like we'll find out in the coming weeks. thankfully, jpg/png pixel dupes seems to scale excellently, so we are good for today

misc auto-resolution stuff

• reduced flicker on the main 'auto-resolution' review panel as it works a real rule
• fixed a 'hey I guess the auto-resolution preview panel is 1200px wide now' bug on certain rule-fetch validation errors
• renamed 'A will pass' comparison UI labels to 'A will match'
• the preview panel lists now scroll one row per scroll (it was doing 3 before, lol, and the default height is ~2 rows)
• the auto-resolution preview panel (particularly the 'will be actioned' list, with its third column) should size itself better if you have large thumbnails
• if the auto-resolution preview panel had results but the search changes and now we have no resullts, the test result lists will now appropriaterly clear themselves
• the preview result lists now generally clear nicely before a new search is started, so if something actually goes wrong, you don't have old test results hanging around
• fixed incremental potential duplicates search on non-pixel-duplicate searches

unit tests

• wrote unit tests for the MetadataConditional
• wrote unit tests for all the predicate types that can work in the MetadataConditional
• wrote unit tests for the Selector
• wrote unit tests for the one-file Comparator
• wrote unit tests for auto-resolution rules: for editing at the db level; for syncing to existing, new, dissolved, and resolved potential duplicate pairs; performing search; performing resolution

misc

• fixed a typo bug that broke the maintenance job that resets all potential duplicate pair search. it was related to the recent auto-resolution integration
• fixed a typo bug that was stopping the 'review accounts' repository admin panel from opening
• added an FAQ about the extensionless files that appear in your file storage if you sync with a repository
• added LINUX DEBUG: Do not allow combined setGeometry on mpv window to options->media playback. if you have crashes on X11 in v614 when zooming mpv windows, give it a go and let me know what happens
• thanks to a user, added a note to the install help that if you are on Linux & Wayland, adding the WAYLAND_DISPLAY environment variable, which forces the program to run in XWayland, seems to relieve many UI bugs (issue #1695)

boring cleanup

• cleared out a surplus entry in db init related to duplicates auto-resolution
• tweaked the 'don't show hovers/hide cursor if a dialog is open' tests in the media viewer, ignoring them if we are the child of one lol