hydrusnetwork/hydrus v596

Version 596

on GitHub

misc

• due to an ill-planned parsing update, several downloaders' hash lookups (which allow the client to quickly determine 'already in db'/'previously deleted' sometimes) broke last week. they are fixed today, sorry for the trouble!
• the fps number on the file info line, which was previously rounded always to the nearest integer, is now reported to two sig figs when small. it'll say 1.2fps and 0.50fps
• I hacked in some collapse/expand tech into my static box layout that I use all over the place and tentatively turned it on, and defaulting to collapsed, in the bigger review services sub-panels. the giganto-tall repository panel is now much shorter by default, making the rest of the pages more normal sized on first open. let's see how it goes, and I expect I'll put it elsewhere too and add collapse memory and stuff if that makes sense
• the 'copy service key' on review services panels is now hidden behind advanced mode
• tweaked some layout sizers for some spinboxes (the number controls that have an up/down arrow on the side) and my 'noneable' spinboxes so they aren't so width-hesitant. they were not showing their numbers fully on some styles where the arrows were particularly wide. they mostly size stupidly wide now, but at least that lines up with pretty much everything else so the number of stupid layout problems we are dealing with has reduced by one
• the frame locations list under options->gui has four new buttons to mass-set 'remember size/position' and 'reset last size/position' to all selected
• max implicit system:limit in options->search is raised from 100 thousand to 100 million
• if there is a critical drive problem when adding a file to the file structure, the exact error is now spammed to a popup and log. previously, it was just propagated up to the caller

advanced parsing

• I messed up the 'hex' and 'base64' decode stuff last week. we used to have hex and base64 decode back in python 2 to do some hash conversion stuff, but it was overhauled into the content parser hash type dropdown and the explict conversion was deprecated to a no-op. last week, I foolishly re-used the same ids when I revived the decoding functionality, which caused a bunch of old parsers like gelbooru 0.2.5, e621, 4chan, and likely others, which still had the no-op, to suddenly hex- or base-64-afy their parsed hashes, breaking the parse and lookup
• this week I redefined the hacky enums and generally cleaned this code, and I am deleting all hex and base64 string conversion decodes from all pre-596 parsers. this fixes all the old downloaders by explicitly deleting the no-op so it won't trouble us again
• if you made a string converter in v595 that decodes hex or base64, that encoding step will be deleted, sorry! I have to ask you to re-make it

advanced db maintenance

• added a 'connect.bat' (and .sql file) to the db dir to make it easy to load up the whole database with 'correct' ATTACHED schema names in the sqlite3 terminal
• added database->db maintenance->get tables using definitions, which uses the long-planned database module rewrite maintenance tech ( basically a faux foreign key) to fetch every table that uses hash_ids or tag_ids along with the specific column name that uses the id. this will help with various advanced maintenance jobs where we need to clear off a particular master definition to, as for instance happened this week, reset a super-huge autoincrement value on the master hashes table. this same feature will eventually trim client.master.db by discovering which master definitions are no longer used anywhere (e.g. after PTR delete)

client api

• thanks to the continuing efforts of the user making Ugoira improvements, the Client API's /get_files/render call will now render an Ugoira to apng or animated webp. note the apng conversion appears to take a while, so make sure you try both formats to see what you prefer
• fixed a critical bug in the Client API where if you used the file_id(s) request parameter, and gave novel ids, the database was hitting emergency repair code and filling in the ids with pseudorandom recovery hashes. this wasn't such a huge deal, but if you put a very high number in, the autoincrement hash_id of the hashes table would then move up to there, and if the number was sufficiently high, SQLite would have trouble because of max integer limits and all kinds of stuff blew up. asking about a non-existent file_id will now raise a 404, as originally intended
• refactored the note set/delete calls, which were doing their own thing, to use the unified hash-parsing routine with the new safety code
• if the Client API is ever asked about a hash_id that is negative or over a ~quadrillion (1024^5), it now throws a special error
• as a backup, if the Client DB is ever asked about a novel hash_id that is negative or over a ~quadrillion (1024^5), it now throws a special error rather than trigger the pseudorandom hash recovery code
• the Client API version is now 74

boring duplicates auto-resolution stuff

• fleshed out the duplicates auto-resolution manager and plugged it into the main controller. the mainloop boots and exits now, but it doesn't do anything yet

boring cleanup

• updated the multiple-file warning in the edit file urls dialog
• gave the Client API review services panel a very small user-friendliness pass
• I converted more old multi-column list display/sort generation code from the old bridge to the newer, more efficient separated calls for 10 of the remaining 43 lists to do
• via some beardy-but-I-think-it-is-ok typedefs, all the managers and stuff that take the controller as a param now use the new 'only import when linting' ClientGlobals Controller type, all unified through that one place, and in a way that should be failsafe, making for much better linting in any decent IDE. I didn't want to spam the 'only import when linting' blocks everywhere, so this was the compromise
• deleted the interface modules with the Controller interface gubbins. this was an ok start of an idea, but the new Globals import trick makes it redundant
• pulled and unified a bunch of the common ManagerWithMainLoop code up to the superclass and cleaned up all the different managers a bit
• deleted ClientMaintenance.py, which was an old attempt to unify some global maintenance daemons that never got off the ground and I had honestly forgotten about
• moved responsibility for the remote_thumbnails table to the Client Repositories DB module; it is also now plugged into the newer content type maintenance system
• moved responsibility for the service_info table to the Client Services DB module
• the only CREATE TABLE stuff still in the old Client DB creation method is the version table and the old YAML options structure, so we are essentially all moved to the new modules now
• fixed some bugs/holes in the table definition reporting system after playing with the new table export tool (some bad sibling/parent tables, wrongly reported deferred tables, missing notes_map and url_map due to a bad content type def, and the primary master definition tables, which I decided to include). I'm sure there are some more out there, but we are moving forward on a long-term job here and it seems to work