v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked. See the updated docsite for more details on the new search syntax: https://bitmagnet.io/tutorials/text-search.html
Upgrading
The v0.5.0 upgrade process is fully automatic (unless you've used the /import endpoint, read below). All torrents must be reprocessed before they will work with text search. Just wait for the background job queue to work through them, then enjoy better and faster search!
If you have a large library the queue may take several hours, or even a day or more to work through. To check on progress, visit the /metrics endpoint and note metrics that contain process_torrent. The key ones being:
asynq_tasks_enqueued_total{queue="process_torrent",state="completed"}
asynq_tasks_enqueued_total{queue="process_torrent",state="pending"}
asynq_tasks_failed_total{queue="process_torrent"}
Until the queue has worked down you won't see any newly crawled torrents (though the crawler will be running!)
When you start the container for the first time you should see a progress bar for queueing the reprocess - please don't interrupt the process until you see the completed progress bar. If the process is interrupted you may have to requeue manually using the command bitmagnet reprocess --classifyMode=skip.
Note: I'm aware there are still some performance issues that need addressing for aggregations. These will be addressed separately - this release focuses on text search.
If you've used the /import endpoint, you will lose any classification hints that were part of the import. That is, if torrents were assigned to particular content types, video resolutions etc, this information will be lost and the classifier will attempt to classify from scratch without these hints. To recover the hints, the original import data must be reimported - it was necessary to separate these hints from the final classification result as the 2 things were previously conflated, making it impossible to separate out the data as part of this upgrade.
IMPORTANT: the format for the /import endpoint has changed slightly, please check the updated documentation at https://bitmagnet.io/tutorials/import.html (note there is a slight change to the jq query in the final code snippet).
What's Changed
- Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #91
- Lazy instantiation for services with side effects by @mgdigital in #98
- Search query string rework by @mgdigital in #96
- Cache tweaks by @mgdigital in #99
- Add reclassify command by @mgdigital in #100
- Disable Gorm easer plugin which sometimes causes zero results by @mgdigital in #101
- Bump follow-redirects from 1.15.2 to 1.15.4 in /webui by @dependabot in #102
- Replace problematic chars in tsvector by @mgdigital in #103
- Fix search ordering by @mgdigital in #104
- Fix index out of range by @mgdigital in #105
- Refactor torrent processing by @mgdigital in #106
- Add 0.5.0 upgrade hook to queue reprocessing of all torrents by @mgdigital in #107
- Add missing criteria for collections by @mgdigital in #110
- Video classifier fixes by @mgdigital in #111
- Upgrade dependencies by @mgdigital in #112
- Reprocess tweaks by @mgdigital in #116
- Torrent processor optimisations by @mgdigital in #119
- Torznab: disable total count; fix index out of range by @mgdigital in #120
- v0.5.0 documentation updates by @mgdigital in #108
- Reduce crawler persistence batch size by @mgdigital in #121
Full Changelog: v0.4.1...v0.5.0