What's Changed
- Completely moved the scraper out of process into a new background cli process by @iPromKnight in #4
Redirect execution of background services into the main console stream
All memory used during a sync is now returned to the container after a sync completes.
Introduction of new setting: Zilean__Dmm__EnableScraping
Setting this to false will stop the app from scheduling the scraping job entirely.
This means, because the scraper is a separate executable /app/dmmscraper you could setup your own crontab to call that process on demand whenever you want it to run, externally to the api handling it.
Very useful for kubernetes workloads where you may watch this as a Batch Job.
The Dmm scraper uses the same configuration provider as the main app, so any env vars (Specifically Zilean__ElasticSearch__Url being important in this case) will also be passed to the child process.
This refactor will massively reduce the memory footprint of the processing (for you that Mini)
Full Changelog: v1.2.6...v1.2.7