github datawhores/OF-Scraper 3.11.dev1

latest releases: 2b538b9, 1510c51, a41d5b1...
4 months ago

Optimized Thread and Download Settings

With this release, you no longer need to configure numerous threads and downloads for optimal download performance. The changes in 3.10.x, which now manage download progress based on disk size instead of multiprocess.connect, combined with the new buffered stdout for the rich log handler in this release, have significantly improved the script's download performance.

Recommended Settings:

Threads: 0-3
Max Downloads: Approximately 15-20 across all threads

Changes

Performance

  • add stdout buffer for logs
  • remove queues for many thread instead

metadata

  • change default metadata path to support model name changes

logging

  • add various customhandlers

download

  • move some functions like size_checker into the own file
  • add functions for handling data caching and retrial for download resumption
  • bug fixes
  • improve overall speed monitor by also adding pids after making processes in batch downloaders

other changes

  • re-add removing areas with "-"
  • fix performance config menu
  • fix issue with starting progress_bar in download activity

Don't miss a new OF-Scraper release

NewReleases is sending notifications on new releases.