Cyberdrop-DL/cyberdrop-dl 10.0.0

Cyberdrop-DL V10.0 Start Files

on GitHub

⚠️IMPORTANT

This version is incompatible with any previous release. Config file options need to be manually migrated by user.

CDL will refuse to start if the current database schema is older than v9.15.0. Run cyberdrop-dl database transfer to upgrade older databases

Added

• New --cookies option to read cookies from a file/folder
• New --hashes option to control which hashes CDL computes for new downloads
• New --database-file option
• New --cache-file option
• New download CLI command (replaces --download argument)
• New retry CLI command (replaces --retry-failed and --retry-all arguments)
• New config CLI command
• New cache CLI command
• New hash CLI command
• New report CLI command

Changed

• The config file format has changed. All configs (Auth, Global and Setttings) are now a single file. Several options have new names, new defaults and new groups
• A default config file will no longer be created. You can manually create a default one from the Edit config option on the main menu or running cyberdrop-dl config new
• On Windows, all application files are now stored at %AppData%/cyberdrop-dl
• On Unix systems, application files are stored according to the XDG Base Directory Specification
• Validate config in strict mode. If a config has an unknown entry, CDL will refuse to run instead of ignoring it
• Apprise URLs are now part of the main config file instead of a dedicated apprise.txt file
• Refuse to start if the current database schema is older than v9.15.0
• Cookies will not be automatically loaded from AppData/cookies. Path to cookies needs to be specified with --cookies
• Retry options will no longer use the original path of the first download attempt. Download path will be generated based on current config options (like normal downloads)
• Compute xxh128, md5 and sha256 hashes by default
• --deep-scrape will no longer reset after a single run
• --input-file is now a CLI only arg
• If supplied, --input-file needs to be a valid file that already exists
• Refuse to run if both URLs and --input-file are passed as arguments
• Integers in config options that expect a timedelta value will be parsed as seconds instead of days
• Detect and report BasedFlare anti-bot protection
• Limit max queued downloads of a single site to the config concurrency limit x10 (capped at 50). All scraping for a site will be paused if its queue is full
• Always remove generated id from filenames (Cyberdrop)
• Revert mandatory Content-Length header introduced in v9.15.0

The following options, which were CLI only arguments, now have dedicated config entries:

• --ui (entry: ui.mode)
• --portrait (entry: ui.portrait)
• --stats

Several config options have new names:

• --disable-crawlers -> --crawlers.disabled
• --disable-file-timestamps -> --mtime
• --maximum-audio-duration -> --audio.duration.max
• --maximum-image-size -> --image.size.max
• --maximum-number-of-children -> --max-children
• --maximum-other-size -> --non-media.size.max
• --maximum-thread-depth -> --max-thread-depth
• --maximum-thread-folder-depth -> --max-thread-folder-depth
• --maximum-video-duration -> --video.duration.max
• --maximum-video-size -> --video.size.max
• --minimum-audio-duration -> --audio.duration.min
• --minimum-image-size -> --image.size.min
• --minimum-other-size -> --non-media.size.min
• --minimum-video-duration -> --video.duration.min
• --minimum-video-size -> --video.size.min
• --required-free-space -> --min-free-space
• --print-stats -> --stats , --no-stats
• --send-deleted-to-trash -> --hashing.dedupe.use-trash-bin

Jdownloader

• --jdownloader-autostart -> --jdownloader.autostart
• --jdownloader-download-dir -> --jdownloader.download-folder
• --jdownloader-whitelist -> --jdownloader.whitelist
• --send-unsupported-to-jdownloader -> --jdownloader / --no-jdownloader

Sorting

• --sort-downloads -> --sort / --no-sort
• --scan-folder -> --sort.input-folder
• --sort-folder -> --sort.output-folder
• --sort-incrementer-format -> --sort.formats.incrementer
• --sorted-audio -> --sort.formats.audio
• --sorted-image -> --sort.formats.image
• --sorted-other -> --sort.formats.non-media
• --sorted-video -> --sort.formats.video

Ignore options (filters)

The behavior of --filename-regex has been reversed (files that DO NOT match the regex will be skipped)
The behavior of --before and --after has been reversed and the --exclude prefix removed (they now include instead of excluding files)

• --exclude-audio -> --no-audio
• --exclude-images -> --no-images
• --exclude-other -> --no-non-media
• --exclude-videos -> --no-videos
• --exclude-before -> --before
• --exclude-after -> --after
• --exclude-files-with-no-extension -> --allow-files-with-no-extension
• --filename-regex-filter -> --filename-regex
• --download-tiktok-src-quality-videos -> - --crawlers.tiktok.original

Logs

• --log-level -> --logs.level
• --console-log-level -> --logs.console-level
• --log-folder -> --logs.folder
• --logs-expire-after -> --logs.expire-after
• --main-log -> --logs.files.main with alias --log-file
• --download-error-urls -> --logs.files.download-errors
• --rotate-logs -> --logs-rotate
• --scrape-error-urls -> --logs.files.scrape-errors
• --unsupported-urls -> --logs.files.unsupported

DownloadOptions

• --block-download-sub-folders -> --subfolders, --no-subfolders
• --include-album-id-in-folder-name -> --subfolders.include.album-id
• --include-thread-id-in-folder-name -> --subfolders.include.thread-id
• --remove-domains-from-folder-names -> --subfolders.include.domain
• --separate-posts-format -> --subfolders.separate-posts.format
• --separate-posts -> --subfolders.separate-posts
• --skip-download-mark-completed -> --skip-and-mark-completed
• --max-simultaneous-downloads -> --downloads
• --max-simultaneous-downloads-per-domain -> --downloads.per-domain
• --slow-download-speed -> --slow-speed
• --download-attempts -> --attempts
• --download-delay -> --delay
• --download-speed-limit -> --speed-limit

Generic crawlers

• --wordpress-media -> --crawlers.generic.wordpress-media
• --wordpress-html -> --crawlers.generic.wordpress-html
• --discourse -> --crawlers.generic.discourse
• --chevereto -> --crawlers.generic.chevereto

Removed

• Support for python 3.11
• --log-level and --console-log-level (--logs.level and --logs.console-level) no longer accept integers. Only log level names as valid, ex: INFO, DEBUG, WARNING
• Posts filtering by URL params (Wordpress)
• Kemono support
• Coomer support
• Coomerfans support
• Auto cookie extraction support

Several config options have been removed:

• --appdata-folder
• --add-md5-hash
• --add-sha256-hash
• --auto-import
• --browser
• --completed-after
• --completed-before
• --disable-download-attempt-limit
• --download-tiktok-audios
• --last-forum-post
• --max-items-retry
• --remove-generated-id-from-filename
• --retry-all
• --retry-failed
• --retry-maintenance
• --save-pages-html
• --scrape-single-forum-post
• --sites
• --skip-check-for-partial-files
• --update-last-forum-post
• --ignore-coomer-post-content
• --ignore-coomer-ads

The following authentication entries has been removed:

• Imgur.client_id
• Kemono.session
• Coomer.session

Fixed

• Do not skip initialization segments (HLS)
• 403 errors on embeded videos (twing)
• Switch to v2 API (Filester)