gallery-dl 1.30.0

on Python PyPI

Changes

• raise minimum supported Python version to 3.8 (#7671)
• update extractor names (#7746 #7612)

  • Old	New
    kemonoparty	kemono
    coomerparty	coomer
    koharu	schalenetwork
    naver	naver-blog
    chzzk	naver-chzzk
    naverwebtoon	naver-webtoon
    pixiv:novel	pixiv-novel:novel
    pixiv:novel-user	pixiv-novel:user
    pixiv:novel-series	pixiv-novel:series
    pixiv:novel-bookmark	pixiv-novel:bookmark

  • config settings will automatically use the old values
  • target directories using {category} will use the new category names by default
  • use --compat or "category-map": "compat" to restore old category names
• include exit status bitmasks of NotFoundError (8) and NoExtractorError (64)
  into general HttpError (4) and InputError (32) respectively

Extractors

Additions

• [civitai] add search-images and posts extractors (#7609)
• [comick] add support (#1825 #6782)
• [dankefuerslesen] add support (#7669)
• [dynastyscans] add anthology extractor (#7627)
• [girlsreleased] add support (#6200)
• [girlswithmuscle] add support (#4493 #6016)
• [iwara] add support (#2652 #5840 #7785)
• [kemono] add artists extractor (#7582)
• [misskey] add avatar, background, and info extractors (#5347)
• [motherless] add group extractor (#7774 #7787)
• [naver-chzzk] add comment and community extractors (#7735 #7741)
• [nudostar] add support (#5735 #6556)
• [rawkuma] add support (#4571)
• [redbust] add support (#6759 #6918 #7043)

Fixes

• [4archive] fix thread extractor
• [arcalive] fix download URLs (#7678)
• [arcalive] replace ac-p.namu subdomains with ac-o.namu (#7556)
• [archivedmoe] fix redirection issue (#7652 #7653 #7664)
• [aryion] fix favorite extractor (#7775)
• [batoto] fix downloading manga with alerts/notices (#7657)
• [behance] fix 403 Forbidden errors (#7710)
• [bunkr] fix file downloads (#7747)
• [civitai] fix & update search extractor (#7609)
• [danbooru] fix Ugoira conversions for posts without ZIP:ZipFileName (#7630)
• [deviantart:tag] fix username (#7587)
• [deviantart:tiptap] fix TypeError when textAlign is null (#7639)
• [directlink] fix config lookups by subcategory (#6582)
• [discord] support forwarded messages & handle missing threads (#7706 #7722)
• [furaffinity] fix submissions results (#7759)
• [hitomi] fix negative tag searches (#7694)
• [kemono] fix tagged creator posts
• [mangadex:list] fix config lookups for list-feed subcategory
• [nijie] fix file extraction (#7624)
• [paheal] fix 404 Not Found error for tags with URL encoded characters (#7642)
• [patreon] send Referer header when downloading .m3u8 videos (#7571)
• [patreon] fix campaign_id extraction from Next.js 13 creator pages (#7773)
• [readcomiconline] fix extraction (#7606 #7789)
• [reddit] fix archive IDs of fallback files (#7760)
• [rule34] fix file downloads (#7697)
• [sankaku] fix extracting extended tag categories (#7744)
• [sexcom] prevent .css file downloads (#7632)
• [skeb] fix KeyError - 'frame_rate' (#7798)
• [tiktok] handle exceptions when extracting avatars (#7682)
• [vsco] fix JSON returned by VSCO (#7821)
• [warosu] HTML attribute fix (#7676 #7677 #7777)

Improvements

• [artstation] support downloading .mview files (#7812)
• [civitai] support "My Reactions" results for videos (#7608)
• [e621] support e621.cc/posts URLs (#6809)
• [erome] restructure extractor hierarchy (#7804)
• [everia] prevent redirect when fetching post pages
• [exhentai] ensure file signature bytes aren`t all zero (#4902)
• [exhentai] implement "source": "metadata" (#4902)
• [fanbox] return fileMap files in order (#2718)
• [gelbooru] improve error message for 401 Unauthorized responses (#7674)
• [imagevenue] detect 404 image files (#7570)
• [instagram] provide more descriptive URLs for video_dash_manifest videos (#7631)
• [pinterest] support pin.it redirects to board (#7805)
• [pinterest] match board URLs with query strings (#7805)
• [rule34us] prioritize video.rule34.us for video downloads (#6582)
• [rule34xyz] implement login with username & password (#7736)
• [sankaku] allow passing cookies (#7333)
• [sexcom] support /pics/ URLs (#7611)
• [tiktok] detect login page redirects (#7716)
• [vk] detect challenge page redirects (#7650)
• [vk] prevent 404 Not Found errors for file downloads
• [vk] add continuation message (#7650)
• [warosu] detect missing images by checking hostname (#7698 #7699)
• [ytdl] set domain as subcategory when using Generic extractor (#6582)

Metadata

• [civitai] always provide file[…] metadata (#7548)
• [everia] improve filename by unquoting URLs (#7620)
• [fanbox] extract archives metadata (#7454)
• [gelbooru_v02] extract total/search_count metadata (#7689)
• [instagram] provide post_url for stories and highlights (#7810)
• [kemono:discord] update server & channel metadata (#7569)
• [mangaread] fix manga_alt metadata
• [newgrounds] filter <script> content in tags (#7604)
• [patreon] return metadata for paywalled posts (#6582)
• [pinterest] remove excess whitespace from description fields (#4335)
• [pixiv] remove /jump.php from caption links (#4327)
• [tenor] extract more metadata
• [twitter] extract source_id and source_user metadata (#7470 #7640)
• [twitter] extract sensitive_flags metadata (#2523)
• [vk] fix user metadata extraction

Options

• [civitai] add option to retrieve post metadata (#7548)
• [exhentai] add limits-action option (#6504)
• [fanbox] add fee-max option (#7726)
• [kemono] extend duplicates option (#7696)
• [mangadex] allow ratings to be a (comma-separated) string (#7799)
• [misskey] add include option (#5347)
• [sankaku] remove id-format option (#5073 #6808)
• [webtoons] add banners and thumbnails options (#6468 #7441)

Common

• update browser User-Agents and headers
• allow using predefined Firefox/Chrome headers & ciphers
• allow overriding user-agent when browser is used (#7647)
• support using system certificates via truststore (#6582)
• fix URLs not getting written to -e/--error-file (#7758)
• raise ChallengeError for Cloudflare & DDoS-Guard challenge pages (#1945)
• prevent exceptions for for non-fatal requests (#7598)
• simplify user extractors by using Dispatch mixin
• allow GalleryExtractor instances to return additional asset files

Removals

• [mangasee] remove module

Downloaders

• support dynamic download rate limits (#7638)
• [http] fail downloads with HTML content (#4798 #7697)
• [http] add MIME type and signature check for .html files
• [http] fix potential FileExistsError when .part file moved (#5385)
• [http] implement _http_signature checks (#4902)
• [ytdl] fix KeyError - 'filepath' when using legacy youtube_dl (#6949 #7752 #7824)
• [ytdl] fix postprocessing/merge errors (#7581)
• [ytdl] detect yt-dlp independent of module name (#7599)
• [ytdl] support custom headers when fetching HLS/DASH manifests

Post Processors

• implement shortcuts for mode and event options (e.g. metadata/jsonl@post)
• [exec] implement commands option

Options

• add category-map and config-map (#7612)
• add signals-actions (#1861 #6582)
• add --compat command-line option
• add --Print command-line option
• swap --print and --Print semantics

Cookies

• add native support for LibreWolf profiles (#4101 #7625)
• improve cookie-related logging messages
• update expired cookie messages (#7644)

Formatter

• add D conversion - ISO 8601 string to datetime
• add L conversion - ISO 639-1 code to language name
• change old L/length conversion to n
• implement M format specifier

Optimizations

• replace %-formatted and .format(…) strings with f-strings (#7671)
• improve regular expression usage
• replace match.group(N) with match[N] (#7671)
• update match.lastindex usage
• remove pre-3.8 workarounds (#7671)
• remove @staticmethod decorators
• add request_json() and request_xml() functions
• slightly improve performance of extract functions
• slightly improve filter performance

Miscellaneous

• [actions] add flag and raise actions
• [job] refactor parent-child config path building (#7527)
• [job:data] wrap exceptions in a dict (#7723)
• [path] support character sequences in path-restrict replacements (#1707)
• [pyinstaller] exclude pkg_resources module (#7592)
• [scripts/init] add scripts to generate initial extractor code and test results
• [scripts/options] make output width independent of terminal size
• fix Last-Modified mtime overwriting post processor mtime (#7529)
• use pip for make install (#7628)