Breaking Changes
- Failed requests and unparsable users now raise
AbortReqError,ApiError,XClIdParseErrororUserParseErrorinstead of returningNoneor ending iteration early;Noneand empty results now mean X has no data (#347, by @Flaburgan) tweet_details,user_about,community_infoandparse_trendnow raiseParseErrorwhen the requested object can't be parsed, instead of returningNone;UserParseErroris now its subclass, anduser_aboutreturnsNonefor an unavailable user instead of an empty object (#348, #364)- Made curl-cffi a required dependency and the default HTTP backend, and removed the
twscrape[curl]extra; setTWS_HTTP_BACKEND=httpxto switch back (#363) - With the default backend,
Response.urlis astr,Response.headersandResponse.requestare curl-cffi objects,Response.requestcan beNone, andAccount.user_agentis not sent (#363)
Features
- Added
tweet.articlefor X Articles, with the full article content intweet_details(#323, by @softrebel) - Added parsing for message me, Periscope broadcast, live event, app, promo video and promo image conversation cards, and polls with video (#333, #335, by @Flaburgan)
- Added
reactivateCLI command andAccountsPool.reactivateto mark accounts with a session active again (#334, by @Flaburgan) - Added
retweetedTweetIdandquotedTweetIdwith their string versions toTweet, set even when the referenced tweet is unavailable (#361, by @Flaburgan) - Added media metadata to photos, videos and gifs: IDs, alt text, size, short and expanded URLs, and the source tweet and user (#370, by @Flaburgan)
Fixes
- Fixed quoted tweets appearing as standalone timeline items (#338, by @Matthew-Selvam)
- Fixed items yielded twice when they appear on several pages of a paginated call (#332, by @Flaburgan)
- Fixed tweets being dropped or user profiles being overwritten when X sends an author without a full profile; such tweets now get a placeholder
tweet.user(#344, #345, #352, #353, by @Flaburgan) - Fixed requests failing when X requires GraphQL features twscrape doesn't send yet; they are now enabled and the request is retried (#339, by @niyamvora)
- Fixed
user_by_idbeing blocked by Cloudflare, by using curl-cffi by default (#337, #363) - Fixed requests stopping instead of moving to the next account on a 429 or a rate limit error without rate limit headers (#351, #364)
- Fixed a suspended account stopping the whole request; it's now marked inactive and the next account is used (#211, #364)
- Fixed
trendsfailing for the news, trending and entertainment tabs (#371, by @Many0therFunctions) - Fixed the curl-cffi backend sending auth cookies to other hosts, resolving
socks5://proxy hosts locally, and retrying timeouts (#363)
Improvements
- Retried X timeouts (
DeadlineExceeded) a few times before raising (#347, by @Flaburgan) - Reported Cloudflare challenge pages with a clear
XClIdParseError(#338, by @Matthew-Selvam) - Exited with code 1 without a traceback on failed CLI requests
Maintenance
- Updated GraphQL operation IDs for current X API compatibility
Full Changelog: v0.20.1...v0.21.0