apify/crawlee v4.0.0-rc.1

on GitHub

Breaking since rc.0

• @crawlee/jsdom and @crawlee/linkedom moved out of the monorepo (#4119)
• Crawler-only code moved from @crawlee/core to @crawlee/basic, more internals made private (#4117, #4036, #4060)
• Input handling moved to the Apify SDK, requires apify >= 4.0.0-beta.40 (#4155)
• The request queue is no longer emptied between run() calls (#4056)
• Statistics renames, IRequestManager / IBrowserPool contract updates (#4109, #4061, #4161)

New

• DomCrawler with a pluggable DOM parser (#4104)
• OpenTelemetry instrumentation via @crawlee/otel (#3361)
• skipRequest and afterCommit context helpers (#4158, #4110), initialConcurrency option (#4067), extractMicrodata helper (#3233)

Fixes

HTTP redirect handling for credentials, cookies and request bodies (#4137, #4154, #4136), Cloudflare challenge detection (#4170), a robots.txt 404 now means allow-all (#4171), and more.

Upgrading guide: https://crawlee.dev/js/docs/next/upgrading/upgrading-to-v4

Full Changelog: v4.0.0-rc.0...v4.0.0-rc.1