What's Changed
🎉 New Features
- Added automatic login support by @dogancanbakir in #1620
- Added per-domain page budget to limit pages crawled per domain (0 = unlimited) by @dogancanbakir in #1612
- Added per-host rate limiting with adaptive backoff by @dogancanbakir in #1614
- Made jsluice cgo-free by @Mzack9999 in #1570
- Added page load strategy by @fail-open in #1514
-pls, -page-load-strategy string page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")
🐞 Bug Fixes
- Fixed cleanup of crawler resource lifecycles by @x-stp in #1634
- Fixed headless Chrome profile persistence with
-no-incognitoand-chrome-data-dirby @iamkunal9 in #1608 - Fixed gracefully handling DOM retrieval timeout in headless mode by @trasnake87 in #1592
- Fixed ignored headless options since v1.4.0 by @dogancanbakir in #1622
- Fixed onclick link discovery for dynamic JS navigations by @dogancanbakir in #1624
- Fixed preserving response body/raw and waiting after form submission in headless mode by @XTeam-Wing in #1585
- Fixed output of unvisited urls by @Mzack9999 in #1595
Other Changes
- Added
noneto filters by @Mzack9999 in #1594 - Added helper functions + tests by @Mzack9999 in #1596
- Added deduplication of headless mode output by @dogancanbakir in #1602
- Updated Ubuntu installation instructions by @ecdeveloper in #1628
New Contributors
- @trasnake87 made their first contribution in #1592
- @iamkunal9 made their first contribution in #1608
- @ecdeveloper made their first contribution in #1628
- @x-stp made their first contribution in #1634
- @XTeam-Wing made their first contribution in #1585
Full Changelog: v1.5.0...v1.6.0