PKHarsimran/website-downloader v2.7.0

v2.7.0 — Crawl manifests and focused crawling

on GitHub

Website Downloader v2.7.0 adds machine-readable crawl reports and more control over which sections of a site are mirrored.

Release preparation: changes are in PR #61. This draft targets that PR's tested commit. Merge the PR before publishing; if it is squash-merged or changed, retarget the draft to the final main commit and rebuild its attached packages.

Crawl manifests

website-downloader --url https://example.com --destination example_backup --manifest example_backup/crawl.json

• Versioned JSON containing pages, assets, HTTP status codes, titles, headings, direct page-to-asset references, timestamps, crawl statistics, and errors.
• Explicit downloaded, cached, failed, and skipped outcomes, with reasons for skipped resources and null for unknown HTTP status codes.
• Works with parallel page/asset workers, --update, and --render-js.
• Atomic report writes and partial reports on graceful interruption. Automatic resume is not included.
• Compatible with existing update caches. For older cached pages, direct asset references are null until a fresh download records them.

Addresses #60.

More focused crawls

• Repeat --url or use --url-file to start from several sections of the same site.
• Use --max-depth to limit link depth from each starting URL.
• Skip unwanted sections with --exclude / --exclude-file; excluded page links continue to point at their live URLs.

Includes the previously unreleased work for #55 and #56.

Fixes and documentation

• Preserve actual JavaScript navigation status codes and record rendered HTTP errors as failures.
• Distinguish asset size-limit skips from download errors.
• Keep asset workers running after unexpected processing exceptions.
• Record actual saved page paths when a filesystem fallback is used.
• Include documentation, the CLI wrapper, changelog, and test fixtures in source distributions.
• Document the manifest format and limits, move the donation link below Quick Start, and add a light/dark star-history chart without restructuring the README.

Validation

All 66 tests pass both from source and independently against the installed wheel. Black, isort, Ruff, CLI help, sdist/wheel builds, and Twine checks pass. Real Chromium checks cover rendered metadata/assets and HTTP 404 handling.

Full changelog: v2.6.1...v2.7.0