📦 Website Downloader v2.0 — Release Notes
Tag: v2.0 • Date: 2025-07-19
✨ Highlights
| Area | What’s new |
|---|---|
| Command-line UX | • All interactive input() prompts replaced with argparse flags.--url, --destination, --max-pages, --threads (+ rich --help).
|
| Logging | • Clean, colour-friendly log format (%(asctime)s | %(levelname)s | …).• Fixed %-format bug that broke Python 3.9/3.10 runners. |
| Crawler core | • Correct handling of pretty URLs (/about/)—no more unwanted .html suffixes.• Eliminated “double-domain” save paths. • Thread-pool asset fetcher (default 6 workers). |
| Reliability | • requests session with auto-retry (5×, exponential back-off).• 15 s timeout & path-sanitisation against ..\ / \ tricks.
|
| Performance | • Breadth-first crawl queue + concurrent downloads. • Crawl summary shows total time & average latency. |
| CI / Packaging | • GitHub Actions smoke-test mirrors 5 pages on Python 3.9 → 3.12. • requirements.txt trimmed to requests, beautifulsoup4, urllib3.
|
🚀 Quick-start
python website_downloader.py \
--url "https://example.com" \
--destination "archive" \
--max-pages 50 \
--threads 8If --destination is omitted the script defaults to example_com/.
🔄 Migration notes
- Rename any scripts that relied on interactive prompts—now supply flags.
- Logs are written to web_scraper.log (same folder as the script).
- Default --max-pages lowered 100 → 50; bump it as needed.
Happy mirroring! 🎉