- Persistent Session with Retry Logic: Reuses HTTP connections and automatically recovers from transient network errors.
- Enhanced Logging: Logs are now written to both a file and the console, with detailed timing and status information for easier debugging.
- Selective Resource Downloading: Only internal domain resources are fetched, preventing unwanted external downloads.
- Download Time Estimation: Calculates average page download time, plus estimated total and remaining crawl times.
- Integrated Verification: Automatically calls the check script after downloading, providing a summary of missing files and overall stats.
Why Update?
These improvements make the downloader more robust, user-friendly, and maintainable, giving you detailed insight into download performance and completeness.
What's Changed
- Update website-downloader.py by @PKHarsimran in #1
- Update check_download.py by @PKHarsimran in #2
- 4 error cannot unpack non iterable nonetype object by @PKHarsimran in #5
- Feature/network performance improvements by @PKHarsimran in #6
Full Changelog: v1.0.0...v1.1.0