Release Notes:
BREAKING CHANGE: You must now authenticate to the API, instructions for doing so are in the README.
This is a major update that fundamentally improves the reliability and performance of the archiver by standardizing on the official Internet Archive SPN2 API. To achieve this, the unauthenticated "fire-and-forget" mode has been removed.
All archiving jobs now provide a confirmed success or failure status, eliminating the uncertainty of the previous unauthenticated method. This release also introduces significant performance improvements for polling large archives and provides better feedback to the user through enhanced logging.
What's New
- BREAKING CHANGE: Authentication Now Required: The script no longer supports unauthenticated use. It will now exit with a helpful error message if Internet Archive credentials are not found.
- More Efficient Status Polling: The script now uses the SPN2 batch endpoint to check the status of all pending jobs in a single API request. This dramatically reduces network traffic and leads to faster completion for large archives.
- Improved Logging and Final Summary: The script now provides more detailed logs when gathering URLs from various sources. More importantly, a summary is printed at the end of the run, showing the total number of URLs processed, successful captures, and failed captures.
- Increased Reliability: Fixed a critical bug where a URL could "vanish" from the queue if the API rejected it without returning a job ID. These cases are now correctly identified and counted as failures.
- Major Code Refactoring: The core application logic has been split from a single large file into smaller, focused modules (
cli,sitemaps,workflow). This improves the stability and maintainability of the project for future development.