Release Notes:
This release focuses on hardening the archiver for long-running and unattended operation. It introduces critical safety nets to prevent the script from running indefinitely due to persistent API errors or jobs that become stuck.
This update makes the archiver significantly more robust, ensuring that even very large jobs will eventually complete without manual intervention. The script's intelligence in handling temporary API rejections has also been improved, further increasing its resilience.
There are no breaking changes in this release.
What's New
-
Enhanced Stability for Unattended Archiving: The archiver is now protected against two critical edge cases that could cause it to run forever.
- A URL that consistently failed with a transient error (like
service-unavailable) during status checks will now be marked as a permanent failure after 3 attempts, preventing an infinite re-queue loop. - A 2-hour timeout has been implemented for any job that remains stuck in a "pending" state, ensuring the script can move on from jobs that may be stalled in the Internet Archive's backend.
- A URL that consistently failed with a transient error (like
-
Smarter Retries for API Rejections: If the API accepts a submission but doesn't return a
job_id(often due to rate limits or high load), it is no longer treated as a permanent failure. The script now correctly identifies this as a temporary issue, logs a specific warning, and automatically re-queues the URL for another attempt. -
Adds Debug Logging for Status URLs: When running with
--log DEBUG, the script now prints the direct URL to the API's JSON status endpoint for each successfully submitted job. This provides a convenient way for users to manually inspect the status of a specific capture.