Breaking change: --if-not-archived-within, --capture-cookie, and --user-agent now reject empty values. This release also stops returning stale results on resubmitted URLs, avoids amplifying HTTP 429 rate limits, and fixes several polling and sitemap bugs uncovered by property testing.
Breaking Changes
- Empty option values rejected:
--if-not-archived-within,--capture-cookie, and--user-agentnow exit with an error when given an empty or whitespace-only value. Previously, empty values were accepted and silently omitted from API requests.
Fixes
- Stale results on resubmitted URLs: SPN2 derives job IDs from the URL, so resubmitting a URL can briefly return the previous attempt's result. The archiver now ignores captures older than the submission, as well as errors returned within 5 seconds of submitting. If no newer result arrives before timing out, it falls back to reporting the earlier capture as a success.
- HTTP 429 handling: The HTTP layer no longer retries 429 responses (which previously retried five times, turning one rate-limited submission into six requests). Rate-limited submissions now pause for 60 seconds and re-queue.
- Default User-Agent: Requests now send
wayback-machine-archiver/5.0.0 (+https://github.com/agude/wayback-machine-archiver)instead of the defaultpython-requestsUser-Agent. - Missing jobs in status batches: Jobs omitted from a status batch now time out instead of staying pending forever.
- Backoff when slots are full: Polling now backs off when all job slots are occupied instead of hitting the API every 0.2 seconds.
- Poll-failure counter reset: After giving up on 5 consecutive failed polls, the failure counter was never reset, causing every subsequent job to fail on its first poll error. It now resets properly.
- Malformed status rows: Non-object rows and non-string
job_id,message, orstatus_extfields no longer crash polling. Anullstatus_extnow reportserror:unknown, matching an absent field. - Captures without a timestamp: Successful captures missing a timestamp now link to
https://web.archive.org/web/<url>instead of/web/None/<url>. - Sitemaps: Each sitemap is now fetched once per traversal, and whitespace around
<loc>values is trimmed. - Daily bandwidth errors:
error:max-daily-bandwidth,error:max-daily-bandwidth-from-ip, anderror:max-daily-bandwidth-hostnow log their documented explanations.
Testing
- Property tests: Added Hypothesis property tests in
tests/properties/, which run withjust test. Runjust proptestto execute 20,000 examples per property.