This release comes out of a full review of the tool. It fixes the crawler on sites that use relative links, handles Cloudflare and error captures, and stops the tool from contacting hosts it should not.
Behaviour changes
- The live site is no longer used to fill gaps. A file Wayback does not have is reported as failed. Only a short list of CDNs is ever fetched live: Google Fonts,
code.jquery.comand the Squarespace CDN. - Python 3.10 or newer is required.
- The command exits 1 when nothing was saved, when
WAYBACK_URLis malformed, and when Wayback refuses 5 requests in a row. MAX_FILESbounds attempts, and a value that is not a positive integer is an error.KEEP_REDIRECTIONSis removed. It never did anything.- Server-side pages are stored as
<name>.php.html.
Fixed
- Relative links were requested from Wayback as hostnames, so many sites archived as a mix of unrelated pages and failures (#56).
- A capture that is a Cloudflare challenge or a server error is replaced by the nearest capture with status 200, found through one CDX query (#48, #57).
- A page could make the tool fetch arbitrary hosts directly, for example
http://web.archive.org@169.254.169.254/, and save the response (#55). - Pages that are not UTF-8 no longer lose their accented characters (#56).
- The font clean-up no longer deletes neighbouring CSS rules, and runs in linear time (#55).
- The site's own scripts and images are no longer removed as ads, trackers or Wayback toolbar code.
srcset, fragment links,data-*JSON,<base href>, SVG sprites and media elements are handled correctly, and the files they point at are downloaded (#58). - A missing file costs at most 5 requests instead of 15. Wayback's 429 is retried with backoff (#57).
- Ctrl-C stops the run (#57).
.envis read from the directory the command runs in (#57).OPTIMIZE_IMAGESkeeps transparency, animation and icons (#58).
Measured
The Flask 1.1.x docs snapshot, limited to 40 files: 956 requests and 428 s with 61 failed pages before; 40 pages, no failures and 47 s now.