New features
-
New cdx package for reading and writing CDX files
-
Added --format option to the cdx tool
-
Added a basic dedupe tool that can deduplicate records against a CDX server (such as OutbackCDX)
-
Added a stats tool that prints counts and sizes records by status, type, host
-
WarcReader now has a .position(long) method for seeking to the start of a particular record (if the underlying channel supports seeking)
Bugs fixed
-
HttpRequest/Response.serializeHeader() now returns the exact original bytes for records read from a WarcReader. This means the extract tool no longer reformats extracted HTTP headers.
-
Fixed an issue where null record ids would be written as "" instead of omitting the appropriate header. Notably this means revisit records can be constructed with WARC-Refers-To-Target-URI and WARC-Refers-To-Date but without WARC-Refers-To.