Added
fetch
: add redis --flushdb option #387fetch
: add --report & --cache-error options. --report creates a separate report file, detailing the URL used,
the response, the HTTP status code, and if its a cache hit.
--cache-error is used to also cache errors - i.e. identical fetches will return the cached error. Otherwise, fetch will
request the URL again. #393
Changed
fetch
: fast defaults. Now tries to go as fast as possible, leveraging dynamic throttling (using RateLimit and Rety-After headers)
but aborting after 100 errors. Also added a separate error progress bar. #388- Smarter
tojsonl
. Now scans CSV file and infers data types and uses the appropriate JSON data type #389 tojsonl
is also multithreaded #392stats
: use unwrap_unchecked for even more performance #390fetch
: refactor dynamic throttling #391- Bump sysinfo from 0.24.6 to 0.24.7 by @dependabot in #384
- cargo bump update several dependencies
- pin Rust nightly to 2022-07-23
Fixed
fetch
: fix --http-header parsing bug #386
Full Changelog: 0.59.0...0.60.0