Fixes the cause of two production search outages (2026-09-09, 2026-09-18).
Background metadata refresh consumed the entire Hardcover daily allowance
(5000 requests) and left none for interactive search, so searching returned
nothing for hours at a time.
What went wrong
_hc_refresh_metawas the only one of sixcache_refreshstages called
without the cycle deadline — a one-hour job ran for eleven.- A 429 was retried four times with backoff. Rejected requests still count
against the quota, so each book cost five requests and the loop dug the hole
deeper the longer it ran. - Two stages gathered over a
Semaphore(1), so the first failure left the rest
of the batch running and spending. sync_librarycalled the same unbounded refresh independently, so pausing
cache_refreshdid not stop the drain.
What changed
All Hardcover calls go through one client that paces requests, reads the
published rate-limit headers, and treats the two limits differently: a
per-minute throttle is waited out, the daily one ends the cycle. Background work
stops while 500 requests remain so interactive search keeps working.
Also fixes the UNIQUE constraint failed: book_links.hardcover_id that aborted
every run at the final stage, and adds a .dockerignore.
cache_refresh now completes in about 80 seconds on roughly 60 requests, having
previously never finished at all.