Krawl 2.3.1
A patch release fixing IP enrichment, which had stalled on instances taking heavy
crawler traffic — the dashboard map and IP Insight showed no data for recently
seen IPs.
Fixes
Recently seen IPs are now geolocated first. get_unenriched_ips() selected a
capped batch in physical row order, so on a busy instance the batch worked
through an old backlog while newly arrived IPs — the ones every "recent" view
displays — were never reached. It now orders by last_seen descending.
"No city" is no longer treated as "not enriched". The same query also
required a city, which many datacenter and hosting ranges legitimately lack, so
successfully enriched rows kept re-qualifying for enrichment forever.
Coordinates are now the only completeness test.
Improvements
Single-request IPv6 addresses no longer get a row. Rotating proxy pools burn
a fresh IPv6 address per request — one operator produced ~24,500 addresses from a
single /32, each with exactly one request, and 42% of ip_stats was rows that
would never be seen again. An IPv6 address now earns its row on the second
sighting within 30 minutes. IPv4 is unaffected, and suspicious or
honeypot-triggering requests are always persisted, so nothing worth
investigating is lost.
Substantial drop in requests-per-min due to ignored ipv6 botnet
One geolocation lookup per IPv6 /48. Geolocation describes the network, not
the address, so the result is now reused across a /48 instead of costing an API
call per address. Failed lookups are never cached, and reverse DNS is not shared.
Docs
exclude_cdn and merge_banlists documented for /api/export-ips, including
the five supported CDN providers. The README's 65 environment variables are now
grouped into collapsible sections instead of one long table.
Upgrading
Drop-in — no configuration changes, no schema changes.
Existing rows are not cleaned up automatically. If you already have a large
backlog of un-geolocated IPs, enrichment will now prioritise recent ones but
will still work through the rest at 50 IPs per 5 minutes. Operators wanting to
reclaim the space from rotating IPv6 pools can delete those rows directly and
add the ranges to ignored_ips.
Full Changelog: v2.3.0...v2.3.1