Krawl 2.4.0

A feature release. Krawl now reports which campaign is hitting you rather than only how much traffic arrived, and a multi-pod deployment behaves like one cluster instead of N instances each repeating the same work.

Threat intelligence

Payloads are grouped into campaigns. Every captured file and flagged request body is fuzzy-hashed with TLSH, and digests within tlsh_cluster_threshold of each other join the same cluster. A webshell and its lightly-edited variants have completely different SHA-256 digests and read as unrelated one-off hits; under TLSH they are one campaign with a capture count and a source count. The new Threats tab charts campaign activity over 1, 7 or 30 days, lists the campaigns, and indexes every captured file across all IPs — the same filename arriving from many addresses is the signal that a kit, not a person, is at work.

Hashing runs as the scheduled hash-payloads task rather than at request time, so ingest stays cheap. New captures are hashed on the next run, and existing history is swept backwards once, tracked by a watermark. (#251)

IP Insight shows where an attacker came from and what it left behind. The tab gained referer history — which bait link or external source led that address here — alongside the files it uploaded and the credentials it submitted, each filtered to that IP. Referer capture is what makes a bait chain reconstructable after the fact, rather than a request that appeared from nowhere. (#250)

Scalable mode

Global tasks run once per cluster, not once per pod. Banlist refreshes, cache warmups and retention passes were previously executed by every pod on the same cron, so a five-pod deployment did five times the work and five times the writes. Tasks that affect shared state are now marked single_pod and gated behind a Redis-backed lease; the rest still run everywhere, because per-pod state needs per-pod work. External banlists are fetched once and shared, and a pod joining a running cluster adopts what a peer already fetched instead of paying for the external requests itself. (#296)

Settings → Maintenance runs any scheduled task on demand, and shows which pod last ran each one. A manual run is never gated by the lease: it executes immediately on the pod serving the request, deduplicated only against other concurrent manual runs. (#297)

Pods no longer wipe the shared cache at startup. Every pod flushed the cache on boot, so a rolling restart cold-started the whole cluster once per pod instead of once.

Dashboard

The running configuration is visible in the settings panel, with each value badged when it differs from the shipped default — so "is this instance actually using the setting I think it is" stops being a question answered by reading YAML on a node. Reconciling that view turned up nine settings whose dataclass default and config fallback had drifted apart, which are now the same number in both places and stay that way under test. (#299)

The basemap is configuration, not code. Tile provider, attribution, subdomains and an API key are all settable, so the map can use a keyed provider and lose its watermark. The key is necessarily public — the browser fetches tiles directly — so lock it to your dashboard domain at the provider. (#295)

The dashboard can carry your own identity. dashboard.branding sets the name, link, logo, version visibility and a contact line, for deployments where whoever finds the dashboard should be told which team runs it and how to reach them. Defaults are unchanged.

Clicking an IP opens IP Insight. Every table used to expand a hidden row containing a smaller copy of the tab that the magnifier button already opened. One click, one destination — and tables already filtered to a single address no longer repeat that address in every row.

Performance and stability

Memory is bounded under flood. In-process state that reached 4 GiB during an IPv6 flood is now capped, request bodies are read to a limit regardless of what the client declares in Content-Length, and the server caps its own concurrency and backlog rather than queueing itself into an OOM. IPv6 traffic can be dropped outright with ipv6.ignore, answered from the ASGI scope before anything allocates. Memory growth is exported as Prometheus gauges, so the next surge is observable rather than inferred from a restart.

The database does less work for the same answers. Unique paths are tracked with a HyperLogLog instead of an exact set, the agg:* dashboard caches paginate inside Redis instead of being loaded whole, bundled Redis has a memory ceiling and an eviction policy, and PostgreSQL gets autovacuum and analyze thresholds that actually fire — the stock scale factors are a fraction of the table, so on an append-mostly honeypot table at 1.5M rows they wait for ~300k dead tuples and effectively never trigger, leaving the planner sizing a multi-million-row table from its empty-table estimate.

Startup stopped doing a day's work at once. Seven scheduled tasks no longer also run at boot, the startup cleanup no longer scans the event tables to find candidate IPs, and the ban check is served from an in-memory cache. The Threats tab is pre-warmed like the rest of the dashboard; campaign clustering scans every payload hash in its window, which made it the slowest read in the product when served cold.

Fixes

Enrichment no longer stalls behind its own backlog at startup, and the analyzer no longer starves the tail of the address space.

Krawl's log records stop propagating to the root logger, which duplicated every line when the host application configured its own handler.

Generated pages are served with their real content type. A path ending in .json returned JSON wrapped in HTML, which is an obvious tell.

Docs

The Threats tab, the IP Insight panels and the four dashboard tabs added since 2.3.x are documented; the tab list said five tabs, and there are nine. Scalable mode gains a section on task synchronisation and metric aggregation across pods. The README's environment-variable counts are now computed from the tables they label — three had drifted, one by six.

Upgrading

Docker Compose. The shipped compose files track ghcr.io/blessedrebus/krawl:latest, so pulling is the whole upgrade. Pin the tag to 2.4.0 first if you would rather choose when to move.

docker compose -f docker/docker-compose.standalone.yaml pull
docker compose -f docker/docker-compose.standalone.yaml up -d
# scalable: same two commands against docker-compose.scalable.yaml

Docker run. Pull, drop the old container, start it again with the same flags, volumes and environment you used before — the data lives in the volume, not the container.

docker pull ghcr.io/blessedrebus/krawl:2.4.0
docker stop krawl && docker rm krawl
docker run -d -p 5000:5000 -v krawl-data:/app/data \
  --name krawl ghcr.io/blessedrebus/krawl:2.4.0

Helm. Chart and image are both 2.4.0. Keep your own values file; every new key has a working default.

helm upgrade krawl oci://ghcr.io/blessedrebus/krawl-chart --version 2.4.0 -f values.yaml

From source. Reinstall dependencies — py-tlsh is new and compiles a C++ extension at install time, so this is the one path that needs a toolchain (build-essential on Debian/Ubuntu, gcc-c++ on RHEL).

git pull && pip install -r requirements.txt
uvicorn app:app --host 0.0.0.0 --port 5000 --app-dir src --no-server-header \
  --limit-concurrency 512 --backlog 256

Every existing config key and KRAWL_* variable still works, and schema migrations run automatically at startup — there is no manual step.

Closes #250, #251, #269, #295, #296, #297 and #299.

Full Changelog: v2.3.1...v2.4.0