Krawl 2.3.0

Banlist federation, native Cloudflare WAF integration, malware sample downloads, a
redesigned dashboard, and a memory pass that keeps RSS flat under sustained scanning.

Upgrading is a drop-in. Every existing config.yaml key and KRAWL_* environment
variable still works — this release only adds new ones. Schema migrations run
automatically at startup and are idempotent; back up your database first, as always.

helm upgrade krawl oci://ghcr.io/blessedrebus/krawl-chart --version 2.3.0 -f values.yaml
# or
docker pull ghcr.io/blessedrebus/krawl:2.3.0

Features

🌐 Banlist federation

Krawl instances can now share banlists with each other. Publish yours on an
unauthenticated path, and pull in lists from other instances — or any plain-text IP
list — merged into local ban decisions.

banlist:
  export_path: "/public_banlist.txt"   # publish ours (empty = disabled)
  sources:                             # and merge theirs
    - "https://demo.krawlme.com/das_dashboard/api/export-ips?categories=attacker&fwtype=raw"
  refresh_interval: 3600

Merged sources show up in the dashboard and via GET /api/banlist-sources.

☁️ Cloudflare WAF integration

Push banned IPs straight to a Cloudflare Account IP List for use in WAF rules. A
background task replaces the list wholesale on a configurable interval, with a dashboard
panel to configure it and trigger a sync on demand. Credentials come from the
environment or a Kubernetes Secret and are never written back to disk.

🧪 Attachment downloads for malware analysis

When an attacker uploads a file — a webshell, a malware sample, an exfiltration payload —
the raw request modal now shows a paperclip listing the uploaded files, and downloads the
one you pick. Both multipart/form-data parts and raw-body uploads are recognised.
Treat every download as hostile and open it in a sandbox.

🛰️ CDN provider ranges excluded from exports

Behind a CDN, the address Krawl records can be an edge node rather than the attacker —
and blocking those ranges takes the CDN offline for your own users. Exports can now drop
them, for Cloudflare, Fastly, CloudFront, Google and Bunny.

curl ".../api/export-ips?categories=attacker&exclude_cdn=cloudflare,fastly,cloudfront,google,bunny"

Ranges are fetched live from each provider's published list at export time and cached for
an hour, so they never go stale and nothing is stored locally. In the dashboard's export
dialog they are checkboxes under Skip CDN providers, all five ticked by default. If a
provider's list can't be fetched the export still succeeds — that provider is simply not
excluded, and a warning is logged.

🚫 Ignored IPs

A new ignored_ips list of IPs and CIDR ranges (IPv4 and IPv6) that are never tracked,
banned, exported or persisted — and are purged from the database at startup. Defaults
cover loopback, RFC1918, link-local and CGNAT, so health checks, monitoring probes and
office egress stop polluting your attacker statistics out of the box.

🔐 Sessions and lockout shared across replicas

Dashboard sessions and login attempt counters moved out of per-process memory into the
cache layer: Redis in scalable mode, in-process in standalone.

Previously, running more than one replica meant a cookie issued by one pod was unknown to
the next — the dashboard logged you out at random — and the 5-attempt lockout applied per
pod rather than per attacker, handing a bruteforcer N× the attempts. Both are fixed.

🧠 Memory and throughput

A pass aimed squarely at RSS growth under sustained scanning:

Bounded access-log write buffer, with depth and dropped-row counters exposed to Prometheus
Retention deletes run in batches, with a pre-retention pass pruning non-suspicious rows first
Per-run cap on IPs analysed, so a traffic spike can't queue unbounded work
Global banlist held as a frozenset
Request bodies over 64 KB are no longer buffered
Ban check and deception response paths short-circuit earlier

🎨 Redesigned dashboard UI

A front-end pass that touched most of the interface:

A design token system driving colours, spacing and category styling from a single source shared between the CSS and JS, replacing scattered hard-coded values
The Material Symbols webfont is gone — icons no longer block first render
Expandable table rows across the attack and suspicious-activity panels, with pagination and filtering rather than one flat list
Raw request modal with syntax highlighting and clearer structure
Attack list gained method and a sortable size column, ordering correctly across B/KB/MB
Timed-out IPs panel with an inspect button and sortable columns
Clickable IP rows leading straight to IP Insight, and consistent iconography for ban / exempt / re-enable actions
~840 net new deception page templates (1289 → 2132)

📊 Monitoring

Three new metrics — krawl_timed_out_ips, krawl_write_buffer_rows,
krawl_write_buffer_dropped_total — and a rebuilt Grafana dashboard:

Stats that follow the time picker (they previously showed lifetime totals regardless of it)
Pie charts replaced by a sorted bar gauge and a log-scale trend, so categories orders of magnitude apart stay readable
Per-minute rates instead of per-second flatlines
A $job variable, so one Prometheus scraping two Krawl deployments no longer merges them silently
A System health row: write buffer depth per pod, dropped rows, enrichment and reevaluation backlogs

☸️ Kubernetes / Helm

Externally-managed Secrets for PostgreSQL, Redis, dashboard, AI, canary and Cloudflare credentials, plus extraEnv / extraEnvFrom for External Secrets Operator / Vault
grafanaDashboard.enabled ships the Grafana dashboard as a ConfigMap for the sidecar to discover
values-minimal.yaml and values-standalone.yaml reworked around the settings deployments actually touch, with mode-appropriate defaults

📖 API documentation

FastAPI's native docs are now served at /<dashboard-path>/docs, with the schema at
/<dashboard-path>/openapi.json. Only /api/* routes appear in it.

Bug fixes

Dashboard sessions no longer drop at random across replicas, and login lockout counts per attacker instead of per pod
pre_retention_cleanup honeypot-trigger query was matching the wrong rows
db_retention query logic corrected alongside the batching change
Attack-tab size column now sorts numerically rather than as a string, so MB values order correctly
Malicious IPs render correctly in the merged banlist view
Duplicate method key removed from the log data structure
Security & dependency workflows no longer fail on forked pull requests
Fixed a broken star-history chart in the README

Refactors and breaking changes

No configuration breaks: no config.yaml key or KRAWL_* variable was removed or
renamed. The items below only affect people who patched Krawl's internals or scripted
around its files.

Change	What to do
`src/firewall/` collapsed from a `FWType` ABC plus one class per format into a `FORMATS` dict of functions	`?fwtype=raw\|iptables\|nftables` is unchanged. If you subclassed `FWType` to add a format, add a function to `FORMATS` instead — see `docs/firewall-exporters.md`
Standalone migration scripts deleted; the runner is driven from column/index tables	Nothing. Migrations still run automatically at startup and are idempotent. `scripts/migrate_sqlite_to_mariadb.py` is gone
`docker/docker-compose.dev.yaml` split into `docker/dev/docker-compose.{standalone,scalable}.dev.yaml`	Update your `-f` path. Production files stay in `docker/`; the root `docker-compose.yaml` is still the run-from-repo-root scalable stack
Grafana dashboard JSON is now generated by `scripts/build_grafana_dashboard.py`	Re-import the dashboard — panel queries changed and three metrics are new. Edit the script rather than the JSON
`template_loader.py` + `html_templates.py` merged; shared `pagination()` helper; dashboard auth moved to a FastAPI dependency	Nothing — internal only, same endpoints and same auth requirements

Docs

New Dashboard API and Tarpit references;
Cloudflare Banlist Sync; proxy headers and decoy
subdomains; a refreshed architecture tree and background-task table; a Grafana panel
guide; and nine previously undocumented KRAWL_* variables added to the README.

What's Changed

Banlist and Cloudflare

added public banlist fetch and merge API by @BlessedRebuS in #244
Feat/add community banlist by @BlessedRebuS in #246
Implemented CloudFlare native banlist feature by @BlessedRebuS in #271
Added CloudFlare WAF rules creation from Krawl webhook by @BlessedRebuS in #273
Feat: add cloudflare cdn exclusion by @Lore09 in #274

Dashboard and deception

Feat: improved timedout ips insight and ignored ips by @Lore09 in #238
feat: enhance timed-out IPs table with inspect button by @Lore09 in #239
Feat/add new deception pages by @BlessedRebuS in #248
Enhance raw request modal with highlighted content and improved styling by @federicolepera in #252
feat/add method and size to attack list by @federicolepera in #256
feat: add size-cell class for consistent styling of request size by @federicolepera in #261
Feat/add sortable payloads by @BlessedRebuS in #264
Fixed sorting for MB sizes by @BlessedRebuS in #265
Feat/add downloadable content type by @BlessedRebuS in #267
Feat: improved expanded tabs and externally-managed secrets by @Lore09 in #270
Feat: frontend code cleanup and UI style improvement by @Lore09 in #286
Feat: updated deception templates by @Lore09 in #288

Performance and internals

Feat: ponytail code prune by @Lore09 in #283
Feat: ram usage and disk cleanup optimization by @Lore09 in #287

Deployment and secrets

Feat improved secret management by @Lore09 in #281
Feat: Krawl release 2.3.0 by @Lore09 in #291

Docs

Added doc for proxy headers and subdomains by @BlessedRebuS in #268
fix broken star history chart in README by @OctoBored in #284

Feat: enhance PR linting workflow with auto-fix capabilities by @Lore09 in #245
Prevent Security & Dependencies workflow failures on forked PRs by @BlessedRebuS with @Copilot in #253
Feat: upgrade workflow versions by @Lore09 in #255
fix: change workflow runners to self-hosted by @Lore09 in #259

New Contributors

@federicolepera made their first contribution in #252
@OctoBored made their first contribution in #284

Full Changelog: v2.2.0...v2.3.0

BlessedRebuS/Krawl v2.3.0 on GitHub