BlessedRebuS/Krawl v2.0.0 on GitHub

Krawl 2.0.0

We are proud to announce a major release focused on scalability, AI-powered deception, and a significantly improved dashboard.

The core engine now supports Redis and external PostgreSQL, while a new AI layer generates fake pages and tarpits crawlers dynamically. Soon you will have the possibility to import and export in bulk your deception templates, in order to have an automatically growing deception surface.

Overview and map improvements

New attack trend graph

AI generated pages manager

New features

Redis + external PostgreSQL support in #138, #141, #142
AI-generated fake/deception pages in #151, #153
AI agent tarpit in #167
Configurable garbage collector settings in #129
IP export (multiple formats) in #169
Export of AI-generated pages in #166
Cloudflare real-IP header fix in #150

Improved

Cache efficiency & hit rate in #132, #133, #168
Dashboard load time, visual bugs & detail panels in #139, #157, #162, #170
Suspicious log highlighting in #131
Banned IP response handling in #136, #137
Query optimization & memory performance in #140, #161

New Contributors

@Luca3700 made their first contribution in #156
@JiwaniZakir made their first contribution in #167

Breaking Changes

Removed API Endpoints

The file-based export system has been completely replaced by a real-time, on-demand API.

Removed Endpoint	Replacement
`GET /api/get_banlist?fwtype=iptables`	`GET /api/export-ips?categories=attacker&fwtype=iptables`
`GET /api/download/malicious_ips.txt`	`GET /api/export-ips?categories=attacker&fwtype=raw`

The new /api/export-ips endpoint accepts:

categories (required): comma-separated list from attacker, bad_crawler, regular_user, good_crawler
fwtype (optional, default raw): raw, iptables, or nftables

Action required: Update any cron jobs, firewall scripts (fail2ban, iptables, nftables plugins), or external integrations that called the old endpoints. See the Firewall Exporters documentation for updated examples.

Removed Configuration

Removed	Notes
`exports.path` in `config.yaml`	The entire `exports` section is gone
`KRAWL_EXPORTS_PATH` env var	No longer recognized
`exports.path` in Helm `values.yaml`	Remove from your custom values file
`./exports:/app/exports` Docker volume mount	Remove from your docker-compose overrides

Helm Chart v2.0.0

The Helm values.yaml schema has changed significantly. If you maintain a custom values file, review and update it against the new defaults. Key changes:

Removed: config.exports section
Added: postgres.*, redis.*, migration.*, config.tarpit.*, config.ai.* sections
Changed: mode field now controls deployment strategy (standalone vs scalable)

See the Helm chart documentation for the full values reference.

Deployment Modes: Standalone vs Scalable

Important

Krawl 2.0.0 introduces two deployment modes. This is a major architectural change.
If you are a Kubernetes user or run Krawl in production, read the full guide before upgrading:

Deployment Modes Documentation

At a Glance

	Standalone	Scalable
Database	SQLite (WAL mode)	PostgreSQL 16
Cache	In-memory Python dict	Redis 7 with 3-tier TTL
Max Replicas	1 (SQLite file lock)	Unlimited (shared DB)
Update Strategy	`Recreate`	`RollingUpdate`
External Dependencies	None	PostgreSQL + Redis
Best For	Dev, single-node, low traffic	Production, HA, high traffic

Migrating from Standalone to Scalable

Caution

If you have an existing Krawl v1.x deployment with data you want to keep, you must run the migration Job before switching modes. The migration reads your SQLite database and writes to PostgreSQL — it does not modify or delete the original SQLite file.

Step-by-step migration instructions (including Helm commands) are in the Deployment Modes Documentation — Migration Section.

Topic	Link
Deployment Modes (standalone vs scalable, migration)	docs/deployment-modes.md
Firewall Exporters (new `/api/export-ips` endpoint)	docs/firewall-exporters.md
AI Deception Page Generation	docs/ai_generation.md
Backups (SQLite + PostgreSQL)	docs/backups.md
Helm Chart Reference	helm/README.md