Krawl 2.2.0

Highlights

This release brings a 1200+ page deception library built from real-world attacks, first-class Prometheus metrics & monitoring, custom HTML templating, bulk import/export of deception pages, and a large round of Kubernetes/Helm hardening for scalable deployments.

New Features

1200+ Deception Pages from Real-World Attacks (#224)

by @BlessedRebuS

Krawl now ships with 1289 HTML deception templates under src/templates/deception/, generated from real-world attacks captured across many Krawl honeypot instances. When an attacker probes a known vulnerable endpoint, Krawl serves a realistic page instead of a generic response, capturing their activity.

Templates auto-import on startup, .html extensions are stripped for cleaner URLs, and any path already present in the database is skipped. The community can now open PRs to contribute their own deception templates and expand Krawl's coverage.

Bulk Import / Export of Deception Pages (#197, #198)

by @BlessedRebuS

Deception pages can now be exported and imported in bulk, making it easy to back up, migrate, or share page libraries between Krawl instances. Upload and download file parsing was reworked, page limits for bulk uploads were removed, and the dashboard was updated to drive the new workflow. Kubernetes manifests were also updated to support bulk import.

Custom HTML Templating (#201)

by @BlessedRebuS

Krawl can now serve a fully custom deception page via a user-provided HTML template.

• New custom_template_path config field / KRAWL_CUSTOM_TEMPLATE_PATH env var.
• Docker Compose and Kubernetes manifests mount the template at /templates/custom_page.html.
• Helm support via a disabled-by-default customTemplate feature backed by a ConfigMap.
• Removed redundant Kubernetes manifests.

Self-Hosted LLMs on Kubernetes (#202, #217, #218)

by @BlessedRebuS

Building on the 2.1.0 self-hosted LLM support, local inference engines (llama.cpp and Ollama) can now be deployed directly on Kubernetes via the Helm chart.

• Added Helm values for both Ollama and llama.cpp, with a unified krawl-llm service name.
• Tuned llama.cpp Helm template and fixed template syntax.
• Extended Helm chart documentation with full LLM setup instructions.

Credential Download as ZIP (#223)

by @BlessedRebuS

Captured credentials can now be downloaded directly from the dashboard as a ZIP archive containing usernames.txt and passwords.txt (de-duplicated) — ready to feed into wordlist analysis or threat intelligence pipelines.

Metrics & Monitoring

Prometheus Metrics (#189, #196, #204)

by @carnivuth, @Lore09

Krawl now exposes operational and security metrics in Prometheus format.

• New metrics section in config.yaml (metrics.enabled) to toggle exposure on or off.
• A /metrics endpoint served under the secret dashboard path (returns 404 when disabled).
• All metric definitions centralized in a dedicated src/metrics.py module, with background tasks (analyze_ips.py, dashboard_warmup.py) refreshing metrics as part of their normal execution.
• A custom KrawlMetricsCollector exposes cumulative values as Prometheus counters and recomputes clients_total live at scrape time, eliminating drift and negative values.
• Batch counter reads (get_many) and an optimized get_all in metrics_counters.py reduce Redis round-trips for scalable deployments.

ServiceMonitor for Prometheus Operator (#204)

by @Lore09

• New serviceMonitor section in helm/values.yaml and a servicemonitor.yaml template.
• The ServiceMonitor is only created when metrics are enabled and a fixed dashboard secret path is set, with clear guidance and warnings in NOTES.txt for misconfigurations.

A Grafana dashboard and metrics/monitoring documentation are included to get started quickly.

Bug Fixes

Private/Reserved IPs No Longer Banned or Tracked (#219)

by @Lore09

Private, local, and reserved IP addresses (e.g. Kubernetes health-check sources) were being tracked and could be banned. A new is_local_or_private_ip utility is now used in BanCheckMiddleware, record_access, and increment_page_visit to skip ban and tracking logic entirely for these addresses.

Broken Healthcheck (#220)

by @Lore09

• Added a dedicated /healthz endpoint under the secret dashboard prefix, exempt from ban and deception tracking.
• The Helm deployment now uses the secret dashboard path for HTTP probes (falling back to a TCP probe when no secret path is set); the all-in-one manifest uses a TCP startup probe.
• Health probe requests are suppressed from access logs to reduce noise.

Exponential Backoff Overflow (#221)

by @Lore09

The exponential ban multiplier for repeat offenders could overflow the database integer column and crash with "integer out of range". A new MAX_BAN_EXPONENT constant and _ban_multiplier_for helper cap the multiplier at 2**10, enforced everywhere the multiplier is set.

Helm Service Selector Collision (#216)

by @BlessedRebuS

The Krawl service selector was matching both Krawl and Ollama pods (both used krawl.selectorLabels). An app.kubernetes.io/component: krawl label now distinguishes them.

Boolean Migration Defaults (#203)

by @Lore09

Migration functions were updated to use proper boolean defaults.

Improvements

Database Manager Refactor & Query Optimization (#208)

by @Lore09

• Introduced modular repository classes (CredentialRepo, GeneratedPageRepo, and others) for cleaner database access, with a re-exported public API (db.credentials.count(), db.ip_stats..., etc.).
• Added indexes on the ip_stats table (last_seen, first_seen, reputation_score) to speed up paginated and sorted queries.

Dashboard & AI Generation Improvements (#200, #222, #223)

by @BlessedRebuS, @Lore09

• Removed duplicate AI-generated pages and added generated pages to the search bar.
• Collapsible search-results summary with a cleaner expand/collapse interaction; clicking an IP cell collapses results.
• Direct export download URL with a copy-to-clipboard button (for external systems like OPNsense).
• "Inspect IP" button on attack rows for quick access to IP insight.
• Import all file types from the deception folder (not just .html, including dotfiles); preserve non-.html extensions in URL paths.
• AI generation is now shielded from request cancellation, so slower self-hosted LLMs can complete.
• Access History styling tweaks and a delete button on search results (authenticated only).
• Reordered overview: Top User-Agents above Top IP Addresses.

Massive Code Linting & Security Vulnerability Management (#206)

by @Lore09

A large pass of code linting and dependency/security vulnerability cleanup across the codebase.

Suppress Server Header (#214)

by @BlessedRebuS

Added the --no-server-header flag to uvicorn to suppress the Server header, reducing fingerprinting (fixes #207).

Automated Banlist Updates (#210)

by @carnivuth

Added a GitHub Action that updates the banlist from the Krawl demo instance and opens a PR (closes #77).

Helm & Deployment Tuning (#215, #217)

by @BlessedRebuS

• Clarified Traefik configuration documentation.
• Added an init container for PostgreSQL in scalable deployments.

What's Changed

• Feat/bulk export import deception pages by @BlessedRebuS in #197
• modified upload and download file parsing, modified dashboard by @BlessedRebuS in #198
• Remove duplicates for ai generated pages, added generation pages to searchboar by @BlessedRebuS in #200
• Added custom Krawl HTML templating for deception page by @BlessedRebuS in #201
• add support for self-hosted LLMs on kubenretes by @BlessedRebuS in #202
• Centralize metrics in dedicated module by @Lore09 in #196
• 5 add prometheus exporter for metrics by @carnivuth in #189
• fix: update migration functions to use boolean defaults by @Lore09 in #203
• Feat: adding ServiceMonitor and fixing metrics by @Lore09 in #204
• Feat: massive code linting and security vuln management by @Lore09 in #206
• Feat: db manager refactor and query optimization by @Lore09 in #208
• Added action to update banlist from krawl demo with PR by @carnivuth in #210
• Added --no-server-header directive to uvicorn by @BlessedRebuS in #214
• Fix/adjust helm values by @BlessedRebuS in #216
• Clarified Traefik configuration by @BlessedRebuS in #215
• Added initcontainer for postgresql scalable deploy, tuned llamacpp st… by @BlessedRebuS in #217
• Tuned helm values for local llms, extended doc for helm chart llm sup… by @BlessedRebuS in #218
• Fix: private ips timed out by @Lore09 in #219
• Fix broken healthcheck by @Lore09 in #220
• Implement exponential backoff for ban multiplier by @Lore09 in #221
• Feat UI improvements by @Lore09 in #222
• Improve AI generation, dashboard UI, and add credential download by @BlessedRebuS in #223
• Added 1200+ deception pages generated from real world attacks by @BlessedRebuS in #224
• Feat: release 2.2.0 by @Lore09 in #225
• Pre-release - v2.2.0 by @BlessedRebuS in #226

Full Changelog: v2.1.0...v2.2.0