Highlights
- First-class MCP support for AI agents: exposes TRAWL through Streamable HTTP with tools for reading pages, scraping protected URLs, capturing screenshots, and inspecting browser activity
- Agent-ready structured web content: extracts readable text, links, metadata, screenshots, console output, network activity, redirect chains, and selected response bodies
- Expanded anti-bot coverage: adds detection and resolution for DataDome and AWS WAF challenges, including deterministic escalation to headful browser sessions
- Stronger proxy guarantees: enforces strict per-request proxy routing, preserves multiple
Set-Cookieheaders, supports opting out of Tier 0, and generates certificates accepted by strict OpenSSL clients - More reliable deployments: reconnects Redis after startup failures, waits for Redis readiness in Compose, reaps orphaned browser processes with Tini, and includes FFmpeg in runtime images
- More resilient container builds: tracks rolling GeoLite releases and optionally retains OS font bundles required by spoofed browser fingerprints
- Updated runtime stack: upgrades TRAWL packages and dependencies to their latest mutually compatible versions
Breaking changes
- Environment variables have been standardized and legacy aliases removed. Existing deployments using renamed variables must update their configuration before upgrading.
- The unused
CHROME_EXECUTABLEsetting has been removed. - An empty or unset
REDIS_URLnow explicitly disables the Redis-backed cache.
See the configuration migration guide for the complete list of renamed variables and migration instructions.
Major changes
- 4d4d2d5 expands the MCP server with agent-focused
read,scrape,screenshot, andinspecttools (#116) - 9239e3b adds the original Streamable HTTP MCP server and generic AI integration (#104, closes #67)
- c32d110, 3a01f86, and 043ce19 add bounded response-body capture, browser diagnostics, redirect tracking, and viewport screenshots (#106, #107, and #109). Thanks @emandel2630
- 0e8d93e adds first-class DataDome challenge detection and resolution (#84). Thanks @CappyT
- 0a18827 and fcdaef3 add AWS WAF challenge detection, escalation, and a dedicated waiter (#82, based on #65). Thanks @nandyalu
- 6d85015 enforces strict per-request proxy routing and prevents silently successful responses from broken proxy paths (#86, closes #73)
- 898cf58 preserves and serializes every upstream
Set-Cookieheader (#81, based on #64). Thanks @nandyalu - f4330db introduces
MITM_ALWAYS_SCRAPEfor bypassing Tier 0 when browser processing is required (#111, closes #93) - 7180b76 adds SKI and AKI identifiers to generated MITM certificates for strict OpenSSL compatibility (#115, closes #113). Thanks @jespo2021 for the report
- a7357ba restores Redis connectivity after startup timeouts instead of permanently disabling Tier 2 (#112, closes #92)
- 70235db adds Tini to reap orphaned Camoufox subprocesses (#80, closes #79). Thanks @alekc for the report
- 5775255 installs FFmpeg in the baseline runtime image (#95). Thanks @tomkor
- 75c8bf7 preserves spoofed OS font bundles behind an explicit build argument (#101, closes #97). Thanks @emandel2630
- 80f3e67 replaces fragile GeoLite version pins with the rolling release (#105, closes #98). Thanks @emandel2630
- 7ecfe7e standardizes runtime environment variable naming and removes ambiguous legacy aliases (#114)
- c81d628 gates all container image publishing workflows on successful CI verification
- 50a0e95 prepares the v1.5.0 release, aligns package versions, and updates the project documentation