What's New
Documentation
- Crate READMEs on crates.io — all 7 crates now have detailed README documentation visible on their crates.io pages, with usage examples, API docs, and installation instructions:
- crw-core — config, error handling, SSRF, types
- crw-renderer — FallbackRenderer, SPA detection, CDP, stealth
- crw-extract — markdown/text/HTML, CSS/XPath, chunking, BM25
- crw-crawl — BFS crawler, robots.txt, sitemap, rate limiting
- crw-server — API endpoints, config, Docker, curl examples
- crw-cli — all CLI flags, format examples, stealth/proxy/JS
- crw-mcp — 7 MCP client setup guides (Claude, Cursor, Windsurf, etc.)