Summary
This release introduces a targeted improvement to how URL paths are converted into local filesystem paths when mirroring websites. It improves robustness when handling mixed encodings and malformed filenames, while keeping all existing safety protections intact.
Special thanks to @gsrec for reporting the edge cases and providing helpful examples that led to this improvement.
✨ Improvements
- Decodes URL-encoded path segments (e.g.,
%20→ space) - Trims unnecessary whitespace from individual path segments
- Collapses accidental multi-dot sequences (e.g.,
file....jpg→file.jpg) - Maintains traversal protection (
../prevention) - Preserves per-segment and overall path length limits
- Keeps hashing fallback for overly long filenames
🔒 Compatibility
- No changes to crawl logic
- No changes to download behavior
- No changes to threading or queue handling
- No structural refactoring
- Fully backward compatible
This is a minimal and safe enhancement focused strictly on improving path normalization and edge-case filename handling.
What's Changed
- fix: skip non-fetchable URL schemes (mailto/tel/javascript/data/blob) by @PKHarsimran in #16
- 18 wrong url sanitize by @PKHarsimran in #19
- 20 url to local path updates by @PKHarsimran in #21
Full Changelog: v2.1.0...v2.2.0