github janreges/siteone-crawler v2.3.0

latest releases: v2.6.1, v2.6.0, v2.5.1...
7 months ago

New features

  • --offline-export-preserve-urls — preserve original URL format in offline-exported HTML, CSS, and JS files. Instead of rewriting links to relative file paths (e.g., ../../../designy.html), same-domain links become root-relative (/designy/classic) and cross-domain links stay as full absolute URLs. This is ideal for processing exported HTML with siteone-chunker and RAG pipelines, where links in the resulting markdown chunks must resolve to the actual production website.

Example

# Export with preserved URLs
siteone-crawler --url=https://example.com --offline-export-dir=./export --offline-export-preserve-urls

# Then process with siteone-chunker — links like [About](/about) resolve to the live site
siteone-chunker --html-input=./export --html-rules=rules.yaml

URL conversion rules

Link type Before (default) After (with flag)
Same-domain absolute ../../about.html /about
Same-domain with path ../../../designy/classic.html /designy/classic
Cross-domain https://other.com/page https://other.com/page
Fragment-only #section #section
Special schemes mailto:, tel:, javascript: unchanged

The file naming and directory structure of the export itself remains unchanged — only the href/src attribute values in the exported files are affected.

Full Changelog: v2.2.0...v2.3.0

Don't miss a new siteone-crawler release

NewReleases is sending notifications on new releases.