janreges/siteone-crawler v2.4.0

on GitHub

SiteOne Crawler v2.4.0 is a big step up for website quality analysis: pure-Rust TLS analysis, a much deeper scoring model with actionable fixes, a CI/CD-ready quality gate, and several crawl/export improvements.

Highlights

• Pure-Rust SSL/TLS analysis — a non-validating handshake inspects the full certificate chain (expired, self-signed, mismatched, weak SHA-1/MD5 signatures, weak keys, validity window) and probes supported protocol versions (SSLv3/TLS 1.0–1.3), BadSSL-inspired.
• Much richer quality scoring — the 5-category score (Performance / SEO / Security / Accessibility / Best Practices) now reflects far more signals the crawler already collects, and every deduction carries an actionable "how to fix" hint.
• CI/CD-ready quality gate — baseline regression checks, fail/ignore by finding code, JUnit XML reports and GitHub Actions annotations make --ci first-class in pipelines.
• "Fix first" panel in the HTML report — an impact-ranked list of what costs the most score, each with its remediation tip.

New & expanded analysis

• Security: CSP directive analysis (flags unsafe-inline/unsafe-eval/wildcard sources, nonce-aware), Cross-Origin headers (COOP/COEP/CORP), broader mixed-content detection (active vs passive), and a fix so every Set-Cookie header is evaluated (not just the first).
• SEO: on-page quality findings — rel=canonical presence + off-host mismatch, title/meta-description length & missing, and a noindex site-wide detector (with noindex pages excluded from on-page checks to avoid false positives).
• Performance & caching: cache-effectiveness scoring for static assets, per-page page-weight & request budgets (HTTP Archive-derived), and a p90 HTML-document response-time metric (a fairer "what users feel" proxy than the all-asset average).
• Accessibility: real parser-impact structural checks (duplicate id, dangling ARIA/label references) replacing a dead rule, plus fixes to four common false positives (decorative alt="", wrapping <label>, icon-only controls, semantic-HTML landmarks).

New CLI options

• --url-list=<file> — crawl a bounded set of URLs from a plain-text file (one URL per line).
• --ignore-html-comments — skip links found inside HTML comments.
• --offline-export-no-url-rewriting — keep URLs verbatim in the offline export.
• CI gate: --ci-baseline, --ci-max-score-drop, --ci-fail-on-code, --ci-ignore-code, --ci-junit-file, --ci-github-annotations (see the README CI/CD section).

Fixes

• Offline export: rewrite downloaded external/CDN assets to local paths; keep the path separator when resolving relative asset URLs.
• HTML report: wrap long URL columns instead of overflowing.
• Meta description/keywords extraction is now attribute-order independent.
• --url-list: validate entries, warn on skipped lines, validate the base URL.

Quality & maintenance

• Major dependency upgrade to the latest versions; Rust toolchain pinned to 1.96 via rust-toolchain.toml.
• ~50 new unit/integration tests across the new analysis and CI features.

Downloads

Linux packages (glibc — best performance, requires glibc 2.39+)

Linux packages (musl/static — any Linux, ~50–80% slower)

Quick start

# Extract and run
tar xzf siteone-crawler-v2.4.0-linux-x64.tar.gz
cd siteone-crawler
./siteone-crawler --url=https://example.com

Install via package manager

# Debian/Ubuntu (glibc — Ubuntu 24.04+, Debian 13+)
sudo dpkg -i siteone-crawler_2.4.0-1_amd64.deb

# Debian/Ubuntu (static/musl — older distributions)
sudo dpkg -i siteone-crawler-static_2.4.0-1_amd64.deb

# Fedora/RHEL
sudo dnf install ./siteone-crawler-2.4.0-1.x86_64.rpm

What's Changed

• feat: enhance regex for meta description and keywords extraction to s… by @nlemoine in #97
• feat: add --url-list to crawl a bounded set of URLs from a file by @afu-dev in #102

New Contributors

• @nlemoine made their first contribution in #97
• @afu-dev made their first contribution in #102

Full Changelog: v2.3.0...v2.4.0