github janreges/siteone-crawler v2.4.1

latest releases: v2.6.1, v2.6.0, v2.5.1...
4 months ago

Highlights

This is a patch release on top of v2.4.0 with one user-facing fix.

🐛 Fixed: robots.txt SEO regression (#105)

In v2.4.0, the SEO Indexing column could report every crawled page as DENY (robots.txt) whenever the site's robots.txt contained a Disallow: / under any user-agent block — which is extremely common for blocking specific bots (e.g. GPTBot, CCBot). v2.3.0 was not affected.

The indexing check now reuses the same robots.txt engine as the crawler itself, so it correctly honors:

  • user-agent scoping (only * / SiteOne-Crawler rules apply to us),
  • Allow: overrides,
  • wildcard (*) and end-anchor ($) matching.

A regression test covers the bot-specific Disallow: / case so it can't silently return.

Thanks to @afu-dev for the precise report and the v2.3 vs v2.4 screenshots.


Downloads

Platform Architecture File
Linux x64 siteone-crawler-v2.4.1-linux-x64.tar.gz
Linux arm64 siteone-crawler-v2.4.1-linux-arm64.tar.gz
Linux x64 (musl/static) siteone-crawler-v2.4.1-linux-musl-x64.tar.gz
Linux arm64 (musl/static) siteone-crawler-v2.4.1-linux-musl-arm64.tar.gz
macOS arm64 (Apple Silicon) siteone-crawler-v2.4.1-macos-arm64.tar.gz
macOS x64 (Intel) siteone-crawler-v2.4.1-macos-x64.tar.gz
Windows x64 siteone-crawler-v2.4.1-win-x64.zip
Windows arm64 siteone-crawler-v2.4.1-win-arm64.zip

Linux packages (glibc — best performance, requires glibc 2.39+)

Format Architecture File
Debian/Ubuntu (.deb) x64 siteone-crawler_2.4.1-1_amd64.deb
Debian/Ubuntu (.deb) arm64 siteone-crawler_2.4.1-1_arm64.deb
Fedora/RHEL (.rpm) x64 siteone-crawler-2.4.1-1.x86_64.rpm
Fedora/RHEL (.rpm) arm64 siteone-crawler-2.4.1-1.aarch64.rpm

Linux packages (musl/static — any Linux, ~50–80% slower)

Format Architecture File
Debian/Ubuntu (.deb) x64 siteone-crawler-static_2.4.1-1_amd64.deb
Debian/Ubuntu (.deb) arm64 siteone-crawler-static_2.4.1-1_arm64.deb
Fedora/RHEL (.rpm) x64 siteone-crawler-static-2.4.1-1.x86_64.rpm
Fedora/RHEL (.rpm) arm64 siteone-crawler-static-2.4.1-1.aarch64.rpm
Alpine (.apk) x64 siteone-crawler-2.4.1-r1-x86_64.apk
Alpine (.apk) arm64 siteone-crawler-2.4.1-r1-aarch64.apk

Quick start

# Extract and run
tar xzf siteone-crawler-v2.4.1-linux-x64.tar.gz
cd siteone-crawler
./siteone-crawler --url=https://example.com

Install via package manager

# Debian/Ubuntu (glibc — Ubuntu 24.04+, Debian 13+)
sudo dpkg -i siteone-crawler_2.4.1-1_amd64.deb

# Debian/Ubuntu (static/musl — older distributions)
sudo dpkg -i siteone-crawler-static_2.4.1-1_amd64.deb

# Fedora/RHEL
sudo dnf install ./siteone-crawler-2.4.1-1.x86_64.rpm

Full Changelog: v2.4.0...v2.4.1

Don't miss a new siteone-crawler release

NewReleases is sending notifications on new releases.