Highlights
This is a patch release on top of v2.4.0 with one user-facing fix.
🐛 Fixed: robots.txt SEO regression (#105)
In v2.4.0, the SEO Indexing column could report every crawled page as DENY (robots.txt) whenever the site's robots.txt contained a Disallow: / under any user-agent block — which is extremely common for blocking specific bots (e.g. GPTBot, CCBot). v2.3.0 was not affected.
The indexing check now reuses the same robots.txt engine as the crawler itself, so it correctly honors:
- user-agent scoping (only
*/SiteOne-Crawlerrules apply to us), Allow:overrides,- wildcard (
*) and end-anchor ($) matching.
A regression test covers the bot-specific Disallow: / case so it can't silently return.
Thanks to @afu-dev for the precise report and the v2.3 vs v2.4 screenshots.
Downloads
| Platform | Architecture | File |
|---|---|---|
| Linux | x64 | siteone-crawler-v2.4.1-linux-x64.tar.gz
|
| Linux | arm64 | siteone-crawler-v2.4.1-linux-arm64.tar.gz
|
| Linux | x64 (musl/static) | siteone-crawler-v2.4.1-linux-musl-x64.tar.gz
|
| Linux | arm64 (musl/static) | siteone-crawler-v2.4.1-linux-musl-arm64.tar.gz
|
| macOS | arm64 (Apple Silicon) | siteone-crawler-v2.4.1-macos-arm64.tar.gz
|
| macOS | x64 (Intel) | siteone-crawler-v2.4.1-macos-x64.tar.gz
|
| Windows | x64 | siteone-crawler-v2.4.1-win-x64.zip
|
| Windows | arm64 | siteone-crawler-v2.4.1-win-arm64.zip
|
Linux packages (glibc — best performance, requires glibc 2.39+)
| Format | Architecture | File |
|---|---|---|
| Debian/Ubuntu (.deb) | x64 | siteone-crawler_2.4.1-1_amd64.deb
|
| Debian/Ubuntu (.deb) | arm64 | siteone-crawler_2.4.1-1_arm64.deb
|
| Fedora/RHEL (.rpm) | x64 | siteone-crawler-2.4.1-1.x86_64.rpm
|
| Fedora/RHEL (.rpm) | arm64 | siteone-crawler-2.4.1-1.aarch64.rpm
|
Linux packages (musl/static — any Linux, ~50–80% slower)
| Format | Architecture | File |
|---|---|---|
| Debian/Ubuntu (.deb) | x64 | siteone-crawler-static_2.4.1-1_amd64.deb
|
| Debian/Ubuntu (.deb) | arm64 | siteone-crawler-static_2.4.1-1_arm64.deb
|
| Fedora/RHEL (.rpm) | x64 | siteone-crawler-static-2.4.1-1.x86_64.rpm
|
| Fedora/RHEL (.rpm) | arm64 | siteone-crawler-static-2.4.1-1.aarch64.rpm
|
| Alpine (.apk) | x64 | siteone-crawler-2.4.1-r1-x86_64.apk
|
| Alpine (.apk) | arm64 | siteone-crawler-2.4.1-r1-aarch64.apk
|
Quick start
# Extract and run
tar xzf siteone-crawler-v2.4.1-linux-x64.tar.gz
cd siteone-crawler
./siteone-crawler --url=https://example.comInstall via package manager
# Debian/Ubuntu (glibc — Ubuntu 24.04+, Debian 13+)
sudo dpkg -i siteone-crawler_2.4.1-1_amd64.deb
# Debian/Ubuntu (static/musl — older distributions)
sudo dpkg -i siteone-crawler-static_2.4.1-1_amd64.deb
# Fedora/RHEL
sudo dnf install ./siteone-crawler-2.4.1-1.x86_64.rpmFull Changelog: v2.4.0...v2.4.1