nicobailon/pi-web-access v0.41.0

on GitHub

Some sites turn away AI fetchers at the door, so fetch_content could never read them directly. You can now pick the User-Agent that page fetches send, which gets those pages back without needing a fallback provider. If you run SearXNG on your own machine, it now works out of the box, without a workaround that also opened every other local service to page fetches. Nothing changes unless you set the new option.

Highlights

• Read sites that block AI fetchers: set your own User-Agent with fetchContent.userAgent.
• Use a SearXNG instance on localhost without opening other local services to page fetches.
• A bad User-Agent setting fails with a clear config error and never sends the page to another provider.

Highlights

• Sites that turn away AI fetchers can now be read directly: set your own User-Agent for page fetches.
• A SearXNG instance on your own machine works without opening every local service to page fetches.
• A bad User-Agent setting fails with a clear config error and never sends the page to another provider.

Added

• Set fetchContent.userAgent in web-search.json to choose the User-Agent sent on page fetches. Use it for sites that reject the default OpenAI File Downloader, XaiImageApiFetch/1.0, such as some Australian government sites. Leaving it unset keeps the default. A value that is empty or not printable ASCII fails with a config error before anything is fetched. Thanks to @r1ckyIn for issue #568.

Fixed

• A SearXNG instance on the same machine (searxngBaseUrl on localhost, 127.0.0.0/8, or ::1) now works without adding loopback to ssrf.allowRanges. That workaround also let fetch_content reach every local service. Only the configured SearXNG address is allowed, so a redirect to another local service is still blocked. Thanks to @mikeydiamonds for PR #569.