Reliability release. The main theme is honest error reporting: engines now tell you why a search returned nothing instead of handing back an empty list. Plus batch URL extraction and a rebuilt browser fingerprinting layer.
Highlights
Engines no longer silently return empty results. Captcha pages, soft blocks and genuine "no results" pages are now classified separately and surfaced as captcha_detected, blocked or empty_result error codes. The detection is shared between the browser and raw HTTP paths, so /<engine>/search and /<engine>/parse behave the same across Google, Bing, DuckDuckGo, Yandex, Baidu and Ecosia. Google's /sorry/ interstitial is detected explicitly. This addresses the confusing empty responses reported in #36 and #33.
POST /extract/batch - extract content from up to 20 URLs in one call, returning the {page_content, metadata} array shape Open WebUI's external web loader expects. A URL that fails validation or extraction gets metadata.error and empty content instead of failing the whole batch; each item carries its own took_ms. Contributed by @AIWintermuteAI in #39.
DuckDuckGo pagination fixed. Non-zero offsets returned wrong results because DDG's pagination doesn't work off URL params - the engine now drives the "More results" button. Fixes #40, contributed by @subediparas5 in #41.
Browser fingerprinting moved to native CDP. Dropped the injected patch.js and localized profile files in favor of CDP-level overrides, which is both faster and less detectable. Docker image now ships an updated Chrome.
Other changes
- Proxy sessions are scoped to credentials, so rotating proxies no longer leak sessions between accounts
err_tunnel_connection_failedis now treated as a retryable proxy network error- New
block_resourcesandblock_trackersconfig options to skip heavy subresources in browser mode X-Proxy-AttemptsandX-Use-Profileadded to the default CORS allow-listdocker-compose.yamlnow mountsconfig.yamlby default, with proxy config examples- Fixed a YAML parse error in the OpenAPI spec;
/extract/batchdocumented and exempted from the default request timeout
Upgrade notes
proxies.allow_request_proxy_urlnow defaults totrue. If you expose OpenSERP publicly and don't want callers supplying their own proxy URL, set it back tofalse.- Searches that previously returned
200with an empty array may now return an error withcaptcha_detected/blocked. Handle those codes if your client treats empty as success.
Contributors
Thanks to @AIWintermuteAI and @subediparas5 - both first-time contributors.
Full Changelog: v0.8.6...v0.8.12