karust/openserp v0.7.13

on GitHub

This release larger API refresh: Ecosia support, v2.1 response envelopes, HTML parsing endpoints, region targeting, faster mega search modes, and more observable browser/proxy behavior.

Highlights

• Added Ecosia as a supported search engine.

  • New endpoints: /ecosia/search and /ecosia/image.
  • Added Ecosia support to the CLI and server engine registry.
  • Added raw-mode Ecosia web search support.

• Added HTML SERP parser endpoints.

  • Engines can now expose POST /{engine}/parse for parsing raw SERP HTML into the normalized OpenSERP response envelope.
  • Parser logic is shared with browser/raw search paths, reducing selector drift between modes.
  • Supported parser implementations were added for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia.

• Introduced the v2.1 search response shape.

  • meta.version is now 2.1.
  • Results now use explicit result type values such as organic, ad, answer_box, people_also_ask, and related SERP modules.
  • Ads and organic results are ranked separately while position.absolute preserves the mixed on-page SERP position.
  • meta.engines_responded and meta.engine_errors are available for multi-engine requests.

• Added region targeting.

  • New region query parameter is accepted on search and image endpoints.
  • Google, Bing, DuckDuckGo, and Yandex now map the region hint into engine-specific market/location parameters where supported.
  • Cache keys now include normalized region and market data.

• Added mega search execution modes.

  • mode=balanced keeps the existing parallel aggregation behavior and remains the default.
  • mode=any tries engines sequentially until the first successful response.
  • mode=fast chooses the fastest available engine based on circuit-breaker latency history.
  • New dedupe and merge controls tune aggregation behavior.
  • New app.mega_timeout bounds total mega request time and returns partial results when slower engines miss the deadline.

API Changes

• Added response headers:

  • X-Network-Bytes reports counted network bytes for the request.
  • X-Browser-Profile-Id reports the browser profile IDs used during browser-mode requests.

• Added request header:

  • X-Use-Profile lets debug/controlled requests force a known browser profile ID.

• Error responses now include request_id when available.

• Search errors now expose more precise machine-readable codes, including:

  • all_engines_failed
  • circuit_open
  • request_timeout
  • request_canceled

• Mega search failures now include sanitized per-engine error details in meta.engine_errors.

Compatibility Notes

• The public JSON response schema changed from v1 to v2.1.

  • is_ad was removed; use type == "ad" instead.
  • position.page and position.on_page were removed; use pagination plus position.absolute.
  • domain_info booleans such as is_gov, is_edu, and is_social were replaced by domain_info.category.
  • classification fields are now omitted when empty.

• limit now means the number of organic results requested. Ads may be returned in addition to the requested organic limit.

• Query parameters are now trimmed before validation and cache key generation, so whitespace-only queries are rejected consistently and equivalent query inputs share cache entries.

• Mega search cache keys now include mode, dedupe, merge, region, and normalized engine lists. mode=fast responses are not cached.

Search Quality and Parsing

• Improved SERP result classification and ad positioning.
• Centralized selectors across browser, raw, and parse flows.
• Added parser implementations and tests for multiple engines.
• Improved Bing redirect URL unwrapping and result normalization.
• Improved Baidu, Yandex, Google, Bing, DuckDuckGo, and Ecosia parsing coverage.
• Raw search paths now share common HTTP helpers, status classification, TLS behavior, Accept-Language handling, and parser reuse.

Browser and Proxy Improvements

• Reworked browser profile selection with a larger profile catalog, weighted profile selection, runtime browser-version adaptation, locale/timezone matching, and per-session profile stability.
• Added browser profile reporting in logs and response headers.
• Added X-Use-Profile validation for forced profile selection.
• Improved headless browser fingerprint patching, including plugins/mime types, permissions, WebGL/profile surfaces, and Chrome runtime stubs.
• Fixed request-blocking cleanup so browser resource blocking no longer panics during shutdown/cancellation.
• Improved authenticated proxy handling under concurrent browser requests.
• Browser pool stats and proxy lane stats are exposed more consistently in stats responses.

Reliability and Observability

• Added network byte telemetry for browser and raw HTTP paths.
• Circuit breaker stats now track average successful response latency.
• Dedicated and mega endpoints report richer per-engine failure details.
• Mega search can return partial results instead of waiting indefinitely for slow engines.
• Error detail sanitization masks credentialed proxy URLs before returning metadata.

Documentation

• Updated README examples for Ecosia, region targeting, mega modes, and v2.1 response examples.
• Updated OpenAPI version to 2.1.0.
• Documented new headers, error codes, region parameter, mega mode controls, and updated response schemas.
• Updated configuration example with app.mega_timeout, Ecosia engine settings, and current rate-limit defaults.

Fixes

• Fixed Ecosia proxy policy, validation, raw mode, and OpenAPI integration.
• Fixed browser request-blocking cleanup panic.
• Fixed Baidu raw search to reuse the shared parser and classify captcha/empty-result states more reliably.
• Fixed raw HTTP proxy error wrapping so proxy failures are categorized correctly.
• Fixed cache key normalization for query inputs and proxy market signals.

New Contributors

• @miku made their first contribution in #26

Full Changelog: v0.7.2...v0.7.13