OpenSERP 0.7.2

Highlights

• New API response envelope for search and image endpoints.
• Unified normalized result model with stable IDs, domain enrichment, classification, metadata, and pagination.
• New output formats: json, markdown, text, ndjson.
• Better locale, language, region, and date-filter handling across engines.
• Major browser runtime improvements: persistent browser connection, isolated contexts, profiles, smart waits, resource blocking.
• Better proxy health handling and captcha isolation.
• New OpenAPI spec, architecture docs, contributing guide, CI, and integration tests.

Added

API response model

• Added JSON response envelopes with:

  • query
  • meta
  • results
  • pagination
  • engines_responded
  • engines_failed

• Added normalized Result model with:

  • stable result IDs
  • result type
  • position metadata
  • engine metadata
  • domain enrichment

• Added dedicated image response envelope and image result structure.

• Added result clustering for /mega/search, grouping same URLs across engines.

• Added Bing redirect URL unwrapping before result hashing.

Search features

• Added locale propagation into browser profile and Accept-Language.
• Added improved query handling with locale support and date filters.
• Improved URL builders and added edge-case coverage.

Browser/runtime

• Replaced generic stealth mode with custom browser profile functionality.
• Added browser profiles and profile coherence tests.
• Added persistent browser connection reuse.
• Added isolated browser contexts per request.
• Added configurable resource/tracker blocking.
• Added smart waits for SERP loading.
• Added fingerprint debug tests and endpoint.
• Added graceful shutdown with drain signal and browser cleanup.

Error handling

• Added typed 400 validation errors with stable reason codes.
• Added limit validation range: 1..100.
• Improved error handling across search engines.
• Added safe HTTP response draining/closing.

Proxy and captcha handling

• Isolated proxy health from captcha/parser errors.
• Replaced proxy re-enable thrashing with quarantine behavior.
• Improved captcha detection selectors.
• Added single source of truth for captcha selectors.
• Captcha solver misconfiguration now exits non-zero.

Docs and release assets

• Added OpenAPI spec.
• Added Swagger UI/docs embedding.
• Added architecture documentation.
• Added contributing guide.
• Updated README to match new API format and features.
• Clarified OpenSERP open-source status and third-party affiliation wording.

CI and tests

• Added GitHub CI workflow.

• Added integration workflow.

• Added integration tests for:

  • Baidu
  • Bing
  • DuckDuckGo
  • Google
  • Yandex

• Added sanitized real SERP HTML fixtures.

• Added raw HTML parser tests.

• Added server endpoint coverage.

• Added browser, profile, captcha, middleware, retry, proxy, locale, and response tests.

Docker

• Refactored Docker setup.
• Switched to chromedp/headless.
• Updated Dockerfile and docker-compose configuration.

Changed

• Search internals now distinguish raw engine results from normalized API results.
• Multi-engine search now returns responded/failed engine metadata.
• Non-JSON output formats bypass JSON response cache to avoid cache pollution.
• Logging is now more structured and includes more useful request/search context.
• Request context now propagates through search, retry, limiter, and browser navigation.
• README examples were updated for the new response shape.

Fixed

• Fixed API response format issues.
• Fixed captcha selector detection problems.
• Fixed CI issues.
• Fixed linter errors.
• Fixed unsafe/unclear engine error handling.
• Fixed proxy health behavior where captcha/parser failures could incorrectly affect proxy status.
• Fixed HTTP response cleanup with safe drain-and-close logic.

Breaking / migration notes

• Search endpoints no longer return bare arrays; they now return response envelopes.
• Consumers should read results from results instead of treating the whole response as an array.
• Error responses now include stable machine-readable reason values.
• limit is now validated and must be between 1 and 100.
• Clients relying on old response structs should update to the new v1 schema.

Full Changelog: v0.6.0...v0.7.2