Maintenance release. The main fix: Google results point to real pages again - encrypted /goto links are resolved to destination URLs. Also new: a switch to turn engines off, correct links in extracted content, and configurable CLI logging.
Highlights
Google /goto links resolve to destination URLs. Google now gives automated clients encrypted /goto?url=... redirects instead of result links, so url pointed at google.com and URL-based rank tracking and /mega dedup broke (#53). OpenSERP now resolves each one with a single redirect request: no cookies, the destination page is never fetched, and ads are never touched. It's on by default for /google/* and /mega/*, in browser and raw mode. Use resolve_urls=false to turn it off, or resolve_urls=N to resolve only the top N organic results. The same setting is google.resolve_urls in config and --resolve-urls in the CLI. If Google pushes back (403/429, captcha or consent page), resolution through that IP or proxy pauses for 10 minutes, and results keep the wrapper URL until then.
Wrapped links keep their attribution. When a link stays wrapped (resolution off, paused or failed), OpenSERP recovers the domain and favicon URL from the visible breadcrumb or site name and generates a best-effort stable result ID. Plaintext /url?q= redirects are unwrapped locally without an extra request. Contributed by @ramgalin in #51.
Turn engines off. Set <engine>.disabled: true in config, or OPENSERP_<ENGINE>_DISABLED=true, e.g. OPENSERP_BING_DISABLED=true. A disabled engine gets no endpoints and is left out of /mega, in browser and raw mode. serve refuses to start if every engine is disabled. Requested in #45, contributed by @hasandenizuk in #62.
Relative links in extracted content now resolve correctly. Extraction honors HTML <base href> and the final page URL after redirects, fixing links, images, and canonical URLs in raw and rendered modes, including full-page output. Fixes #42, builds on the base URL fix contributed by @Yoomitech in #43.
Configurable CLI log levels. Set --log_level error|warn|info|debug|trace, OPENSERP_APP_LOG_LEVEL, or app.log_level in config. CLI commands default to warn; serve defaults to info. Logs stay on stderr and results on stdout. Initial support contributed by @Spidy092 in #52.
Other changes
google.resolve_concurrency,google.resolve_timeoutandgoogle.resolve_cooldowntune redirect resolution (defaults: 4, 3s, 10m)- Cache keys include the
resolve_urlssetting - Google raw mode parses and classifies each page once, keeps original ranks with
start, and dedups after resolving links - Missing optional config files no longer produce warnings
- Explicit verbosity flags correctly override invalid log-level values from env/config
- Extraction metadata and content share the same charset-aware HTML document
- Long result display URLs are truncated without breaking UTF-8
- JS and Python quickstart examples print the request ID
resolve_urlsdocumented in README and the OpenAPI spec- Added regression coverage for attribution, redirect resolution, extraction URL resolution, engine toggles, and logging
Upgrade notes
resolve_urlsis on by default. Each Google search now makes extra requests to Google (one per wrapped organic link, 4 at a time, 3s budget) through the same exit as the search, which adds some latency. To keep the old behavior, passresolve_urls=false, or setgoogle.resolve_urls: false/OPENSERP_GOOGLE_RESOLVE_URLS=false.- Malformed or unreadable config files now stop startup instead of continuing with defaults. An absent optional config file is still allowed.
--log_leveltakes precedence over-q,-v, and-d. Explicit-q/-v/-dflags take precedence over env/config log levels.
New Contributors
- @ramgalin made their first contribution in #51
- @Yoomitech made their first contribution in #43
- @Spidy092 made their first contribution in #52
- @hasandenizuk made their first contribution in #62
Full Changelog: v0.8.12...v0.8.14