D4Vinci/Scrapling v0.5

Release v0.5

on GitHub

Scrapling v0.5 upgrades almost everything: fetching, parsing, spiders, browser automation, AI tools, and docs in ten languages 🚀

🚀 New Stuff and quality of life changes

• A much more capable MCP server for AI agents that can solve almost any CAPTCHA (Check the docs):

  • Complete browser workflows: fill forms, click, hover, scroll, use the keyboard, handle dialogs, and wait for results in one ordered sequence. Supports different field types and slower interactions, with actions stopping at the first error.
  • Read and search the current page without reloading it: extract HTML, Markdown, or text from a whole page or one element. Searchable page snapshots show element roles, names, and positions, with focused views that retain the surrounding context.
  • Run custom JavaScript and take focused screenshots: execute scripts, including async work, with access to the page's own JavaScript. Capture individual elements, the viewport, or the full page as PNG or JPEG.
  • Inspect network activity and reuse logins: examine recorded requests and responses, save and restore browser login state, and interact with elements inside open or closed Shadow DOM. Read recorded traffic before closing the session.
  • Use fewer tokens and get clearer errors: leaner tool descriptions and shared guidance reduce repeated text, while browser and request failures reach agents with useful error messages.

• See the requests behind the page. Enable record_requests=True to record completed traffic across loads, actions, and tabs, then inspect saved responses without sending requests again. (Check the docs)

  from scrapling.fetchers import StealthySession
  
  with StealthySession(record_requests=True) as session:
      session.fetch("https://example.com", network_idle=True)
      for response in session.network.search(url_pattern=r"/api/", limit=None):
          print(response.url, response.status, response.body)

  • Filter by URL, method, resource type, or status; retrieve one response by ID. Inspect headers, sent data, API responses, HTTP errors, and redirects through the usual Response API.
  • Retains supported text bodies with size limits, plus metadata and a reason when a body cannot be saved. The default limit is 1,000 requests. Works with sync and async sessions; saved Python history stays readable after close.

• Save a login once and reuse it across runs. save_state(path) and load_state(path) save cookies, local storage, and IndexedDB in Playwright-compatible JSON. Both sync and async browser sessions support them. (Check the docs)

  • Load state before fetching or reloading a page. Requires a started, shared browser context without proxy_rotator; tabs, browser settings, and sessionStorage are not saved.

• Extract open Shadow DOM with pierce_shadow=True, then use CSS or XPath as usual. Available in browser fetchers, sessions, and the CLI's --pierce-shadow option. Preserves slot content without changing the live page. HTML capture excludes closed roots and iframe documents. (Check the docs)

• Choose how your spider explores a site. CrawlSpider and its subclasses, including ContentSpider, support crawl_strategy="bfs" for oldest-first or "dfs" for newest-first scheduling. Priority still wins, and concurrent pages can finish in a different order. (Check the docs)

🐛 Bug Fixes

• Improved Cloudflare solving with delayed-widget detection, fewer unnecessary waits, and bounded passive-challenge and hidden-frame retry loops. Fixes #468 and #509.

• Fixed browser startup from an elevated Windows shell. Fixes #463.

• Kept the built-in stealth flags when adding custom browser flags by @alkaz-nodemaven in #430.

• Improved charset parsing for browser responses, including declarations with different letter case, spaces, or quoted values.

• Stopped logging the absence of a Cloudflare challenge as an error when no challenge needs to be solved. Fixes #459.

• Corrected request fingerprints for query parameters by @yetval in #448, empty or falsy JSON bodies by @ac1982 in #480, and header insertion order by @SarwadnyaMahajan in #466.

• Fixed request data leaking into followed links, including stale body-related headers, by @yetval in #462.

• Prevented blocked responses from being stored as successful development-cache entries by @Kuang-xianxin in #446.

• Preserved checkpoint files when a crawl fails with an exception by @SulimanAbdulrazzaq in #486.

• Corrected robots.txt failure handling so failed lookups are not cached as permission to crawl, by @prateekraawat1 in #443.

• Fixed link extraction with restricted selectors, including empty matches and selectors that target the link element itself, by @SulimanAbdulrazzaq in #452 and #483.

• Fixed sitemap callback routing by @SulimanAbdulrazzaq in #458, and XML parsing with comments or processing instructions by @sup3dev in #450.

• Handled leading byte-order marks in robots.txt and CSV feeds by @SulimanAbdulrazzaq in #453.

• Fixed stealthy_headers handling in spider HTTP sessions, passing the option to the session instead of directly to curl_cffi, by @Yigtwxx in #475.

• Fixed dictionary form encoding for list and tuple values, while preserving request-body precedence and existing None handling, with contributions by @SulimanAbdulrazzaq in #485.

• Fixed case-insensitive merging of session and per-request HTTP headers by @SulimanAbdulrazzaq in #487.

• Fixed CLI requests silently dropping empty or falsy JSON bodies by @SulimanAbdulrazzaq in #484.

• Made Markdown and text file exports consistently use UTF-8 and prevented conversion errors from truncating an existing output file, by @SulimanAbdulrazzaq in #488.

• Fixed curl2fetcher() handling of curl's user-agent and proxy options by @SulimanAbdulrazzaq in #490.

• Fixed Markdown blocks running together when several selected elements are converted through Response.markdown() or CLI extraction, with contributions by @Yigtwxx in #432.

• Fixed hidden-content cleaning for AI output to keep visible content such as opacity:0.95, recognize more hidden styles, and remove hidden slots, with contributions by @Yigtwxx in #460.

• Fixed noise-tag removal stopping early around nested tags by @Yigtwxx in #474.

• Fixed generated CSS and XPath selectors for the first of several same-tag siblings by @yellow-dog-dog in #495.

• Resolved an lxml HTML parser deprecation warning by @afsuyadi in #493.

Docs

• The full website now speaks ten languages: English, Arabic, Spanish, Brazilian Portuguese, French, German, Simplified Chinese, Japanese, Russian, and Korean. Guides, API references, tutorials, and the changelog are translated, with Arabic RTL support and language switches that keep your page and section.
• Clearer guides, README examples, and setup steps, covering checkpoint data, browser encodings, saved state, network history, Shadow DOM, and crawl order.
• Updated the Agent Skill, its examples, references, and downloadable archive for the current browser and MCP tools.
• Moved the changelog to the website and added access to each page's Markdown source.

Project

• Refreshed Scrapling's logo, cover images, and website presentation.
• Expanded tests for browser actions, dialogs, extraction, screenshots, JavaScript, network recording, state files, Shadow DOM, and crawler fixes.
• Added a browser-test marker and separate browser, async, and parallel test runs. Improved Windows checkpoint and CSV test portability by @diegobravtraps-oss in #457.
• Updated the shared multilingual documentation build setup and contribution guidance.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors