v0.0.31 introduces two powerful new robots to the Maxun engine: Crawl and Search. These features allow you to automate data discovery at scale, whether you are mapping entire domains or researching the web via natural language.
🕸️Crawl
Stop manually feeding URLs. Crawl intelligently discovers and extracts entire websites.
- Intelligent Discovery: Uses both Sitemap parsing and Link following to find every relevant page.
- Granular Scope Control: Target exactly what you need with Domain, Subdomain, or Path-specific modes.
- Advanced Filtering: Use Regex patterns to include or exclude specific content (e.g., skip
/admin, target/blog/*). - Depth Control: Define how many levels deep the robot should navigate from your starting URL.
Deep_Crawl.mp4
🔍 Search
Turn search engine queries into structured datasets.
- Query Based: Search the web with a search query - same as you would type in a search engine.
- Dual Modes: Use Discover Mode for fast metadata/URL harvesting, or Scrape Mode to automatically visit and extract full content from every search result.
- Recency Filters: Narrow down data by time (Day, Week, Month, Year) to find the freshest content.
Deep_Search.mp4
Hybrid Access
Manage these robots through our no-code dashboard or trigger them programmatically via our Developer SDK.
What's Changed
- fix: remove
onCancelbutton from robot config page by @amhsirak in #929 - feat: cleaner alert notifs by @amhsirak in #927
- feat(ui): match theme of standard & outlined alerts by @amhsirak in #928
- feat: !show rows per page for runs & robots by @amhsirak in #930
- fix: show formats selected for scrape by @amhsirak in #931
- chore: pass prompt by @RohitR311 in #932
- feat: auto name lists for ai mode by @RohitR311 in #933
- feat: !wrap robots table w table head by @amhsirak in #934
- feat: display api key creation date by @amhsirak in #935
- feat: run scroll customization by @amhsirak in #936
- feat(core): real time progress update for runs by @amhsirak in #937
- feat: add crawl and search by @RohitR311 in #938
- chore: pre-release v0.0.31 by @amhsirak in #940
Full Changelog: v0.0.30...v0.0.31