Critical Fixes
- Recursive Sitemap Parsing: Fixed an issue where sites using "Sitemap Indexes" (e.g., RecipeTinEats, IAmAFoodBlog) would return "0 URLs found". The scraper now correctly drills down into sub-sitemaps.
- Smarter Recipe Detection: Added fallback detection for recipes using CSS classes (
wp-recipe-maker,tasty-recipes) even if standard JSON-LD metadata is missing or broken. - Robots.txt Support: Now checks
robots.txtfor the official sitemap location before guessing standard paths.
Under the Hood
- Robust Networking: Implemented
HTTPAdapterwith automatic retries to handle flaky server connections. - Environment Variables: Full support for loading variables from a
.envfile for easier local testing.