The most significant update to Fredy yet 🎉
Until now, Fredy used an XHR-based approach to extract information from various providers. For each run, it sent a GET request to gather the required data. However, this approach had two major drawbacks:
- It relied on a library that is no longer maintained.
- With the growing prevalence of SPAs, XHR requests could no longer retrieve the necessary source code, as it is loaded dynamically.
To address these issues, I decided to rewrite Fredy using Puppeteer (https://pptr.dev/). With Puppeteer, Fredy can control an actual browser and behave like a real user, allowing it to wait for dynamically loaded content.
At the same time, I removed ScrapingAnt. While ScrapingAnt was incredibly useful in the early stages, it has become more of a burden lately. It was originally intended to assist with pages like Immoscout that have advanced bot protection, but even ScrapingAnt is now frequently detected as a bot. Additionally, contacting support at ScrapingAnt has proven nearly impossible.