What's changed
New features
- Now the
Responseobject returned from all fetchers is the same as theAdaptorobject except it has these added attributes:status,reason,cookies,headers, andrequest_headers. Allcookies,headers, andrequest_headersare always of typedictionary.
So your code can now become like:Instead of before>> from scrapling import Fetcher >> page = Fetcher().get('https://example.com', stealthy_headers=True) >> print(page.status) 200 >> products = page.css('.product')
But I have left the>> from scrapling import Fetcher >> fetcher = Fetcher().get('https://example.com', stealthy_headers=True) >> print(fetcher.status) 200 >> page = fetcher.adaptor >> products = page.css('.product')
.adaptorproperty working for backward compatibility. - Now both the
StealthyFetcherandPlayWrightFetcherclasses can take aproxyargument with the fetch method which accepts a string or a dictionary. - Now the
StealthyFetcherclass has theos_randomizeargument with thefetchmethod. If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
Bugs Squashed
- Fixed a bug that happens while passing headers with the
Fetcherclass. - Fixed a bug with parsing JSON responses passed from the fetcher-type classes.
Quality of life changes
- The text functionality behavior was to try to remove HTML comments before returning the text but that induced errors in some cases and made the code more complicated than needed. Now it has reverted to the default lxml behavior, you will notice a slight speed increase to all operations that counts on elements' text like selectors. Now if you want Scrapling to remove HTML comments from elements before returning the text to avoid the weird text-splitting behavior that's in lxml/parsel/scrapy, just keep the
keep_commentsargument set to True as it is by default.
Note
A friendly reminder that maintaining and improving Scrapling takes a lot of time and effort which I have been happily doing for months even though it's becoming harder. So, if you like Scrapling and want it to keep improving, you can help by supporting me through the Sponsor button.