Main changes
HTML parser rework
Rspamd now parses HTML using a DOM model, so it can build and construct a parsed tree of HTML tags instead of a simple ad-hoc parsing. Adding this feature was quite untrivial, as it required full rewrite of the HTML related code. However, we can see that the new parser can deal with emails in HTML format that were completely ruined by a previous parser. Furthermore, better representation of HTML could lead to more sophisticated rules that take HTML structure into account.
CSS parser
Rspamd now has an embedded CSS parser that is currently limited to simple selectors but it could be extended in future. In conjunction with the modern HTML parser, CSS support is very valuable to extract the content from emails and distinguish visible and invisible content precisely.
Amazon S3 support
Rspamd now includes AWS API support (e.g. their signatures schema) which allows to interact with Amazon cloud services directly from Lua API. A simple plugin that stores all messages in AWS S3 cloud has also been written.
DMARC reporting rework
Rspamd DMARC reporting has proven to be troublesome in the previous version. Therefore, I have decided to move reports sending logic to a dedicated tool called rspamadm dmarc_report. The usage of this tool and DMARC reports in principle is already documented.
DMARC munging support
From version 3.0, Rspamd supports DMARC munging for the mailing list. In this mode, Rspamd will change the From: header to some pre-defined address (e.g. a mailing list address) for those messages who have valid DMARC policy with reject/quarantine that would otherwise fail during mailing list forwarding. An example of this technique is defined here.
External relay plugin
Many plugins, such as SPF, are using IP address provided by an MTA to Rspamd directly. However, in many cases there is a trusted relay that do some initial processing and hides the actual sender's IP address. With help of this new plugin, Rspamd can now 'fix' this issue by treating sender's IP as it is reported by that trusted relay.
Bayes export tool
Rspamd now allows to save and restore bayes tokens using rspamadm bayes_dump subcommand. This feature allows to move tokens between instances of Rspamd, to merge them and to analyse them manually.
Pyzor support
Rspamd now supports Pyzor via external services plugin, thanks to @defkev for this work!
Monitoring rework
Monitoring tools are now awaken less frequently reducing load on the external modules.
Other fixes and improvements
Core and API
- Fixed retries on broken maps servers
- Fixed handling of spaces in MIME From domains
- Fixed handling of invalid IDN domains containing 8-bit characters
- Fixed if..else..elseif handling in Lupa templates
Plugins
- Neural: fixed profile filtering (offline) & unwanted retraining (online)
- Fixed escaping in Clickhouse plugin
- Fixed missing config schema & reconfigurability in RBL plugin (disabling RBLs; selectors)
Rules
- Fixed CTYPE_MIXED_BOGUS for text attachments
- Fixed PCRE-mode handling of BITCOIN_ADDR rule
- Fixed REPLYTO_ADDR_EQ_FROM for normalised addresses