domainaware/parsedmarc 11.3.0

on GitHub

Security

• The seen-aggregate-report cache no longer keeps every distinct report in memory for the life of the process (GHSA-9fw6-mwqr-p43m, affects 11.2.2 and earlier). The cache that recognizes a duplicate aggregate report is meant to forget a report after one hour, but the expiringdict package it is built on removes an expired entry only when that same key is looked up again, and otherwise drops its oldest entry only once it holds max_len entries, which was 100 million. A report that is never received again is never looked up again, so in watch mode every distinct aggregate report stayed in memory until the process exited. The cache key was the reporter's name, the Report-ID and the policy domain as JSON, all three chosen by the sender with no length limit below the 32 MiB decompression cap: ten emails each carrying a 5.6 KB gzip attachment with a 5 MiB <report_id> left 64 MiB held by the cache, and none of it was released two hours after the one-hour expiry. Expired entries are now removed each time a report is added to the cache, the cache holds at most 100,000 entries, and the key is a SHA-256 digest of the three fields, so its size no longer depends on the report. Compatibility: the cache is process-local and never saved, so nothing persisted changes; a report is still recognized as a duplicate for one hour. Reported by helixnebulatech.
• Reverse DNS lookups are limited to 120 seconds per aggregate report when they fail or are slow, and to 600 seconds in total (GHSA-rwfh-ww9c-jm9m, affects 11.2.2 and earlier). Each <record> of an aggregate report triggered one blocking reverse DNS (PTR) lookup of its source IP, one record after another, with no bound on the number of lookups or the time they took, and the sender chooses those IPs. An IP whose reverse zone never answers costs the whole DNS lifetime, dns_timeout times the number of nameservers: 2 s × 4 default nameservers = 8 s. Repeating one such IP did not help, because a result with no PTR whose AS domain is not in the reverse DNS map is deliberately kept out of the IP address cache, so the same IP was looked up again in every record. Measured against a nameserver that never answers, parsing took 2.2 s per record with one nameserver and 8.2 s per record with four, so a 1 MB report of about 3,500 records held up processing for hours, and no other message was processed meanwhile. Lookups that return no PTR, and lookups that take at least one second whether or not they return a name (a sender who controls the reverse zone could otherwise answer every query with a name just before the timeout), now share a budget of 120 seconds per report. Fast lookups that find a PTR are not counted toward it, so a large legitimate report keeps its enrichment, but every lookup counts toward a second budget of 600 seconds per report for all reverse DNS lookups, since a sender who controls the reverse zone could otherwise answer every query just under one second (5,000 distinct source IPs resolving in 100 ms each use 500 seconds). Once either budget is spent, the report's remaining records are enriched from the local IP database and reverse DNS map only, and one warning is logged: "Reverse DNS lookups for this report failed or were slow for 120 seconds; the remaining records are enriched from the local IP database and reverse DNS map only", or "Reverse DNS lookups for this report took more than 600 seconds; ..." with the same ending. Results produced after the limit are not cached, so those IPs are looked up normally in the next report. Each distinct source IP is also looked up at most once per report. Deployments with offline = True make no DNS queries and were never affected. Reported by helixnebulatech.
• A report value that PostgreSQL rejects no longer holds back the whole batch (GHSA-fxr7-7h9q-v949, affects the PostgreSQL output in 11.2.2 and earlier). The PostgreSQL output inserted report values into typed columns without checking them, and three values a sender controls were rejected by the server or the driver: an aggregate <count> above 2,147,483,647 (integer out of range in the INTEGER message_count column), an SMTP TLS sending-mta-ip or receiving-ip that is not an IP address (invalid input syntax for type inet), and a NUL character in a failure report's sample, for example a Subject: of =?utf-8?q?a=00b?= (PostgreSQL cannot store NUL in text or jsonb). A fourth trigger, an SMTP TLS start-datetime PostgreSQL cannot read, has mostly been rejected by the parser since 11.2.2; the remainder (partial dates such as 2016, which the parser still accepts and PostgreSQL may reject) now raise PostgreSQLRejectedReport like any other rejected value and no longer hold back the batch. Each rejection was recorded as an output error, which marks the whole batch as not saved, so none of the batch's messages was archived, including legitimate reports from other senders. In watch mode the batch was retried max_unsaved_retries times and then moved to Archive/Unsaved; in one-shot mode (cron) the retry counter restarts with every run, so the batch stayed in the inbox and its legitimate reports were sent again to every output that does not deduplicate on every run. An aggregate record whose count is outside 0 to 2,147,483,647 is now skipped with a "Could not parse record" warning (a report with only that one record becomes invalid, as with any other bad record), and an SMTP TLS session count outside that range makes the report invalid; the bound matches PostgreSQL's INTEGER count columns (the Elasticsearch and OpenSearch outputs install no mapping for any count, so those clusters map each one dynamically as long, a range the bound also keeps every count within). An SMTP TLS sending-mta-ip or receiving-ip that is not an IP address (or carries an IPv6 zone ID) is dropped from the failure details with a warning, which also keeps it out of the Elasticsearch and OpenSearch ip fields. An IPv6 zone ID (fe80::1%eth0), which the inet type cannot hold, is removed from an aggregate record's source_ip and a failure report's Source-IP (everything from the first %) with a warning naming the field, and the record or report is kept with the bare address. The PostgreSQL output replaces NUL with U+FFFD, the replacement character, in every text, text array and JSONB value it writes. Any other value PostgreSQL rejects (a psycopg.DataError) now raises the new PostgreSQLRejectedReport, a subclass of PostgreSQLError whose message names the report; the CLI logs it as an error and skips that report in the PostgreSQL output only, and the batch is saved and archived as usual. With fail_on_output_error set it is an output error that stops the run, as before. The in-memory max_unsaved_retries counter in one-shot mode is unchanged: the batches it still holds back are the ones an output could not save at all, such as an unreachable server, which the documentation describes as the intended safe direction. Reported by helixnebulatech.
• The PostgreSQL output no longer sends the database password in plain text when an attacker declines TLS (GHSA-xvp7-7762-gx9g, affects the PostgreSQL output in 11.2.2 and earlier). PostgreSQLClient connected without an sslmode, so libpq used its default, prefer: it asks the server for TLS and, when the answer is no, continues over plain TCP and answers whatever authentication request comes next. That answer arrives before any encryption, so an on-path attacker who declined TLS and then requested a clear-text password received the database password (CWE-319), and with it every stored report. The output now connects with sslmode=require by default, so a connection without TLS fails before the user name or password is sent. The new [postgresql] sslmode option (and the sslmode argument of PostgreSQLClient) sets any libpq mode, and sslrootcert names the CA certificate that verify-ca and verify-full check the server against. When sslmode is not set, an sslmode in connection_string or the PGSSLMODE environment variable is respected. A warning is logged at startup when a TCP host other than localhost or a loopback address is used with a mode that does not verify the server certificate (require with an existing sslrootcert file is not warned about, because libpq then verifies the certificate as verify-ca does). With a connection_string, nothing parsed from the string reaches the log or an error message: a connection failure is reported with a fixed message (plus the server's SQLSTATE code, when it sent one) instead of the driver's, and the startup warning names neither the server nor any other value from the string. libpq ends a URI's user name and password at the first @ or /, so the rest of a password containing either becomes the host, port, database name, or a query parameter, which the warning and the driver's errors would otherwise repeat (postgresql://u:SECRET/x@db/d failed with invalid integer value "SECRET" for connection option "port"). The separate host, port, user, password, and database settings keep the driver's message, with URL credentials masked. As a convenience, a URI that libpq would parse that way (an unencoded @ in its host list or database name, or in a query value before the first /, or an @ after a host whose port is neither empty nor a number) is refused with a fixed error before anything is logged or a connection is attempted; this recognizes the common mistake but is not what keeps the password out of the logs. Percent-encode @ and / in a URI's user name or password (%40, %2F). A keyword/value connection string is not checked, because its password is a keyword of its own. Any other error while connecting, such as the UnicodeEncodeError psycopg raises with the whole connection string it built when the password holds a character it cannot encode, is replaced with a fixed message as well, and the CLI no longer chains the PostgreSQL client's error to the RuntimeError it raises. A table creation failure is likewise reported with a fixed message (plus the SQLSTATE) instead of the server's, which can quote a setting from the string's options (tablespace "X" does not exist for -c default_tablespace=X), and a connection whose setup or table creation fails is closed instead of left open. Compatibility: a server that offers no TLS at all now fails with libpq's "server does not support SSL, but SSL was required". Set [postgresql] sslmode = disable (or prefer) to allow it again, recommended only for a Unix socket or the loopback address. require still accepts any certificate, so a remote server should use sslmode = verify-full with sslrootcert. Reported by helixnebulatech.
• An Elasticsearch or OpenSearch password containing /, ? or # no longer appears in the startup error (GHSA-74xv-rmpx-2pjm, affects 11.2.2 and earlier; an incomplete fix for GHSA-hjr3-rx4c-24cv). 11.2.0 masked the credentials in hosts URLs logged with --debug, but not in the error raised when the client cannot be built. When the password in a hosts URL contains an unencoded /, ? or #, the URL cannot be parsed, and the client library's error quotes the whole URL (Elasticsearch) or the start of the password (OpenSearch). parsedmarc repeated that text at ERROR level, without --debug, on the console, in log_file and in any log pipeline: Output client error: Elasticsearch: Could not parse URL 'https://dmarc:<password>@es.example:9200' and Output client error: OpenSearch: Port could not be cast to integer value as '<start of password>'. The error was also chained to the library's, so a failed SIGHUP reload, which logs the full traceback, printed it a second time. A hosts value that cannot be parsed now fails with a fixed message that names no part of it: "Elasticsearch: could not build a client from the hosts setting (details withheld because they may include credentials; URL-encode special characters in the password, or use the separate user and password settings)", and likewise for OpenSearch. Any other Elasticsearch or OpenSearch startup error keeps its text with URL credentials masked, and neither is chained to the original error any more. Reported by helixnebulatech.
• The downloaded IP database is cached in a private per-user directory and checked before it is used (GHSA-5f56-8ph8-v9q5, affects 9.6.0 through 11.2.2). load_ip_db() cached the IPinfo Lite database it downloads at a fixed path in the shared system temporary directory, /tmp/parsedmarc/ipinfo_lite.mmdb on Linux. Another local user could create /tmp/parsedmarc first; parsedmarc accepted the existing directory, wrote the download through a symlink planted at ipinfo_lite.mmdb.tmp, and used whatever file was at ipinfo_lite.mmdb whenever the download was skipped or failed, which that user could force, with no check of its owner or content. That let them overwrite any file the parsedmarc user can write (its configuration, a Gmail token.json, an output file) with the download, or supply the MMDB file that decides every source IP's country and ASN and that maxminddb then parses. The database is now cached in $XDG_CACHE_HOME/parsedmarc or ~/.cache/parsedmarc (%LOCALAPPDATA%\parsedmarc\cache on Windows), created with mode 0700 and used only if it is a real directory, not a symlink, that the user can write to and, on POSIX, owns and is not writable by its group or by others. Otherwise parsedmarc logs a warning and uses a new private temporary directory for that run, downloading the database again on every run, and never changes the permissions of a directory it did not create. The download is written to a new file created exclusively with mkstemp(), opened with maxminddb to check that it is a valid MMDB, and only then moved into place with an atomic os.replace(), which replaces a symlink rather than following it. A cached copy is used only if it is a regular file and, on POSIX, owned by the user; otherwise the bundled copy is used. The old /tmp/parsedmarc/ipinfo_lite.mmdb is no longer read, and can be removed by hand. Only multi-user hosts whose temporary directory is shared were exposed; the Docker image and systemd units with PrivateTmp= were not. Reported by helixnebulatech.
• One aggregate report too large for the Logs Ingestion API no longer stops the Log Analytics output from publishing the batch (GHSA-rg4h-58fm-8pc9, affects the Log Analytics output in 11.2.2 and earlier). The Azure Monitor ingestion SDK packs whole reports into upload calls of up to 1 MiB and never splits a report, so a report whose JSON is larger than that is sent as a call of its own, and the Logs Ingestion API rejects any call over 1 MB. An aggregate report of a few thousand records is that large, while its gzipped XML is only a few kilobytes. The rejected upload raised an error before the failure and SMTP TLS streams were sent, and parsedmarc treated it as an output error: the whole batch, legitimate reports from other senders included, was retried and finally moved to the Unsaved folder. A report whose JSON is over 1,000,000 bytes is now left out with a warning that names it, and the other reports are uploaded. Each of the three streams is now published even when another fails; a failed upload is still reported as an output error, naming every stream that failed. Reported by helixnebulatech.

Added

• [postgresql] sslmode and sslrootcert options (and PostgreSQLClient arguments) to choose the TLS mode of the PostgreSQL connection and the CA certificate that verifies the server.

Changed

• mailsuite 2.3.7 or later is now required. It fixes mailsuite's parse_email() keeping the carriage returns of a CRLF message's address headers, which left a trailing \r in the from, sender and delivered-to values, or added an extra entry to to, cc, reply-to and bcc when the value ended with a comma; empty values are now treated as absent for both CRLF and LF input (seanthegeek/mailsuite#74). parsedmarc parses report emails with its own code, not mailsuite's parse_email().
• parse_report_file() now refuses an input larger than 32 MiB before reading it into memory (#949). The size cap (MAX_DECOMPRESSED_REPORT_SIZE) was applied only after the whole input had been read, so a path, a bytes object or a file object larger than the cap was held in memory first. The size of a path and the length of a bytes, bytearray or memoryview input are now checked up front, and a file object is read at most one byte past the cap (it is left positioned there). The input is rejected with ParserError and the same message as before (the report is larger than 32 MiB). A valid compressed report is smaller than its decompressed form, so the one cap applies to both.
• A report dated more than two days in the future is now rejected (#947). With Elasticsearch or OpenSearch output a report's own date names its dated index (dmarc_aggregate-YYYY-MM-DD), and nothing compared that date with the present, so a report with <begin>4102444800</begin> created an index for the year 2100. An aggregate report whose begin or end date (a report split per day names an index from each day it covers), an SMTP TLS report whose start-datetime, or a failure report whose Arrival-Date is more than 2 days ahead of the current time (MAX_REPORT_FUTURE_DAYS) is now rejected with InvalidAggregateReport, InvalidSMTPTLSReport or InvalidFailureReport, so a mailbox message is moved to the Invalid folder. The two days absorb a reporter's clock skew and time-zone mistakes; dates in the past are not bounded, so archives of old reports still parse.
• Python 3.10 is no longer supported, and Python 3.15 is. Python 3.10 reached end of life in October 2026; requires-python is now >=3.11. Python 3.15 is now tested in CI and listed as supported.

Bug fixes

• One invalid report attachment no longer stops the rest of an mbox file from being parsed. parse_report_email() raised a bare ParserError for a message whose aggregate XML or SMTP TLS JSON attachment was invalid (for example an aggregate report whose date range spans more than 366 days), and the mbox loop only skipped InvalidDMARCReport, so that one message aborted the whole file. It now raises InvalidDMARCReport, or InvalidSMTPTLSReport for an SMTP TLS attachment (both still ParserError subclasses), and the mbox loop skips both with a warning and parses the rest. A message that cannot be parsed at all still stops the file with a ParserError. Mailbox connections were not affected. Reported by mjatin2007.
• Unknown child elements under an aggregate record's <identifiers> and inside a <reason> are no longer copied into the parsed report (#948). _parse_report_record() copied both mappings whole, so an element that RFC 9990 Appendix A and RFC 7489 Appendix C do not define reached the JSON, Kafka, S3 and webhook outputs with whatever structure the report gave it. Only header_from, envelope_from and envelope_to, and a reason's type and comment, are kept. Compatibility: a consumer of those outputs that read such an extra key no longer receives it.
• The Elasticsearch and OpenSearch aggregate document classes declare message_count and original_timespan_seconds as long. Both were written = Integer without parentheses, which assigns the field class itself and declares no field, so reading either attribute from an _AggregateReportDoc returned the class instead of the stored value. They are now Long(), not Integer(), because that is the type the clusters already hold: parsedmarc does not install the document classes' mappings (the indexes and templates it creates declare only the date fields), so Elasticsearch and OpenSearch have always mapped both fields dynamically as long. The documents sent are unchanged, and existing and new indexes are unaffected.