domainaware/parsedmarc 11.2.0 on GitHub

New features

Grafana Loki output. A new [loki] section pushes every aggregate, failure, and SMTP TLS report record as one JSON log line to the Loki push API (/loki/api/v1/push), using the same flat records as the syslog and GELF outputs plus report_type and domain. Streams are labeled job=parsedmarc and host=<FQDN>, overridable and extendable with labels, plus the record fields named in label_fields (default report_type, domain). Entries are timestamped at ingest by default, or at the begin of the reported period with timestamp = report_begin. Basic authentication, a bearer token, and a tenant ID (X-Scope-OrgID) are supported, and large reports are split into pushes of batch_size lines. Redirects are not followed, so a 3xx response counts as a failed push. A failed push raises LokiError, so it counts as a failed save and holds the batch's mailbox messages back for a retry. Needs only httpx, so it is part of the base install.
Grafana dashboard for the Loki output. dashboards/grafana/Grafana-DMARC_Reports-Loki.json mirrors the PostgreSQL dashboard's layout with LogQL queries against the Loki output's lines, filtered by the domain stream label. The dashboard dev stack now runs Loki, seeds it with the sample reports, and imports the dashboard.
Elasticsearch and OpenSearch output can now write to data streams (#842, relates to #476). The new use_data_streams option in [elasticsearch] and [opensearch] (default False) saves each report type to one data stream named like today's indexes without the date suffix (dmarc_aggregate, dmarc_failure, smtp_tls, with index_prefix, index_suffix, or the index_prefix_domain_map prefix applied as before), instead of creating an index per day or month. That gives each report type one stable name, keeps the shard count down, and lets the cluster roll over and delete old data. parsedmarc puts an index template named after each stream, whose index_patterns is the exact stream name so it never applies to the dated indexes, and creates the stream; it does not create or manage retention policies. On Elasticsearch, attach an ILM policy through an optional <stream>@custom component template, which the template includes; on OpenSearch, an ISM policy attaches itself through its ism_template. docs/source/elasticsearch.md and docs/source/opensearch.md have example policies. Existing dated indexes are left alone and keep matching the dashboards' dmarc_aggregate*, dmarc_f*, and smtp_tls* patterns next to the stream, and the duplicate check searches both, so a report saved before the switch is not saved again after it. monthly_indexes is ignored when use_data_streams is on, and a warning says so. If a concrete index or alias already holds a stream's name, as an undated index from a release before 5.0.0 might, saving fails with an error that says so instead of writing into it. Every saved document now also carries an @timestamp field, in both modes: each record's begin date (date_begin) for aggregate reports, which is the report's begin date unless parsedmarc split a report spanning more than a day into daily records; the arrival date for failure reports; and the report's begin date for SMTP TLS reports. These are the same dates that pick the dated index. Documents are now saved with op_type=create, which data streams require; without a document ID this is the same as the default for dated indexes.

Security

S3 object keys and metadata are now sanitized, and each key ends in a hash of the fields that identify the report (GHSA-7wqr-f6g9-cm29, affects 11.1.0 and earlier). S3Client.save_report_to_s3() put the sender-controlled report ID into the object key as-is, and sent org_name and the other metadata values as-is. A report ID such as ../../x added path segments to the key, an ID could carry \, %, and the other characters AWS lists as "characters to avoid", or control characters, which AWS says "might require special handling", and a long ID could push the key past the 1,024-byte limit (AWS, "Naming Amazon S3 objects"). botocore rejects non-ASCII metadata with ParamValidationError, so a report from an organization with a non-ASCII name was never saved, and its batch was held back and retried. A bare CR or LF in a metadata value made botocore raise HTTPClientError: Invalid header value before sending, which held the batch back the same way, while CR or LF followed by a space or tab (an obs-fold) was sent in the x-amz-meta-* header; the 2 KB metadata limit (AWS, "Working with object metadata") was not enforced. The key also identified a report only by its type, begin day, and report ID, so anyone who could send reports to a monitored mailbox could replace another reporter's object for that day by reusing its report ID. The report ID is now percent-encoded into a single key segment and shortened so the whole key fits in 1,024 bytes. The encoded characters are /, every AWS "character to avoid" (\ { } ^ [ ] " < > ~ # | ` and %), control characters, and non-ASCII characters, so % appears in a key only as the escape character. The punctuation real report IDs use, such as :, @, ., and _, is kept. Each metadata value is percent-encoded (%, control characters, and non-ASCII characters) so urllib.parse.unquote() reverses it, and is cut to 256 characters, which keeps the total under 2 KB. Each key now ends in _<hash>.json, where the hash is the first 16 hex characters of a SHA-256 hash of the report type and the fields that identify the report (for aggregate reports: organization name, report ID, policy domain, begin date, and end date; for SMTP TLS reports: organization name, report ID, begin date, end date, and each policy domain). Enrichment such as reverse DNS is not hashed, so a retried report still overwrites its own object. This narrows the overwrite but does not close it: these fields are not secret, and a forged report whose reporter name, report ID, policy domain, and dates all match a real one still replaces it. Turning on S3 Versioning on the bucket keeps both. Compatibility: every report's key changes. Objects saved by earlier versions keep their old keys and are not rewritten, and a batch that is retried across the upgrade is saved again under its new key. Reported by helixnebulatech.
Credentials in URLs and the IPinfo API token are no longer written to the log (GHSA-hjr3-rx4c-24cv, affects 11.1.0 and earlier). The [elasticsearch] and [opensearch] hosts settings and the [splunk_hec] url setting accept URLs, and a URL can carry credentials (https://user:secret@host:9200). With --debug, the "Initializing Splunk HEC client", "Initializing Elasticsearch client", "Initializing OpenSearch client", and "Connecting to OpenSearch" lines wrote the configured value as-is, so the password went to the console and to log_file. With --verbose or --debug, parsedmarc also forwards the httpx library's HTTP Request: line, which httpx writes with the full request URL, so credentials in a webhook or Splunk HEC URL were logged on every request, as were credentials in a DNS over HTTPS URL in [general] nameservers (dnspython 2.9 and later make those requests through httpx2, which writes the same line under its own logger name), and the ipinfo_api_token was logged on every IPinfo lookup as the token query parameter. The password, or the whole userinfo when it has no :, is now replaced with *** in these lines (https://user:***@host:9200), and so is the token query value. Credentials in a [general] nameservers entry are also masked in the DNS pre-flight error, and credentials in either that setting or a --nameservers entry are masked in the --debug reverse DNS lookup errors, which quote each failing DNS over HTTPS URL, and in the error for an unusable https:// or tls:// entry, which was logged for each report that then failed to parse; such an entry, including one whose password contains an unencoded /, ?, or # (see below), is now rejected before any query is sent. The download URLs ipinfo_url, reverse_dns_map_url, and psl_overrides_url ([general]) can carry credentials too: they were logged as-is at --debug in the "Trying to fetch" lines, and when a download failed with an HTTP error status, the warning (shown with --warnings, --verbose, or --debug) included the full URL. Both are masked now: the failure warning shows the HTTP status code and the URL with the password replaced by ***. A malformed [postgresql] connection_string could put the whole connection string, or part of the password, into the output-client error logged at the default level; such errors now show a fixed message with no detail, and other PostgreSQL connection errors have URL credentials masked. The same masking applies to the URL in the "Initializing Loki client" line and in Loki push-failure errors ([loki] url, new in this release). The host and port are still logged (apart from the over-masking described below), and the credentials are still sent to the server unchanged. A password with an unencoded /, ?, or # ends the URL's authority early (RFC 3986, section 3.2), so httpx reads the text after the colon as the port and raises InvalidURL("Invalid port: 'p4ssw0rd'"), with a chained ValueError that repeats that part of the password. That text reached the log from [loki] url (at startup and on a SIGHUP reload), from [splunk_hec] url (on every batch, as "Splunk HEC Error"), from the webhook URLs (on every report, as "Webhook Error"), and from ipinfo_url (in the "Failed to save IP database" warning). A malformed psl_overrides_url crashed startup with the traceback, because InvalidURL is not an httpx.HTTPError. These now log or raise a fixed message, "(details withheld because they may include credentials)", with no chained exception. A malformed psl_overrides_url now logs a warning and falls back to the local_psl_overrides_path file when that is set, or else to the bundled file, and a malformed reverse_dns_map_url or ipinfo_url is reported as an invalid URL, not as "Not a valid CSV file" or "Failed to save IP database". Such a URL also has no @ in its authority, so the --debug "Trying to fetch" and "Initializing" lines printed it as typed; they now mask everything from the start of the authority through the last @ (https://***@host:3100). A URL with no credentials but an @ in its path or query is masked the same way in these lines (https://host/path@file is logged as https://***@file). The documentation now recommends separate credential settings instead of credentials in the URL or connection string: user and password (or api_key) for [elasticsearch] and [opensearch], user and password for [postgresql], username and password (or token) for [loki], and token for [splunk_hec]. Reported by helixnebulatech.
mailsuite 2.3.3 or later is now required. mailsuite 2.3.3 (GHSA-9jx3-p8wm-g8vw) makes the Maildir connection return a message's stored bytes from fetch_message() and rename the file in move_message(), instead of re-serializing it. A report email that cannot be re-serialized (the GHSA-qxwv-2vwj-v828 messages) is therefore now moved to the Invalid folder on Maildir too, instead of staying in the inbox and being logged on every run.

Bug fixes

Failure reports are now saved to S3. save_report_to_s3() read report["report_metadata"] for every report type except SMTP TLS, but failure reports have no report_metadata (see FailureReport in parsedmarc/types.py), so every failure report raised KeyError since S3 output was added in 7.0.0. The error was counted as an S3 output error, so the batch holding the report was held back. A failure report is now saved under <path>/failure/year=YYYY/month=MM/day=DD/<reported domain>_<hash>.json, dated by its UTC arrival date, with reported_domain and arrival_date metadata. Its hash covers the report type, the arrival date, reported domain, original mail from, original recipient, original envelope ID, source IP address, and the message sample.
DNS over HTTPS works again under dnspython 2.9 (#935). dnspython 2.9.0 moved DNS over HTTPS from httpx to httpx2, and dns.query.https() now raises ValueError: session parameter must be an httpx2.Client when it is given an httpx.Client. parsedmarc always passed an httpx.Client, so every https:// nameserver failed, and the resolver retried it until the lookup timed out. The shared DoH client is now built from the HTTP library dnspython itself imported: httpx2 on dnspython 2.9+, httpx on 2.7 and 2.8. The nameservers pre-flight check also reported that failure as no PTR record for 1.1.1.1, because get_reverse_dns() returns None for every DNS error. The check now queries the PTR record directly, so it reports no PTR record only when the nameserver answers that the record does not exist, and otherwise shows the resolver's actual error. The httpx2 and httpcore2 loggers are now included in --debug dependency logging, since dnspython 2.9 DoH traffic is logged under those names. An invalid dns_test_address is now reported by name in the pre-flight error. Reported by florianutz.
Keep distinct aggregate reports when one reporter reuses a Report-ID for different policy domains. RFC 9990 §3.5.1 scopes Report-ID uniqueness to the same domain. Direct report files, mbox files, and mailbox ingestion now all include the policy domain in one shared dedup key, using a JSON array to avoid ambiguous field separators. ASCII domain case differences still count as the same domain (RFC 4343 §3), without changing the returned report. Same-domain repeats remain suppressed — including a report passed both as a file and inside an mbox in one CLI run — and mailbox keys are still committed only after a successful save callback.