domainaware/parsedmarc 10.4.0 on GitHub

New features

n_procs parallel parsing now covers messages from mbox files and mailbox connections (#147), not just report files passed directly as CLI arguments: IMAP, Microsoft Graph, Gmail API, and Maildir connections, including watch mode. get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox(), and watch_inbox() all gained an n_procs keyword argument. Parallel workers are now a reused process pool for the whole run, with a bounded submission window that keeps at most roughly 2 * n_procs messages in flight at a time so memory stays bounded even for huge mboxes; the main process sends periodic IMAP keepalives while workers parse.
Added ParserConfig, a single frozen dataclass carrying every parsing/enrichment option (#503): offline mode, IP database path, reverse DNS map and PSL overrides paths/URLs, DNS nameservers/timeout/retries, strip_attachment_payloads, normalize_timespan_threshold_hours, and the three caches the parser and enrichment code share across calls (IP address info, seen aggregate report IDs, and the reverse DNS map). parse_aggregate_report_xml(), parse_aggregate_report_file(), parse_failure_report(), parse_report_email(), parse_report_file(), get_dmarc_reports_from_mbox(), get_dmarc_reports_from_mailbox(), and watch_inbox() all accept a keyword-only config= argument; when it's provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument continues to work unchanged when config= is omitted. Every explicitly constructed ParserConfig owns fresh, isolated caches; omitting config= falls back to the existing process-wide shared default caches, unchanged and identity-preserved (parsedmarc.IP_ADDRESS_CACHE, parsedmarc.SEEN_AGGREGATE_REPORT_IDS, parsedmarc.REVERSE_DNS_MAP). Caches never cross multiprocessing worker boundaries: ParserConfig.__getstate__ drops the three cache fields when a config is pickled for a worker, and each worker accumulates its own from that point on. keep_alive and n_procs remain separate keyword arguments — they control process/worker orchestration, not parsing or enrichment behavior, so they're deliberately not ParserConfig fields. See the new "Using parsedmarc as a library" section of the usage docs for an example.
Added a "Domain policy" dropdown filter to the Splunk aggregate DMARC dashboard (#854): filters every panel on the published DMARC policy (published_policy.p) or subdomain policy (published_policy.sp), making it easy to see which domains with a none policy are ready to move to quarantine or reject. It sits immediately before the "Message disposition" dropdown and offers the same choices (any/none/quarantine/reject, defaulting to any).
New [mailbox] options delete_aggregate, delete_failure, delete_smtp_tls, and delete_invalid (#256): control message deletion per report type instead of all-or-nothing. Each defaults to the value of the overall delete option, so existing configurations behave exactly as before; set one to override just that type — for example delete = True with delete_failure = False deletes processed aggregate and SMTP TLS report messages while archiving failure report messages. delete_invalid covers messages that could not be parsed, so they can be kept in the Invalid archive subfolder for debugging even when everything else is deleted. get_dmarc_reports_from_mailbox() and watch_inbox() gained matching delete_aggregate, delete_failure, delete_smtp_tls, and delete_invalid keyword arguments, where None (the default) means "inherit delete". The mutual exclusion between deletion and test mode is now evaluated against the effective per-type flags: any type that would be deleted still raises a ValueError alongside test, but delete = True with all four per-type options explicitly False is now valid with test enabled, since nothing would be deleted.
New [mailbox] option max_unsaved_retries (default 2): how many times a batch of messages whose reports could not be saved is retried before its messages are moved to the Unsaved archive subfolder and stop being retried. 0 moves messages on the first failed save; negative values are rejected with a ValueError. Failures are counted in memory, per message and per process, so the cap applies across watch-mode checks within one long-running process rather than across separate one-shot runs. See the mailbox-persistence bug fix below for the full behavior; the cap is deliberately low because each retry re-delivers the batch to every output destination that does not deduplicate. get_dmarc_reports_from_mailbox() and watch_inbox() gained a matching max_unsaved_retries keyword argument.
New [general] option archive_directory: move successfully processed local report files into a dated archive tree (<archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/); files that fail to parse as a report go to <archive_directory>/Invalid/, while files that fail for other reasons (e.g. transient I/O errors) are left in place so a later run can retry them; existing files are never overwritten (numeric suffix); files already under the archive directory are excluded from later runs. Applies only to report files passed directly as local path arguments (#570).

Changes

The DMARC pass/fail pie chart and over-time chart are now named for DMARC compliance in the OpenSearch Dashboards/Kibana and Splunk dashboards and the Elasticsearch-backed Grafana dashboard (the PostgreSQL Grafana variant has no pass/fail charts and is unchanged): "DMARC compliance" and "DMARC compliance over time" ("DMARC Compliance" / "DMARC Compliance Over Time" in Grafana, which title-cases panel names), replacing the previous mix of "Passed DMARC", "DMARC Passage", and "DMARC passage over time". This matches the existing "% DMARC Compliant" table column and the "Message volume and DMARC compliance by from domain" panels. The Splunk dashboard's pass/fail filter dropdown label was renamed to "DMARC compliant" to match.
The Grafana dashboard's Guide panel walkthrough now matches the current panel layout, directing readers to the "Top 2000 Message Sources by Reverse DNS" and "Message volume and DMARC compliance by from domain" tables and describing the compliance percentage they show, replacing stale references to a from-domain list "on the right" and a "Message From Header" table that no longer exists.
get_dmarc_reports_from_mailbox() gained an optional keyword-only save_callback parameter, invoked once per fetched batch with a ParsingResults dict holding only that batch's newly parsed reports, after parsing but before any of the batch's messages are deleted or moved out of reports_folder. Returning False reports the batch as unsaved: its messages stay in place for retry and the aggregate-report dedup cache is not updated for that batch, so the retry reparses those reports rather than skipping them as duplicates. Raising counts as an unsaved batch too — the same retention and max_unsaved_retries bookkeeping runs — and the exception is then re-raised, so a callback that fails by raising (the CLI's does, under fail_on_output_error) is still bounded by the retry cap even when the caller's watch loop swallows the exception, as mailsuite's IMAP and Maildir backends do. Any other return value, including None, commits the batch exactly as before, which is what save_callback=None (the default) does for every batch. As a related, strictly safer side effect of staging the dedup keys per batch, a mid-batch crash now leaves SEEN_AGGREGATE_REPORT_IDS unmodified for that batch instead of partially populated.
watch_inbox()'s callback is now passed through as save_callback, so it runs once per batch before that batch is archived or deleted, with only that batch's reports, rather than once afterward with the whole check's accumulated results. It may return False to defer a batch to the next check.
A single-shot CLI run configured with both file/mbox inputs and a mailbox connection now emits output in two passes instead of one: mailbox-derived reports are saved through the new save_callback before get_dmarc_reports_from_mailbox() returns, and file/mbox-derived reports are saved in a second pass afterward. append_json/append_csv are already append-safe and watch mode already called its callback repeatedly, so this changes how often output happens in the single-shot case, not what is written. When a mailbox connection is configured and the file/mbox inputs yielded no reports (none given, or none parsed), the second pass is skipped entirely rather than printing an empty JSON document.

Bug fixes

Mailbox messages are no longer archived or deleted until the output destinations confirm the reports were saved (#242). Previously get_dmarc_reports_from_mailbox() archived or deleted every successfully parsed message before the CLI wrote anything to Elasticsearch/OpenSearch/Splunk/S3/Kafka/PostgreSQL/etc., so an output outage meant the report was never persisted anywhere while its source message was already gone from the reports folder, with no way to recover it. Each batch is now written to every configured destination first; if any destination fails, the whole batch's messages stay in the reports folder and are retried on the next run or watch-mode check. This is all-or-nothing per batch (archiving on partial success would still permanently lose the data for whichever destination failed) and applies independently of fail_on_output_error, which only controls the process exit code. Unparseable messages carry no report data and are still filed under Invalid (or deleted per delete_invalid) as before. To bound re-delivery to destinations that do not deduplicate, a message whose batch has failed max_unsaved_retries + 1 times (three by default) is moved to <archive_folder>/Unsaved and stops being retried — never deleted, whatever the delete options say. Retries can produce duplicates in Kafka, Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the --output JSON/CSV files; Elasticsearch, OpenSearch, and PostgreSQL dedupe via AlreadySaved, and S3 overwrites the same key. Thanks to @mkilijanek for the original implementation in #823.
extract_report() now accepts plain uncompressed JSON. The JSON magic-byte constant was written as b"\7b", which Python reads as the octal escape \7 (BEL, 0x07) followed by a literal b — not 0x7B, the { every RFC 8259 JSON object begins with — so the JSON branch never matched and plain JSON input was rejected as "Not a valid zip, gzip, json, or xml file". Every in-tree caller pre-guarded with its own zip/gzip or { check, which masked the dead branch; the practical impact was on extract_report() as a public API and on application/tlsrpt+gzip email attachments whose payload is actually uncompressed JSON. Now b"\x7b".
An SMTP TLS report with an empty policies list no longer crashes the CLI when index_prefix_domain_map is configured. parse_smtp_tls_report_json() accepts an empty list, but get_index_prefix() unconditionally indexed policies[0], raising IndexError. Such a report has no domain to map, so it is now treated as unmappable — excluded from prefix-mapped output like any other unmapped-domain report — instead of crashing.
A failed --output file write no longer escapes uncaught. save_output() was the only output destination the CLI did not wrap in a try/except, so a full disk or an unwritable output path crashed the run instead of being recorded like every other destination's failure. It is now caught and recorded as a File output error, which also means it correctly blocks mailbox archiving like any other failure.
The emailed summary (smtp_host / Microsoft Graph) once again respects index_prefix_domain_map filtering of SMTP TLS reports. process_reports() applies that filter in place to the dict it is given, and it is now given each mailbox batch and the file-derived snapshot rather than the combined results — so the combined dict passed to email_results() no longer inherited the filtering. It is filtered explicitly instead, so a configuration combining index_prefix_domain_map and an email destination excludes unmapped-domain SMTP TLS reports from the emailed summary, matching what was saved.
Fixed the broken "XML files" link on the Splunk docs page: it pointed at the old splunk/ repository path instead of dashboards/splunk/, where the dashboards have lived since the dashboards/ directory was introduced.
A report file whose parsing raised an unexpected non-parser exception no longer hangs the CLI forever. The old direct-file parallel implementation spawned a fresh child process per file and blocked the parent on a pipe read; a child that crashed with anything other than a ParserError never wrote to that pipe, so the parent waited indefinitely. The new implementation returns the error as a value and logs Failed to parse <path> instead.
Direct-file parallel parsing no longer stalls a whole batch on one slow file, and no longer spawns a fresh interpreter per file. The old implementation processed files in hard batches of n_procs, so a single slow file delayed every other file in its batch; the new pooled-worker implementation streams files through a reused pool instead.
A one-shot mailbox run (no --watch) now honors [general] dns_timeout and dns_retries (#503). The CLI's call to get_dmarc_reports_from_mailbox() never forwarded those two options, so every one-shot mailbox run silently used the library's own hardcoded default (dns_timeout=6.0, dns_retries=0) regardless of what the operator configured; watch-mode runs were unaffected since the watch_inbox() call site did pass them. The CLI now builds a single ParserConfig per run (via the new _build_parser_config() helper) from parsed options and passes it as config= to every parsing entry point — the direct-file, mbox, one-shot mailbox, and watch call sites, plus the SIGHUP config-reload path — instead of forwarding option keyword arguments by hand at each call site.
Lazily-triggered reverse DNS map loads no longer silently clobber configured PSL overrides with the bundled defaults (#503). get_ip_address_info() and get_service_from_reverse_dns_base_domain() load the reverse DNS map on first use rather than eagerly (this also happens independently in each n_procs worker process, since a worker starts with an empty map), and load_reverse_dns_map() reloads psl_overrides.txt at the same time so map entries that depend on the current overrides fold correctly. That reload previously called load_psl_overrides() with no path/URL arguments, so the first lazy map load in a run overwrote any operator-configured PSL overrides file with the bundled defaults. get_ip_address_info() and get_service_from_reverse_dns_base_domain() now accept psl_overrides_path/psl_overrides_url parameters and thread them through to load_reverse_dns_map(), so a lazy load applies the same configured overrides an eager one would.
get_dmarc_reports_from_mailbox() and watch_inbox()'s dns_timeout default changed from a stray 6.0 to DEFAULT_DNS_TIMEOUT (2.0 seconds, parsedmarc/constants.py), matching every other parsing entry point; normalize_timespan_threshold_hours also now defaults to the float 24.0 on both, rather than the int 24. This is a minor behavior change for library callers who invoke either function directly without passing dns_timeout: DNS queries now time out after 2 seconds by default instead of 6, matching parse_report_file() and the CLI's own default.