domainaware/parsedmarc 10.3.0

on GitHub

New features

• Added a per-domain DMARC compliance percentage to the aggregate dashboards of every provider (#112): OpenSearch Dashboards/Kibana, Grafana Elasticsearch, Grafana PostgreSQL, and Splunk. The from-domain volume table on each dashboard is now "Message volume and DMARC compliance by from domain", with columns for From Domain, Messages, and % DMARC Compliant.
  • On OpenSearch Dashboards/Kibana, the table is now a TSVB visualization using a Filter Ratio metric (passed messages over total messages per header_from), since the previous agg-based data table can't compute a per-domain ratio. Editing the imported visualization on Kibana 8.x requires first enabling the metrics:allowStringIndices advanced setting.
• Directory paths are now accepted as file_path CLI arguments (#397): a directory expands to the report files inside it using shell-glob semantics (dotfile entries excluded, subdirectories skipped by default), and the new -r/--recursive flag descends into subdirectories and enables ** recursion in glob patterns.

Changes

• Updated the pinned dev-tooling versions and converted the codebase to f-strings: ruff 0.15.21 → 0.16.0 and pyright 1.1.410 → 1.1.411 in the [build] extra. All str.format() calls were converted to f-strings (ruff rules UP030/UP032), except where the f-string form would require Python 3.12's quote reuse inside expressions — those keep .format() with the redundant positional indices removed. Because ruff 0.16.0 greatly expanded its default lint rule set (adding rule families such as BLE, SIM, C4, DTZ, and import sorting, some of which conflict with deliberate house style — e.g. BLE001 flags the parser's intentional broad catches), [tool.ruff.lint] now selects its rule set explicitly: the pre-0.16 defaults (E4/E7/E9/F) plus the modern-type-hint and f-string UP rules. Adopting any of the newly-default rule families is left as a deliberate future per-family decision. Alongside the conversion, a few pre-existing string defects the conversion surfaced were cleaned up: log/exception messages that embedded runs of indentation whitespace via backslash line continuations (and one missing sentence separator in the since-option warning) now read cleanly, and the Splunk HEC output builds its newline-delimited payload by joining a list instead of repeated string concatenation. No functional behavior changes.

Bug fixes

• The Elasticsearch/OpenSearch aggregate dashboards' over-time charts (and the Grafana ES dashboard's summary pies and time series) bucketed on the multi-valued date_range field; a date histogram counts a report once per value, double-counting any report whose begin and end dates fall in different buckets. All date histograms and time-range filters now use the single-valued date_begin, matching the report-begin semantics of the PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
• Aggregate-report policy and authentication result words are now normalized to lowercase (#288): reporters that emit mixed-case values such as Pass no longer create duplicate result categories in outputs and dashboards.
• The results email (SMTP and Microsoft Graph) is no longer sent when no reports were parsed (#200): previously an empty run — an empty inbox, or one where every message was invalid — still emailed a zip of headers-only CSVs. The email step is now skipped with an INFO log when the run produced no aggregate, failure, or SMTP TLS reports.
• The DKIM/SPF alignment-detail tables on the Kibana/OpenSearch Dashboards, Grafana (Elasticsearch), and Splunk aggregate dashboards no longer show a selector × domain × result cross-product (#169). Elasticsearch/OpenSearch flatten the dkim_results/spf_results object arrays (they are dynamic-mapped as object, not nested), so stacking terms aggregations on their subfields produced every combination of selector/domain/result across a report's signatures, each repeating the full message count. Aggregate documents now also carry dkim_results_combined and spf_results_combined — one "selector / domain / result" ("scope / domain / result") string per auth result — and the dashboards aggregate those instead; the Splunk detail panels now pair the values with mvzip/mvexpand. Documents saved by older versions are now backfilled automatically at startup (a non-blocking, idempotent background task); the documented _update_by_query command (see the Elasticsearch docs page) remains available for running the backfill manually. The Grafana "DKIM Alignment Details" panel's dmarcian.com DKIM-checker data link was removed because it required the separate domain/selector columns. The SMTP TLS visualizations had the same class of defect; that fix is described in its own entry below.
• The Kibana/OpenSearch Dashboards aggregate dashboard now includes an "Auth result filters" control panel above the SPF/DKIM details tables, with dropdowns for DKIM selector/domain/result and SPF scope/domain/result, so results can still be filtered by individual components alongside the combined per-signature columns.
• SMTP TLS visualizations had the same cross-product defect as the DKIM/SPF alignment tables (#169): policies is an object array, and each policy's failure_details is itself an object array inside it, so stacking terms aggregations on their subfields cross-products the same way. SMTP TLS documents now also carry policies_combined — one "policy_domain / policy_type" string per policy — and failure_details_combined — one "policy_domain / policy_type / result_type / sending_mta_ip / receiving_ip / receiving_mx_hostname" string per failure detail — and documents saved by older versions are backfilled automatically at startup the same non-blocking, idempotent way as the DKIM/SPF backfill; the equivalent manual _update_by_query command is documented on the Elasticsearch docs page. Also fixed two adjacent dead fields found while making this change: _SMTPTLSFailureDetailsDoc declared additional_information_uri, but add_failure_details passed it to the constructor as additional_information, so it was never actually populated on the declared field; and receiving_mx_hostname, which add_failure_details has always stored, had no field declaration at all. Both are now correctly wired in the Elasticsearch and OpenSearch outputs.
• The Grafana "Map of Message Source Countries" panel's markers now scale with message volume and use a higher-contrast style. Previously the markers were fixed-size 5 px dots at 50% opacity in dark green, which were nearly invisible on the dark basemap.
• Corrected the dead _SPFResult.results (plural) field declaration to result, matching what was always written to it.
• An mbox-only run no longer shows a misleading, permanently-stuck 0it progress bar, and mbox parsing now shows a real per-message progress bar on interactive terminals (#147). The CLI's progress bar only tracks report files passed directly as arguments — n_procs parallel parsing also only applies to those — so runs whose only input was an mbox file displayed an empty bar while messages were parsed sequentially with no visible progress (per-message progress is logged at INFO, hidden in --silent/config-file runs). The empty bar is no longer created, get_dmarc_reports_from_mbox() now wraps its message loop in a tqdm bar that auto-disables on non-TTY output, and the n_procs documentation now states the direct-file-arguments-only scope explicitly.