domainaware/parsedmarc 10.4.1

on GitHub

Bug fixes

• The automatic dkim_results_combined/spf_results_combined and policies_combined/failure_details_combined backfills now cover the indexes named by index_prefix_domain_map and by a previous index_suffix (#868). The startup migration derived its Elasticsearch/OpenSearch index names from the [elasticsearch]/[opensearch] index_prefix/index_suffix options alone, while the save path honors index_prefix_domain_map as well. In a multi-tenant deployment whose per-tenant prefixes come from that map — with no index_prefix set, as the multi-tenant documentation instructs — the guard count therefore ran against dmarc_aggregate*/smtp_tls*, patterns that match none of the real <tenant>_dmarc_aggregate-* indexes. Since the query passes allow_no_indices=True, a zero-match wildcard returns a count of 0, which is indistinguishable from "already backfilled", so the backfill was skipped silently, with no log line at any level, leaving the 10.4.0 dashboards' "DKIM details" table empty and every "SPF details" row labelled none. The migration now targets one index name per tenant prefix in the map — normalized by the same helper the save path uses, so the two cannot drift — plus the unprefixed name, since aggregate and failure reports for a domain that is absent from the map are still saved without a prefix and every report type may have indexes predating the map. A configured index_prefix still wins outright and suppresses the fan-out, matching save-time precedence, so such a deployment never submits an _update_by_query against an index pattern it does not write to. The index_suffix axis is widened the same way: the unsuffixed pattern is now targeted alongside the suffixed one, so documents indexed before the suffix was configured are backfilled too — note that on a shared cluster the unsuffixed pattern also matches other deployments' suffixes. The resolved target index names are logged at debug level, and a SIGHUP configuration reload re-resolves them, so a newly onboarded tenant is covered without a restart. Finally, an index_prefix_domain_map YAML file that is not a mapping of tenant names to lists of domain names, all strings, is now rejected at startup with a ConfigurationError. Every other shape failed silently rather than loudly: a non-string key raised mid-save, a scalar value matched the wrong domains (in on a str is a substring test, so "example.co" in "example.com" is True), and a non-string list item such as tenant_a: [42] simply never compared equal to any domain.
• The legacy published_policy.fo index migration works again, and is back for Elasticsearch as well as OpenSearch. parsedmarc releases before 5.0.0 declared that field as an integer, so their indexes mapped it as long, which cannot hold the multi-value fo settings reports carry (0:1, d:s); 5.0.0 both fixed the declaration and added a migration that rebuilds such an index as a -v2 index with the text/keyword shape. That migration worked against the Elasticsearch 6 clusters of the day, but has been unable to complete since mapping types were removed from Elasticsearch 7 — and they never existed in OpenSearch — for two reasons: it read the field mapping in the old response shape that nests fields under a mapping type name, so the type check always fell through; and its put_mapping() call passed a doc_type argument that neither current client accepts, which would have raised TypeError if the check had ever passed. It now reads either response shape and uses each client's current put_mapping() signature. It is also retry-safe: an attempt interrupted between creating the -v2 index and deleting the original used to leave debris that made every later startup fail on "resource already exists", so the index was never migrated; a leftover target is now discarded first, which is safe precisely because the original is only deleted once the reindex has succeeded. The whole migration is verified end to end against live Elasticsearch 8.19 and OpenSearch 3 clusters, including that recovery path. The index names it checks are also corrected: it takes exact names rather than wildcard patterns, because 5.0.0 introduced date-suffixed index names in the same release that fixed the declaration, so an affected index has no date component — but it can be prefixed or suffixed, since both options date back to 4.1.0, so the configured index_prefix/index_suffix are applied. As with the backfill, an index_suffix also gets the unsuffixed name checked, for history predating the suffix. index_prefix_domain_map prefixes are deliberately not applied, since that option arrived in 8.19.0, long after the mapping was fixed; this migration renames the index it rebuilds — documents are reindexed into the -v2 index before the original is deleted, so none are lost, but the name changes — and so it must not be pointed at indexes belonging to a tenant it is not migrating. The Elasticsearch copy of the migration, removed as unreachable during the #806 client migration, is restored now that the cause is understood. Note that Elasticsearch 8 refuses to open an index created before 7.0, so an affected index reaches a supported cluster only by being carried forward through a reindex, which preserves the old mapping whenever the destination index is pre-created from it — the standard reindex procedure.