IncidentRelay v2.4.0 — Alert Reopening, Recovery Reliability & Lifecycle Hardening

IncidentRelay 2.4.0 introduces flapping-aware AlertGroup reopening, more reliable recovery notifications, new monitoring integrations, and significant improvements to lifecycle consistency, audit logging, security, and deployment reliability.

This release builds on the first-class Incident architecture introduced in 2.3, making alert recovery and reopening more predictable while improving operational visibility and backward compatibility.

Highlights

🔄 AlertGroup Reopening & Flapping Protection

IncidentRelay now supports reopening recently resolved AlertGroups when the same technical issue occurs again.

Opt-in reopening through Event Orchestration.
Configurable reopen_window_seconds (0–86400 seconds).
Reuse of matching resolved AlertGroups instead of creating unnecessary new groups.
Preservation of previously resolved child Alerts and their history.
New child Alert creation for every reopened occurrence.
Fresh acknowledgment, escalation, and reminder cycles.
Concurrency-safe reopening across PostgreSQL, MySQL, and SQLite.
Automatic transition of eligible linked Incidents back to investigating.
Reopening events visible in AlertGroup Timeline and Explain Trace.

Backward compatibility: Reopening is disabled by default. Existing alert grouping behavior remains unchanged unless explicitly enabled.

🔔 Reliable Recovery Notifications

Recovery notification delivery has been significantly improved to reduce notification noise and prevent stale or duplicate recovery messages.

Configurable recovery notification delay for flapping protection.
Immediate technical resolution with optionally delayed external notifications.
Cancellation of pending recovery notifications when an AlertGroup reopens.
Durable per-recipient recovery delivery tracking.
Independent retry handling for failed notification targets.
Exponential retry delays with a maximum of five attempts.
Partial and failed recovery delivery states recorded in the timeline.
Protection against stale notifications from previous recovery cycles.
Improved handling of notification worker restarts and interrupted deliveries.

The default recovery delay remains 0, preserving existing immediate notification behavior.

🔌 New Monitoring Integrations

OpenSearch Alerting

Native webhook endpoint: /api/integrations/opensearch.
Support for active and resolved monitoring events.
Stable deduplication based on monitor, trigger, and optional bucket identity.
Severity normalization and Event Orchestration support.

Elastic Kibana Alerting

Native webhook endpoint: /api/integrations/kibana.
Support for per-alert active and recovery notifications.
Stable alert identity using space, rule, and alert identifiers.
Integration with existing Routes, escalation policies, and notifications.

Both integrations support the standard IncidentRelay alert lifecycle and existing route intake authentication.

✉️ Flexible Email Notification Recipients

Email notification channels now support three recipient modes:

Assignee: Send to the assigned responder (existing default).
Fallback: Send to the assigned responder, or to a configured fallback address when unavailable.
Fixed: Always send to a configured email address, such as a shared operations mailbox.

This makes email notifications more flexible for unassigned alerts, shared mailboxes, and parallel notification workflows.

Existing email channels retain their current behavior.

🛡️ Unified Lifecycle & Audit Logging

Lifecycle and audit recording have been standardized across the application.

Consistent audit action naming using canonical domain.verb identifiers.
Unified audit provenance across API and external action transports.
Atomic database mutations and audit writes for supported lifecycle and configuration operations.
Improved rollback behavior when audit persistence fails.
Consistent handling of repeated lifecycle commands.
Improved attribution of authenticated users, API tokens, and action sources.
Preservation of technical AlertGroup events separately from administrative audit records.
Migration of legacy audit action names to the normalized contract.

These changes improve traceability and help prevent partially committed operations.

🖥️ User Interface & Operational Visibility

Improved AlertGroup reopening history and timeline navigation.
Direct navigation from reopened events to the corresponding child Alert.
Clearer separation of AlertGroup and Incident lifecycle information.
Better visibility into recovery notification states.
Improved cross-entity navigation between related Alerts, AlertGroups, and Incidents.
Updated configuration interfaces and localization for new integrations and notification options.

🐳 Docker & Deployment Hardening

Improved non-root container execution.
Consistent UID/GID handling for application processes.
Safer ownership reconciliation for existing Docker volumes.
Protection against symbolic-link traversal during volume initialization.
Privilege dropping before database migrations and application startup.
Better support for existing installations migrating to non-root containers.
Improved runtime configuration and secret-handling safeguards.

🧪 Test Suite & CI Improvements

This release also includes substantial test infrastructure improvements.

Removed redundant tests while preserving unique regression coverage.
Consolidated overlapping alert notification and business service scenarios.
Added automatic detection of exact duplicate test functions.
Improved regression tests for transactional audit behavior.
Expanded lifecycle, recovery delivery, and concurrency coverage.
Added JUnit XML and coverage reports as downloadable GitHub Actions artifacts.
Validated compatibility across Python 3.10–3.14.
Improved SQLite and PostgreSQL test coverage.

Latest release branch CI results:

Python 3.12: 2,199 passed, 3 skipped.
PostgreSQL: 2,200 passed, 2 skipped.
Python coverage: 72.92%.
All six test jobs passed.
No exact duplicate test functions detected.

Upgrade Notes

Database migrations

Apply the database migrations included with this release before starting the upgraded application services.

Important new migrations include:

20261014090000_recovery_notification_deliveries
20261024000001_normalize_audit_actions

Back up the database before upgrading.

AlertGroup reopening

The feature is opt-in and requires an active Event Orchestration with set_grouping configured. Existing orchestration behavior is unchanged by default.

Recovery notification delivery

The default remains immediate delivery. Deployments enabling delayed recovery should review the per-recipient retry behavior and operational requirements.

Before downgrading to a release without the recovery delivery ledger, pending recovery deliveries must be drained or reconciled to avoid possible duplicate notifications.

Docker deployments

Review persistent-volume permissions and any custom container user or security-context settings when upgrading an existing installation.

Email channels

Existing channels continue using assignee-based delivery. Fallback and fixed recipient modes are optional.

Documentation

Full Changelog

[Compare v2.3.1...release/2.4.0](v2.3.1...release/2.4.0)

Thanks to everyone contributing to IncidentRelay through bug reports, feature requests, testing, and code contributions.

IncidentRelay — Open-source, self-hosted on-call management and incident response.

roxy-wi/IncidentRelay v2.4.0 on GitHub