What's new in v0.29.0
v0.29.0 is about incidents that never reached the Hub. A daemon keeps its incidents in a ring of [daemon.incidents] max_retained entries, 200 by default, with no TTL, and PerfSentinelHub read them only through its hourly poll. An incident recorded and then evicted between two polls stayed in the daemon's archive and never reached the Hub, the fleet's long memory. A daemon that exports its findings to the Hub now pushes its incidents along with them, and every later state of each one. No detector, normalizer or signature changes.
Incidents travel with the findings
With [daemon.hub_export] and [daemon.incidents] both enabled, the incident store hands the exporter a copy of an incident at each of the three moments it changes: when it is recorded, when its alert resolves and its end arrives, and when the settle pass, about three trace TTLs later, adds the findings that arrived late. No configuration key is added. The incidents go to the Hub's /api/import/incidents, derived from the configured endpoint by replacing its /api/import/findings suffix, with the same source_id and the same X-API-Key. Each incident is sent exactly as GET /api/incidents serves it, so the push and the poll share one format and the Hub merges the two copies under one rule.
Pending incidents sit in a latest-value table keyed by incident id. A newer copy of a pending incident replaces it, and a late acknowledgement of an older batch cannot remove the newer one. The table holds at most min(max_pending, 1000) incidents, since one incident can carry a thousand frozen findings, and evicting from it counts on perf_sentinel_hub_export_incidents_dropped_total. Batches hold at most 10 incidents and 4 MiB of JSON. An incident that alone exceeds 4 MiB is dropped and counted, and the Hub's poll can still copy it.
Two halves, each on its own backoff
The exporter task now runs a findings half and an incidents half. They share the response classification, the jitter and the handling of a busy Hub's 503, but each keeps its own failure count and its own pace: an incident route that fails backs off from about a second upward while findings keep flowing every flush_interval_secs, and a findings retry does not cut the incidents' backoff short. The graceful drain at shutdown empties both tables within the existing 10-second budget.
A Hub that predates the route answers 404 or 405. The daemon then stops pushing incidents until it restarts, logs one WARN, and clears the table, counting its entries as dropped. Findings keep flowing. A Hub with sign-in on answers 401 instead, which the daemon cannot tell from a wrong key, so it retries and logs. Upgrade the Hub first.
In the lab, a daemon with max_retained = 2 recorded five incidents. The Hub held all five without polling, and then the end of one once its alert resolved.
A smaller ring once the Hub keeps every incident
The daemon still freezes and keeps its incidents: the frozen findings come from a ring that moves on within minutes, and the settle pass, query monitor and a daemon without a Hub still need them. What changes is advice. Once perf_sentinel_hub_export_incidents_pending stays near 0, the ring no longer has to hold a poll interval's worth of incidents, and a smaller max_retained saves memory, since the default 200 can pin 200,000 frozen findings in the worst case. RUNBOOK.md says how far to lower it.
Upgrade impact
- Ship PerfSentinelHub 0.5.0 before the daemons. A daemon with both blocks enabled starts pushing incidents on upgrade. An older Hub answers 404 with sign-in off, which stops the incident push quietly, and 401 with sign-in on, which the daemon retries and logs as a
WARNuntil the Hub is upgraded. perf-sentinel-corebreaks struct literals ofQueryApiState. It gains a crate-private field for the incident export and is not#[non_exhaustive], which is why this release bumps the minor./metricsgains two label-free series,perf_sentinel_hub_export_incidents_pending(gauge) andperf_sentinel_hub_export_incidents_dropped_total(counter). The findings series keep their names and labels, so no existing alert changes.- No configuration key is added or removed, no finding appears, disappears or changes signature, acknowledgments keep matching, the embedded reference data keeps its vintages, and MSRV stays 1.99.0.
Full detail in CHANGELOG.md.
Verifying this release
# Binary integrity via SLSA build provenance attestation (Build Level 2)
gh attestation verify perf-sentinel-linux-amd64 \
--repo Institut-du-Numerique-Responsable/perf-sentinel
# A periodic disclosure produced by this binary, with its attestation and bundle
perf-sentinel verify-hash --report perf-sentinel-report.json \
--attestation perf-sentinel-report.intoto.jsonl --bundle perf-sentinel-report.sig \
--expected-identity "https://github.com/Institut-du-Numerique-Responsable/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.29.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify. Releases up to v0.26.2 were attested under robintra/perf-sentinel and keep their own --repo and identity.