What's new in v0.15.0
v0.15.0 is about the things a report did not say. A schema refactor read as a regression and failed the CI gate that was supposed to catch regressions. A window the daemon dropped instead of archiving left no hole for the integrity chain to find, so a period could under-report itself and still verify. And a database statement that the traces had covered carried no marker saying so, because the cross-reference only ever looked at the statements that had already produced a finding.
Three of these change what you read on identical data, and each has a paragraph in the upgrade section below.
A refactor read as a regression
diff matches findings on (finding_type, service, source_endpoint, pattern.template). The template is part of the identity because it is what distinguishes two N+1 findings on the same endpoint, but that makes any change to the query itself a change of identity. Rename a column, add a WHERE clause, move a host, and the same problem left the run as one resolved finding and re-entered it as one new one. A PR gate reading new_findings failed on the refactor that touched nothing about the anti-pattern.
A second pass now pairs those leftovers. Findings that survived the first match unpaired are reduced to (detector, service, endpoint, grouping) and matched again, and a pair lands in mutated_findings, counted neither as new nor as resolved. The pass never guesses: one candidate on each side pairs directly, and when several share the reduced key the code_location anchor (filepath plus function) arbitrates, pairing only an anchor that carries exactly one finding per side. Every other combination stays unpaired and keeps its old reading of resolved plus new, which is the reading that loses nothing.
A mutation is not silent about severity. The text report prints the pair under Mutated findings with the transition spelled out, Warning→Critical, and colors the marker red when the severity worsened rather than reusing the amber that says "changed". The dashboard's Diff tab carries the same pairs as their own cards and counts them on a ~ n mutated pill, and the diff CSV export gains a mutated row type. SARIF is unchanged and still emits new_findings only, which is what it has always documented, and a mutated pair is by construction not in that list.
mutated_findings is omitted from the JSON output when empty, so a consumer parsing the 0.14.0 shape sees the same document it saw before on a run with no mutation.
An acknowledgment that outlived its template
The same drift reaches acknowledgments, where the consequence is quieter. A signature covers a template, so a template that changes leaves the entry matching nothing, and the unmatched_acknowledgment warning said the finding no longer fires. On a query that was refactored rather than fixed, that reads as a green light to delete the suppression.
The warning now names a candidate successor when exactly one current finding shares the acked entry's detector, service and endpoint with a different template hash, and says what that means: the template drifted (schema or query change), re-acknowledge the new signature if the reason still holds. It is checked against the acknowledgment's own structured fields rather than inferred from the signature alone, and with anything other than exactly one candidate the warning keeps its previous wording.
The ack is never carried over. A signature is a suppression boundary, and moving one because two findings look related is how a suppression outlives the decision behind it.
A traced statement with nothing to say so
pg-stat --traces and mysql-stat --traces mark a ranked statement [seen in traces] when the traces carried the same SQL template. The cross-reference only ever consulted the templates that had produced a finding, so a statement that is heavily used, fully traced and perfectly healthy came back unmarked, indistinguishable from one the traces never saw. Both subcommands now cross-reference against every SQL template the run observed.
That makes a matched share meaningful, and both surfaces report one. They spell it out rather than labelling it:
Also seen in the traces: 1 of 15 statement(s) here, accounting for 15.4% of the calls the database counted.
A floor, not a sampling rate: the database counts since its last statistics reset, the traces cover one capture window.
The share of statements and the share of calls are two different numbers, and a bare trace-matched: 1/15 templates, 15.4% of calls gives a reader nothing to anchor either one to. It is never called coverage, and the reason now ships next to the figure instead of living in a code comment: pg_stat_statements counts since the last statistics reset while the traces cover one window, so the number is a floor on how much is traced. The dashboard panels carry the same two lines.
Coverage you can actually measure
pg-stat --baseline <earlier-export> --traces <file> measures what the previous section refuses to claim. Given two pg_stat_statements snapshots framing the trace window, the call delta between them is what the database executed during that window, and the traced span counts on the same templates are what was captured. The ratio is an empirical figure rather than a floor, reported as trace_coverage:
Empirical coverage: 6 of 8 executed calls traced (75.0%) across 1 template(s)
A counter that went backwards between the two snapshots means the statistics were reset mid-window, and that template's delta is meaningless. Those templates are skipped and counted in a warning rather than folded in, because a negative delta clamped to zero would quietly deflate the figure. --baseline requires --traces at the argument level, so a baseline with no counts means the trace ingest failed, and that case says so instead of reporting nothing.
A dropped window left no hole
The daemon hands each per-window disclosure archive to a writer over a bounded channel, 256 entries deep, and drops the window when the channel is full. The policy is deliberate, a stalled filesystem must never block the analysis path, but until now the loss was a warn log and nothing else. The archive hash chain only advances on a successful write, so a dropped window leaves no gap in seq, and verify-hash cannot find after the fact something that was never written.
perf_sentinel_archive_windows_dropped_total is the scrape-visible witness, labelled by reason: channel_full, writer_exited, serialize_error, write_error. All four are pre-warmed to zero at startup so a healthy archive and a missing metric stay distinguishable on a scrape.
A counter only helps whoever was scraping at the time, so each archive line also carries the daemon-lifetime cumulative count at write time, as drops. Two consecutive lines whose drops differ bracket the loss between them, which lets disclose reconstruct the gaps from the archive alone. The periodic disclosure schema moves to perf-sentinel-report/v1.7: integrity.trace_integrity_chain gains windows_dropped, the windows produced but not archived over the period, derived from those deltas, and drop_counter_resets, the daemon restarts inside the period, each one making the first figure a lower bound for the gap it spans.
Two blind spots stay open and no counter can close them. A writer task that dies discards its queued backlog uncounted, and a hard kill before the shutdown drain finishes does the same. Alert on rate(perf_sentinel_archive_windows_dropped_total[5m]) > 0 and read a nonzero rate as "the archive under-reports this period, by at least this much".
An archive truncated by its own rotation
The archive writer tracks how many bytes it has appended so it can stop at the configured cap, and it resynced that count by stat-ing the archive path. A rotation moves the file out from under the open descriptor, so the path then named a file that did not exist yet and stat-ed as zero bytes. The writer believed it had the full cap available, and a later set_len cut the archive back to empty.
It reads the count from the descriptor it actually holds now, which follows the file through a rename. When the stat itself fails it keeps the previous count rather than assuming zero, since the previous count is the offset of the last complete window and zero is the one value that authorises the truncation. Both drop paths also name perf_sentinel_archive_windows_dropped_total in their warning, so the log line points at the counter that measures the loss.
Two dashboard colors resolved to nothing in the same release. A grouping pill referenced a --surface-1 token that does not exist, so its background fell back to transparent and the pill lost the contrast separating it from its row, and the API-key modal's 401 hint referenced --color-text-warning, leaving the one line that explains why the modal opened rendered in the inherited body color. Both use the tokens the rest of the sheet uses now, --surface-2 and --warn-fg.
Upgrade impact
A diff that reported resolved plus new for a refactored query now reports one mutation. new_findings and resolved_findings both shrink by one per paired mutation, so a gate wired to either count changes verdict on identical inputs. That is the point of the change, but a pipeline asserting an exact count needs the new one. Ambiguous cases are unchanged.
mutated_findings is a new JSON key, absent when empty. A consumer of the 0.14.0 diff shape parses an unchanged document on a run with no mutation.
pg-stat and mysql-stat mark more statements [seen in traces] on identical input. The marker now reflects "the traces carried this template" rather than "this template produced a finding". Both also print two new summary lines under --traces.
Periodic disclosures produced by this binary declare perf-sentinel-report/v1.7. A validator pinned to v1.6 rejects them. windows_dropped and drop_counter_resets are absent, not zero, on archives written before the counter existed, so an old archive never reads as a clean period it cannot vouch for. Re-hashing a v1.6 report on this binary keeps its content_hash.
Archive lines gain a drops field, and the chain still verifies across the upgrade. The line hash is computed over whatever keys the body carries, so lines written by 0.14.0 and by 0.15.0 verify side by side in one archive.
h2 is bumped to 0.4.19 against RUSTSEC-2026-0258. It reaches perf-sentinel transitively through hyper, which every outbound HTTP path uses and which the daemon's listeners sit on. Patch-level, semver-compatible, cargo audit and cargo deny check both green.
The daemon API surface is unchanged, the Prometheus label sets gain one bounded reason family and change nothing existing, the file-based --input paths are unchanged, and the minimum supported Rust version stays 1.97.1.
Validation
The full Rust suite passes (3300 tests), clippy is clean at -D warnings across all targets, cargo fmt --check passes, and the default and --no-default-features builds both pass. The dashboard work rests on the Playwright suite, 49 specs against a real browser.
The mutation pairing is covered on both sides of its own rule: the unambiguous pair, the anchor-arbitrated pair, and the ambiguous case that must stay resolved plus new. The archive truncation fix is exercised through a real rotation rather than a mocked stat. The drop counter is asserted to pre-warm to zero for all four reasons, since a counter that only appears once it fires cannot be told from one that is not wired.
This revision carries a simulation-lab PASS, the first since v0.13.0, recorded in release-gate/lab-validations.txt. It covers 38 of the 79 scenarios, not the whole suite: the 0.15.0 surface and schema v1.7, six new ecosystem scenarios, every scenario whose assertions this cycle corrected, and what the version bump moved. The untouched families (limit, multistack, energy backends, resilience) were not re-run, and the ledger entry says so.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.15.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.