What's new in v0.24.0
A daemon that restarted came back with no incidents. Every incident, every close and every settle had been written to [daemon.incidents] archive_path, and nothing ever read that file back, so /api/incidents and the Grafana incident panels started empty after each restart and each upgrade. Alerting filled them again, including for workloads the daemon had never traced, each of which raised an incident carrying no finding at all.
v0.24.0 reloads the ring from the archive at startup, keeps the shipped alerting rules to the services the daemon actually ingests, and hands a suggested fix to findings that had none.
The ring the archive already held
At startup, before the API serves, the archive is streamed line by line and replayed in the order it was written, so the last record of each id wins and the last max_retained incidents to arrive come back in the order the ring held them. The read takes the last 256 MiB at most, so an unrotated file cannot hold the daemon past its liveness probe, and it starts on a record boundary rather than on whatever byte the cut landed in. A line that does not parse, a torn write sealed at open, is skipped with one warning, nothing is appended back, and the read uses the same symlink guards as the append side.
The reload is what the archive was always for, and it is not a perfect replay: the file records changes, not their kind, so a close or a settle whose own record was cut away, dropped or aged out counts as an arrival on the way back in. docs/RUNBOOK.md says where that shows.
Two changes to one incident, in the order the ring applied them
A new incident, a close and a settle each handed their record to the archive writer after releasing the ring's lock. A resolved delivery racing the settle pass could leave the older record last, and the last record of an id is the one that counts, for the startup reload as for any reader of the file, so an incident came back open or without its settled findings. The record is now handed over before the ring unlocks. The hand-off still never waits: a full channel drops it and counts it on perf_sentinel_incidents_archive_failed_total, as before.
One incident by id
GET /api/incidents takes an id and answers with that one incident, or [] when the ring does not hold it, ignoring service, namespace, offset and limit. A client holding an id no longer pages through incidents that each carry up to 1000 frozen findings to pick one out, and finds it wherever a newer incident pushed it. A daemon older than 0.24.0 ignores the parameter and answers with the page.
The findings dashboard uses both halves of that: Incident findings asks by id, and the Incidents table pages the whole ring from a new Incident skip rows variable, where it used to read limit=50 and reach nothing older.
Alerts for services the daemon has never seen
The shipped incident rules, examples/incident-alerts-prometheus-operator.yaml and its VictoriaMetrics twin, select on a container name, which matches workloads the daemon does not trace. Each restart or OOM of those raised an incident with no finding behind it. The four rules now subtract a perf_sentinel:untraced_services:1d recording rule over perf_sentinel_service_io_ops_total, evaluated every two minutes rather than read over a day by each rule at every evaluation.
Whatever the record lacks lets an alert through, deliberately: it is empty before its first evaluation, and empty again once perf_sentinel_service_io_ops_overflow_total is nonzero, since a service past the daemon's 1024-service cap has no series of its own. An alert that derives no service still reaches the daemon, which refuses it and counts it on perf_sentinel_incidents_rejected_total. Under the VictoriaMetrics operator the recording rule needs the VMAlert's remoteWrite, without which vmalert refuses every rule it selects.
The baseline a reader could not see
GET /api/acks listed only the runtime acks of the daemon's JSONL store. A reader that mirrors the ack state could not learn that the CI TOML baseline acknowledges a finding once that finding had left the findings ring, because only GET /api/findings carried the annotation, and only for the findings it still held.
include_toml=true adds the active baseline entries, mapped onto the same fields, by from acknowledged_by, at from acknowledged_at exactly as the file writes it, expires_at as the end of the expiry day in UTC, and every row then names its source, daemon or toml. The default response is unchanged, the same key gates both, a malformed value answers 400 only after the key check, and the 1000-row cap holds over the two sources together. perf-sentinel ack list, the live panel of the HTML report and the Grafana table stay on the default, because a baseline ack cannot be revoked through the API.
Fixes that were never offered
Four paths reached suggested_fix = None and now do not. A finding whose only framework signal was an OpenTelemetry Java agent scope got nothing, not even the java_generic fix, because the language-from-scope step knew the Go, npm, NuGet, Ruby and PHP prefixes but not the agent's own io.opentelemetry.<library> convention. Cross-trace slow findings never went through the suggestions step at all, so a slow_sql on a code_location that resolves to java_jpa carried no fix. The four structural types, serialized_calls, excessive_fanout, chatty_service and pool_saturation, carried neither instrumentation scopes nor a code_location, so no framework and no language could be inferred from them: each now carries those of one representative call it already references, the first call of the serialized sequence, the first child of the fan-out, the first outbound call of the chatty trace, the first SQL span of the saturated service. And a type its framework has no fix for, redundant_http or pool_saturation under java_jpa, now falls back to that framework's language generic instead of returning nothing, which was less than a finding with no framework signal received.
The java_generic N+1 text also stopped promising an ANY(?) binding to every JDBC driver. It names PostgreSQL, where that syntax works, and points at the JDBC IN clause section of the OpenTelemetry Java instrumentation docs.
Two structural findings under one anchor
diff pairs a finding across two runs when its template changed but its identity held, and it arbitrates several candidates on their code_location. Since the structural types gained one, two different patterns that begin with the same call shared an anchor, and a fixed fan-out beside a new one could pair as a single mutated finding, with the new problem disappearing from new_findings. The four structural types are out of that arbitration now. A lone candidate on each side still pairs directly.
A dump that analysed to nothing
This release moves opentelemetry-proto to 0.33.0, which ignores a field it does not know where 0.32 refused it. An OTLP/JSON file whose keys are spelled in snake_case, resource_spans, then deserialized into an empty request, and every span in it disappeared without a word: the run reported zero events, zero traces and no error. The protobuf JSON mapping allows both spellings, so a conformant file vanished. The reader now gets OTLP/JSON spells its keys in camelCase: found "resource_spans", expected "resourceSpans", and a camelCase document that genuinely carries no resource span is still not an error.
Reading back further than the ring
The findings dashboard gains a collapsed History (Hub) row. Its other tables read the daemon's ring, which is bounded by volume and not by time, so the picker could not look back however long Prometheus kept the metrics beside them. The new row reads PerfSentinelHub 0.3.0 through a second, optional Infinity datasource, and lists one row per problem the Hub saw in the chosen environment on a day of the picker's range, with First seen, Last seen and Status computed for that environment. It carries no occurrence column, since ring-relative counts do not add up over months.
A new first column, Ack, on the three findings tables opens the Hub's ack page for that signature, which is how a finding the ring no longer holds gets acknowledged from Grafana. __inputs is unchanged, so the dashboard still imports without a Hub: the row then errors when opened and nothing else moves. Dashboard version 11, and the info severity of its tables is blue instead of a text color that a color-text cell drew dark whatever the theme.
Upgrade impact
- Incidents survive a restart, where
[daemon.incidents] archive_pathis set. The first start after the upgrade repopulates the ring from a file the daemon has been writing all along, so/api/incidentscomes back with history it never had before. Nothing is appended back and no configuration key moves. - Findings gain fixes they did not carry. A
suggested_fixappears on Java agent findings with nocode_location, on cross-trace slow findings, on the four structural types, which also gaininstrumentation_scopesand acode_location, and wherever a framework had no fix for a type. A report compared against a 0.23.0 one will show those fields filled, on the same findings with the same signatures. diffreads two structural findings as two. What used to pair as one mutated finding, when both began with the same call, is now one resolved finding and one new one. That is the 0.23.0 reading restored, since only this release gave those types thecode_locationthe arbitration used.- A snake_case OTLP/JSON file now fails loudly. A pipeline that fed one and read zero events gets a non-zero exit and a message naming the spelling. Re-export in camelCase, which is what every OTLP exporter writes.
- The shipped incident rules alert on less. A service the daemon has not ingested for a day raises nothing, which is the point, and under the VictoriaMetrics operator the new recording rule needs
remoteWriteon the VMAlert or vmalert refuses every rule it selects. - The findings dashboard is at version 11. Re-import it to get the incident paging, the
Ackcolumn and the readableinfoseverity. TheHistory (Hub)row is optional and needs a Hub, without one it errors only when opened. perf-sentinel-coregains three items behind thedaemonfeature,daemon::incidents::load_archive,daemon::incidents::LOAD_TAIL_BYTESandIncidentStore::with_incidents. Nothing is removed and no public type changes shape.- Two query parameters are added and none is removed, no configuration key is added or removed, no metric or wire format changes, no finding appears or disappears, the acknowledgment signatures are untouched, and MSRV stays 1.98.1.
Full detail in CHANGELOG.md.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.24.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.