What's new in v0.23.0
A cross-trace correlation said that two findings in two services tended to come together, and nothing about the order or the gap between them. Every finding of an analysis tick was stamped with the tick time, so only findings analysed in the same tick could pair, median_lag_ms was always 0, and which side was source and which target followed the order of the batch. The daemon also missed a template slow a handful of times in a quarter of an hour but never three times in one batch: it stayed in the duration histogram and never became a finding.
v0.23.0 pairs findings on the time they happened, counts slow episodes across batches, and shows every time a person reads in that person's time zone.
Pairing on the time the findings happened
Findings now pair on their own first_timestamp, across ticks. The earlier one is the source, whatever arrived first, and the lag is the gap between the two first spans rather than between two ticks.
Confidence was inflated as well. Pair counts and source totals were counted over windows out of phase with each other, and a source that had left the window counted as 1. Both counts now share the same half-window buckets, a pair counts in the bucket of its source occurrence, and a pair whose source has no occurrence left in the window is not reported. Expect fewer pairs above min_confidence.
Measured in the simulation lab with traces stamped at chosen times, one trace per batch: A at t and B at t + 5 s report a median_lag_ms of exactly 5000, where 0.22.2 reports the gap between two ticks, 1501. C at t and D at t + 7 s sent D first keep C as the source at 7000. On the lab's usual Java traffic, correlation settles at 24 pairs where the 0.22.2 pass reported 198, with lags from 6 s to 42 s where it read 0.
A window sized by endpoints, not by occurrences
The correlator used to keep every finding occurrence of [daemon.correlation] window_minutes. Pairing now keeps only the findings analysed in the last lag_threshold_ms + 2 x trace_ttl_ms, about a minute of traffic at the default 30 s TTL, and the window only sets the span of the per-pair and per-endpoint counters. Each distinct endpoint is stored once and shared by every pair that names it, so memory follows the number of distinct endpoints in the window rather than the traffic, and a 1440-minute window becomes practical.
window_minutes is validated between 1 and 10080 at config load when correlation is enabled. The median lag keeps 64 samples per pair instead of 256, and when max_tracked_pairs trips, eviction drops the lowest-count pairs first and the stalest among equal counts.
Both ends of a pair
GET /api/correlations gains source_sample_trace_id, the trace of the source-side finding of the latest co-occurrence, next to sample_trace_id on the target side, so both sides of a pair open in Explain. On the HTML report the source side of a correlation card now opens Explain on that trace when no finding of the report matches it, as the target side already did, and the correlations CSV export ends with source_sample_trace_id and sample_trace_id columns.
Slow, but never enough times in one batch
The daemon counts slow occurrences across analysis batches. A template slow in slow_query_min_occurrences separate episodes within [detection] slow_query_window_minutes, 15 by default, yields a slow_* finding with the same shape and signature as a batch cross-trace one, so it folds with them in the findings store. Episodes are counted per type, service, grouping and normalized template, and slow spans within max(60 s, 1.5 x trace_ttl_ms) of an episode's first span count as one, so an isolated slow span or a short lock stays in the histogram. Spans a batch already reported are not counted again, and a key reports at most once per window.
The finding's trace_id is the trace of the slowest span of the episode that fired it, analysed in that batch, so /api/explain finds it. At most 1024 keys are tracked, and the new counter perf_sentinel_slow_window_keys_refused_total counts the slow spans refused past that cap. analyze and the other batch commands are unchanged.
In the lab, three episodes of one slow span each, 65 s apart, give one slow_sql finding with 3 occurrences on the trace that fired it, and nothing with the window at 0 or on 0.22.2.
Times in the reader's time zone
Every time a person reads is now in local time, the browser's on the HTML report and the machine's on the terminal. On the report that is the window under a finding, which dates both sides when they fall on different local days, the span times in the Explain tree, whose tooltip keeps the exact timestamp, and the Expires column of the live-mode acknowledgments table. On the terminal it is the Window: line of analyze, report, diff and inspect, the Period: line of query correlations, and the times printed by ack create and ack list. JSON, SARIF, NDJSON, CSV, archives, periodic reports and the daemon API stay in UTC.
On the terminal the zone is TZ, then the system's. A TZ naming a zone the system database lacks resolves against an IANA database embedded in the binary. That is what makes it work in the FROM scratch container image, which ships no time zone database and would otherwise fall back to UTC without a word. Checked in the lab on that image: under TZ=Asia/Tokyo the Window: line reads nine hours later than under TZ=UTC, and the JSON output is byte-identical between the two.
Four dependencies moved
clap 4.6.7, clap_complete 4.6.11, ureq 3.4.2 and toml 1.1.6, all patch releases and semver-compatible. chrono-tz 0.10.4 joins for the embedded time zone database.
Upgrade impact
- Correlations change. Fewer pairs clear
min_confidence,median_lag_msis no longer 0, and a pair can swap itssourceandtargetagainst 0.22.2, since the earlier finding now leads. A dashboard or an alert keyed on correlation counts will see them drop. - The daemon reports slow findings it used to miss. The window is on by default, so a service with recurring slow templates gains
slow_*findings after the upgrade, with the signatures a batch run would give them.[detection] slow_query_window_minutes = 0turns it off. - Two values now stop the daemon at startup.
[daemon.correlation] window_minutesoutside 1 to 10080 when correlation is enabled, and[detection] slow_query_window_minutesabove 60, are refused at config load instead of running. - Terminal text shows local time. A script that parses the
Window:line or theack listcolumns should setTZ=UTCor read the JSON output, which is unchanged. - Breaking,
perf-sentinel-coreonly.CrossTraceCorrelation,CorrelationConfig,DetectionConfigandMetricsStateeach gain a public field and none is#[non_exhaustive], so code building them by struct literal needs the new field. - One configuration key, one metric, one API field and two CSV columns are added, no route or key is removed, the acknowledgment signatures are untouched, and MSRV stays 1.98.1.
Full detail in CHANGELOG.md.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.23.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.