What's new in v0.17.0
A report now opens on what costs the most, in the dashboard, the TUI and the JSON. Building that turned up two things that had been quietly wrong: the sort applied its own inverse, and the backend query subcommands threw away spans they had already correlated. Separately, a daemon on a memory limit no longer gets OOM-killed by a wide topology.
The correlator that OOM-killed the daemon
ingest collected every pair the cross-trace correlator refused at the max_tracked_pairs cap into a HashSet whose only outputs are a length and a boolean. That set is sized by the batch's findings times the occurrences inside lag_threshold_ms, never by the cap.
1500 services, 80 traces/s, 15 s batch cadence
~1.6 M distinct keys per batch
34 MiB table, 51 MiB live across the doubling rehash
inside a 256 MiB container
Lowering max_tracked_pairs tenfold made the peak worse rather than better: a refused pair is by definition one the cap did not admit, so the cap can remove at most its own size from a set of millions. Refusals are now deduplicated up to 8192 distinct pairs per batch and counted past that, which holds the structure near 140 KiB. perf_sentinel_correlator_pairs_evicted_total stays exact for every topology small enough for an exact number to tell an operator anything.
A deployment that worked around this by disabling correlation can turn it back on.
Impact first, and the sort that ran backwards
tempo, jaeger-query and report accept --sort, the severity and impact keys analyze already took. It applies before the sinks, so a wide sweep comes back ranked instead of leaving the caller to re-sort it.
without --sort opens on n_plus_one_sql at 5 avoidable ops
--sort impact opens on n_plus_one_http at 39
The dashboard and the TUI now open on impact too, and the impact control comes before the severity one in both. A reader arriving on a report is asking what costs the most, so the answer leads.
The rewrite that made this shared had a defect worth naming: it applied the inverse permutation, so on any input whose sort permutation holds a cycle of three or more, --sort impact could open on the lowest-impact findings and hand the trace embed the wrong rows. The pinning test asserted only the first element, which an inverse rotation also satisfies. Two tests now pin the full order, one of them a three-cycle that fails on exactly this mistake.
The spans the backend queries had already thrown away
tempo and jaeger-query called pipeline::analyze, which is analyze_with_traces(...).0. The spans were correlated and then dropped, so a report rendered from --format json announced it had no span tree. The effect was sharpest in trace-ID mode, where an operator fetches exactly one trace by its identifier and got back a report that refused to draw it.
Both subcommands now carry the masked spans of their findings' traces. The field is skip_serializing_if = "Vec::is_empty" and only traces a finding points at are carried, so a clean run grows by zero bytes and archived baselines stay byte-identical. inspect --input gains the same trees.
The selection rule behind --max-traces-embedded now lives once, in report::embedded, instead of three hand-rolled copies, and the embed happens after the sort rather than before it. Trimmed first, a report kept the detector-order head and could open on its highest-impact finding while showing it without a tree.
Caps this client applies, named as such
tempo --max-traces is bounded to 1..=10000, the range jaeger-query already enforced. It was a bare usize, so --max-traces 999999999 parsed, reached the backend, and failed downstream on a limit nobody had named.
The Tempo search-response cap moves from 1 MiB to 16 MiB. It was sized for the limit=500 its own comment cites and then applied to every value, so a legitimate --max-traces 5000 died on a limit of this client's own making. Measured: a realistic Tempo summary carrying the root service, name, duration and span set runs about 210 bytes, which put the old ceiling near 4700 traces.
A body that outgrows one of these now says whose limit it hit and which remedy applies. A search overrun says to lower --max-traces, a single trace past the per-trace cap says the flag cannot shrink it.
Upgrade impact
- Two defaults change. The dashboard and the TUI open on impact rather than on severity and trace-id order, and
reportsorts by impact when no--sortis given.sstill cycles the TUI through all three. - The ingest log line is shared.
tempoandjaeger-querylogIngested events from the backendwith the backend in abackendfield, where each used to name itself in the message text. Grepping the old literal has to match the field instead. - Two dashboard colours move, both to clear the 4.5 WCAG AA floor for text that size. Light
--brand-textgoes#11845dto#107a56, which had fallen to 4.09, 4.12 and 4.37 against three surfaces. Dark--text-3goes#7d8c83to#95a199, at 3.97 and 3.50. - No library break this time.
perf-sentinel-corekeeps its 0.16.0 signatures, the CLI surface only gains flags, the daemon API is untouched, and MSRV stays 1.97.1.
Full detail in CHANGELOG.md.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.17.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.