What's new in v0.11.0
v0.11.0 is about telling one deployment from another. A finding carries where it happened, not just what happened, and every surface that groups, compares or correlates keeps two deployments apart instead of folding them into one line. Along the way, a scraper that never asked for exemplars stopped being served them, a batch run stopped scoring with an estimate while an operator believed it was measuring, and the report finally draws the warnings it had been carrying all along.
One finding per deployment, on an attribute you choose
A shared cluster runs the same service for several namespaces or several customers, and until now perf-sentinel had no way to say so. The same N+1 in catalog-svc on prod and on staging folded into one finding with the occurrence counts added together, and an operator reading it could not tell whether the problem was everywhere or in one place.
[detection] grouping_attributes names the resource or span attributes that separate one deployment from another, defaulting to ["k8s.namespace.name", "service.namespace"]. The first one present on a span decides finding identity, every present one is captured, and each surface labels the value with the attribute it came from, so tenant.id=prod never reads as k8s.namespace.name=prod. A shared cluster groups by tenant.id with a one-line config change and no code.
There is no tenant in the OpenTelemetry semantic conventions, which is why this is a list you configure rather than a field the tool assumes. Nothing is hardcoded to Kubernetes: the defaults are two stable conventions, and any attribute your instrumentation emits works.
The configured order reaches every ingest door alike, batch JSON, daemon OTLP over gRPC and over HTTP, Tempo and the Jaeger Query API. Getting that wrong on one door would have made findings from that source silently merge, so the identity is selected once, before sanitization caps the attribute list, and shared behind a single accessor rather than reconstructed per call site.
Identity is the pair, not the value. Recurrence grouping, diff identity, cross-trace correlation pairing and the HTML culprit index all key on (key, value), so a prod that came from tenant.id never folds into a prod that came from k8s.namespace.name. Slow findings split by grouping inside a single trace, chatty calls count per grouping, and /api/correlations stopped inventing a causal link between separate deployments.
Acknowledgment stays deliberately grouping-blind. Grouping answers who is affected and where, an acknowledgment answers whether the code is accepted debt, and that decision does not vary by deployment. One acknowledgment therefore covers every grouping and reordering the list never invalidates one. Since a row-level Ack silencing a sibling row would be a surprise, the acknowledgment dialog names every grouping it is about to cover, including the ungrouped rows.
A scraper was minting a metric name per scrape
/metrics served OpenMetrics exemplars to any scraper whose Accept header contained a */* wildcard, treating it as an opt-in. vmagent sends Accept: text/plain;version=0.0.4;*/*;q=0.1, a non-RFC form that asks for Prometheus text and tolerates anything else at low quality. It received the exemplar suffix, which its parser cannot read, and took perf_sentinel_io_waste_ratio 0.60 # {trace_id="..."} 1.0 for a metric name.
One dead series per scrape. A real cluster accumulated 6,216 junk metric names against 41 real ones in a few hours.
Exemplars now require an explicit, non-refused application/openmetrics-text. A wildcard is not an opt-in, and every valid zero quality form from q=0 through q=0.000 is honored as the refusal it is.
A batch run said nothing about the backend it ignored
Energy and grid-intensity backends are scraped by the watch daemon and by nothing else. An operator who configured [green.alumet] and ran analyze got carbon figures computed from the I/O proxy estimate over embedded intensity data, with no indication that the backend had been read and ignored. The same data, run through the daemon, produced different numbers.
A batch run that finds any of the seven configured now says so, names the sections it ignored, and states that every carbon figure in the report is the estimate. It does not scrape: attributing the power drawn right now to traces recorded yesterday would produce a wrong number rather than an absent one.
disclose gained the matching honesty. A period mixing pre-canonical archives with newer ones passes --intent official validation, because the canonical threshold is the maximum across windows, while the canonical tier silently omits the older ones. It now reports how many windows carried no canonical waste figure.
The dashboard draws the warnings it was already carrying
Report-level warnings were embedded in the HTML payload and never drawn. The min_usable_span_ratio warning, whose whole purpose is to stop a false green, had been invisible in the dashboard since it shipped.
They render in a banner placed outside the tab panels, so a warning about the run stays readable whichever tab you are on. It is informational by default and takes the warn tone only for a kind that reports lost data, since a tuning hint claiming something is broken is its own kind of noise. Code fragments in warnings and suggestions read as code: [green.alumet] and `WHERE ... IN (?)` are chips in the report and plain text in the terminal, the TUI and SARIF.
Evidence you can audit
The finding detail and the CSV export now carry the captured attributes, the classification, the observation window, the timing statistics and the masked evidence. Culprit spans are listed individually rather than as a count, so the claim "14 near-identical queries" can be checked against the 14 rows that produced it.
A cross-trace finding states how much of its total is present in the representative trace, since the tree shows one trace and the count spans many. A report produced before this release cannot rebuild exact span ids, so it keeps matching spans grouped rather than pretending to a precision it does not have.
The dashboard filters on the exact finding type and on the effective key=value grouping. A span clock is labelled UTC only when the timestamp says so, and zero-duration spans stay out of the peak evidence.
Tempo told an empty result from an unusable one
A trace whose spans all filter out, a SERVER-only hop or a span with no db.system, reported no traces found for the given search criteria. Tempo had returned the trace all along, and the operator went hunting for something that was never missing. The two cases now have distinct errors, and the new one names what to look at.
Upgrade impact
Existing acknowledgments stay valid. No signature changed, and acknowledgment remains grouping-blind by design.
Expect findings to split. If your spans carry k8s.namespace.name or service.namespace, findings that previously folded across deployments now separate, so counts per finding drop while the finding count rises. This is the point of the release, but it will change the shape of a report you know by heart. Set grouping_attributes = [] to keep the previous behaviour.
A diff against a pre-0.11.0 baseline stays comparable. The comparison detects that the baseline carries no grouping and falls back to the old identity, rather than reporting every finding as new.
This is a minor bump, not a patch, for a Rust source-compatibility reason. SpanEvent and Finding gained public fields. Older JSON without the grouping fields stays readable everywhere, and the archived per-window JSON the daemon writes keeps its shape, so a disclosure content_hash computed on 0.10.0 verifies unchanged here.
One /metrics behaviour change. A scraper that does not explicitly accept application/openmetrics-text no longer receives exemplars. If you relied on a wildcard Accept to get them, send the media type.
Validation
The full Rust suite passes (3062 tests), clippy is clean at -D warnings across all targets, cargo fmt --check passes, and the default and --no-default-features builds both pass. The browser suite covering the HTML dashboard passes.
The simulation lab validated the release against the surfaces it touches rather than in full, which the ledger entry records. The first run remained fail-closed and found a real defect, a grouping value missing from the correlations CSV export, which was fixed before the rerun passed all 17 grouping assertions. A new grouping-identity scenario covers the dimension end to end, and the metrics, Tempo, Jaeger, Zipkin, PostgreSQL, MySQL, configuration-warning, HTML, CSV, correlation and disclosure scenarios were extended to assert it.
A Tempo concern raised during the gate was proved not to be a regression: identical trace payloads produce the same event counts on 0.10.0 and 0.11.0, and 0.11.0 additionally preserves the expected grouping.
Verifying this release
# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
--repo robintra/perf-sentinel
# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
--expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.11.0" \
--expected-issuer "https://token.actions.githubusercontent.com" \
--verify-binary ./perf-sentinel-linux-amd64gh CLI 2.49 or newer required for gh attestation verify.
Full Changelog: v0.10.0...v0.11.0