robintra/perf-sentinel v0.19.0 on GitHub

What's new in v0.19.0

0.18.0 gave the daemon's metrics a service label. The question it raised on the first install that had two of anything was which deployment a service name belonged to. The shipped dashboard could not answer it: its Daemon namespace variable selects where the daemon runs, not where the analysed traffic runs, so a checkout deployed in prod and in staging was one series and one number.

One service, two deployments, one line

perf_sentinel_findings_total, perf_sentinel_slow_duration_seconds and the three perf_sentinel_service_*_io_ops_total counters gain a grouping label next to service. Its value is the finding's effective grouping, the first attribute present from [detection] grouping_attributes, which defaults to k8s.namespace.name then service.namespace, so on Kubernetes it is the namespace the analysed workloads run in. Findings have carried that identity since 0.11.0, the metrics had not.

The label is not called namespace on purpose. Prometheus Operator attaches a target label by that name, and under the chart's honorLabels: true the daemon's value would win, which would break the dashboard's Daemon namespace variable and every namespace=~"$namespace" filter on it.

Caps that count pairs, not values

Cardinality stays bounded by a second gate after the service caps, on admitted (service, grouping) pairs:

findings and analysis-side I/O    512 pairs
slow duration histogram          256 pairs
ingest I/O ops                  4096 pairs
past a cap    the pair keeps its service, only the grouping folds to "_other"

Capping pairs rather than distinct grouping values keeps the bound proportional to the series that actually exist. 11 services in 20 namespaces is 220 pairs and never folds. 100 services in 10 namespaces is 1000 pairs and folds every pair past the 512th, where a cap on grouping values alone would have folded nothing while the series count doubled.

Folding only the grouping half is what keeps the arithmetic intact: sum by (service) over the new label equals the 0.18.0 per-service series exactly, and sum() the pre-0.18 one. Measured on the simulation lab against a replay with the label switched off, 110 groupings across 40 services:

4400 pairs offered
 512 / 256 / 4096 admitted, exactly the three caps
sum by (service)    identical to the unlabeled replay
sum()               identical, 29480 = 29480

Three counters keep the folding observable, perf_sentinel_analysis_grouping_overflow_total, perf_sentinel_slow_duration_grouping_overflow_total and perf_sentinel_service_io_ops_grouping_overflow_total, all counting attributions folded rather than data lost. A tenth tuning advisor rule names the three caps when any of them moves. None ships as an alert, for the same reason as the service overflow counters in 0.18.0: reaching a cap is a state the daemon works normally in.

A waste ratio that cannot exceed 1

A finding whose spans come from several services charges every service's share under the finding's own grouping. That is exact rather than approximate: the N+1 and redundant detectors partition their groups by the grouping identity, so every span a finding charges already carries it, and the per-(service, grouping) ratio of service_avoidable_io_ops_total to service_analyzed_io_ops_total cannot exceed 1. OpenMetrics exemplars follow the split too, keyed on (type, severity, service, grouping).

Upgrade impact

Breaking for any alert or dashboard matching the five series without aggregating, the way 0.18.0 was for service. perf_sentinel_findings_total{severity="critical"} now returns one series per (service, grouping) pair, and an unaggregated alert on it fires per pair. Wrap it in sum(), or sum by (service) to keep the 0.18.0 shape.
[daemon] per_grouping_labels = false restores the 0.18.0 shape, the label stays declared but empty on every series, which PromQL treats as absent. Unlike per_service_labels it governs the three per-service I/O counters as well.
An install already running per_service_labels = false needs per_grouping_labels = false too to keep the histogram's pre-warmed unlabeled series. Since 0.19.0 that series is minted at startup only when both knobs are off, so on the default it appears with the first slow span instead.
The label carries the value alone, without its attribute key. Two spans grouped by different configured attributes that share a value share one series, where the findings keep them apart. Configure a single key when that distinction matters.
Breaking, perf-sentinel-core only. DaemonConfig gains per_grouping_labels, GreenSummary.avoidable_per_service is keyed by (service, grouping) instead of the service alone, and GreenSummary gains an in-process analyzed_per_service with the same key. Neither struct carries #[non_exhaustive]. MetricsState's public vectors change arity, so a downstream with_label_values at the 0.18 arity panics at runtime. DetectConfig, score_green, snapshot_service_io_ops, PerEndpointIoOps and the acknowledgment signatures are unchanged.
The shipped Grafana dashboard gains a Grouping variable ahead of Service, reaching the eleven panels of the row now titled Analysis (filtered by grouping and service), and narrowing the service list to the groupings picked.
/api/config reports per_grouping_labels next to per_service_labels, and the query monitor Config tab shows it. Acknowledgment signatures, detection verdicts, CLI surface and the rest of the daemon API are unchanged, MSRV stays 1.97.1.

Full detail in CHANGELOG.md.

Verifying this release

# Binary integrity via SLSA Build L3 attestation
gh attestation verify perf-sentinel-linux-amd64 \
  --repo robintra/perf-sentinel

# A periodic disclosure produced by this binary
perf-sentinel verify-hash --report perf-sentinel-report.json \
  --expected-identity "https://github.com/robintra/perf-sentinel/.github/workflows/release.yml@refs/tags/v0.19.0" \
  --expected-issuer "https://token.actions.githubusercontent.com" \
  --verify-binary ./perf-sentinel-linux-amd64

gh CLI 2.49 or newer required for gh attestation verify.