robintra/perf-sentinel v0.5.16

on GitHub

What's new in v0.5.16

v0.5.16 consolidates four findings surfaced by simulation-lab validation after v0.5.15 was deployed on a real cluster.

The daemon's /metrics endpoint now selects its content type from the client's HTTP Accept header in three modes. Pre-0.5.16 the format was decided by has_exemplars() alone. A client requesting Accept: application/openmetrics-text; version=1.0.0 against a freshly-started or quiet daemon received text/plain; version=0.0.4 instead of the OpenMetrics 1.0 payload it asked for, defeating strict scrapers. The new dispatch is keyed off the header: clients explicitly requesting OpenMetrics get an OpenMetrics 1.0 conformant body (with # EOF and exemplar annotations when present) regardless of metric set state, clients sending no header or */* keep the legacy 0.5.15 behavior (OpenMetrics when exemplars are present, plain Prometheus otherwise), and clients accepting only text/plain without */* get a strict plain-Prometheus body with no exemplars and no # EOF. The legacy fallback is intentional, it preserves vmagent and curl by default so existing Grafana exemplar pipelines do not regress.

The /api/export/report cold-start path now returns 200 OK with an empty Report envelope instead of 503 Service Unavailable. Pre-0.5.16 returned 503 with {"error": "daemon has not yet processed any events"}, which tripped Kubernetes liveness probes targeted at the endpoint and confused CI scripts that treated 5xx as a daemon health issue. The new shape is a complete Report with findings: [], green_summary: GreenSummary::disabled(0), analysis.events_processed = 0, and warnings: ["daemon has not yet processed any events"] (a new additive field). Operators detect cold-start via the warnings array or analysis.events_processed == 0 instead of via the HTTP status code. The double-counter guard (events_processed_total > 0 AND traces_analyzed_total > 0) is preserved internally so the snapshot stays self-consistent during the trace_ttl_ms / 2 window between the first event ingest and the first eviction tick. This is a behavior change at the HTTP status code level. Consumers that explicitly switched on 503 must update their checks. The Electricity Maps audit chip continues to surface on cold-start snapshots: green_summary.scoring_config is re-applied from the daemon's startup config on both the cold-start and the warm path, locking in the 0.5.12 audit-trail contract.

The MAX_JSON_DEPTH = 32 cap (uniformized in 0.5.15 across Native, Jaeger and Zipkin ingest paths) is now exercised at the boundary by six new tests: depth-31 must parse cleanly, depth-33 must reject with PayloadTooDeep, for each of the three formats. The depth-50 stress tests added in 0.5.15 stay in place.

The runbook gains a "Inspecting the daemon's HTTP endpoints" section (EN + FR) documenting the kubectl port-forward + curl pattern required to inspect a distroless daemon image. The daemon image does not ship curl or wget, so kubectl exec ... -- curl http://localhost:14318/... fails with executable file not found. The recommended path is kubectl port-forward followed by a local curl from the operator's host. The kubelet liveness probe uses a TCP check, not an HTTP call, so the distroless image does not affect liveness or readiness.

Changed

• /metrics content negotiation now keys on the Accept header. Three-mode dispatch documented above. The OpenMetrics media-type detection is token-aware and case-insensitive, and skips tokens explicitly refused via q=0 per RFC 7231 §5.3.1. Hostile or unrelated tokens such as application/openmetrics-text-foo do NOT trigger the OpenMetrics path. The */* wildcard detection remains substring-based to handle the non-RFC variant some scrapers (vmagent in particular) emit, where */* appears as a parameter rather than a comma-separated token.
• /api/export/report cold-start path returns 200 OK with an empty envelope. Pre-0.5.16 returned 503 Service Unavailable. New shape includes warnings: ["daemon has not yet processed any events"]. Use analysis.events_processed == 0 or the warnings field as the new signal.

Added

• Report.warnings: Vec<String> field, additive on pre-0.5.16 baselines via #[serde(default, skip_serializing_if = "Vec::is_empty")]. Populated by the daemon's cold-start path. Empty in CLI batch output (pipeline::analyze). The report --before <baseline> flow continues to parse any 0.5.x baseline.
• Six boundary tests on the MAX_JSON_DEPTH = 32 guard, two per format (depth-31 must parse, depth-33 must reject) for Native, Jaeger and Zipkin.
• Nine tests on /metrics Accept negotiation covering all three modes at unit level (MetricsState::negotiate(accept)) and integration level (axum router with explicit OpenMetrics and vmagent-style headers). The select_format dispatch helper has dedicated coverage for substring attacks and q=0 refusal.
• One cold-start scoring_config propagation test to lock in the Electricity Maps audit chip on the cold-start path.
• Distroless inspection guide in docs/RUNBOOK.md (EN) and docs/FR/RUNBOOK-FR.md (FR), with the kubectl port-forward + curl pattern and a summary of the three-mode Accept negotiation.

Behavior

• Report.warnings is the canonical signal for "daemon is in cold-start" on /api/export/report. The HTML dashboard does not yet render a banner for this field (planned for a future release), CLI tools and consumers can detect it programmatically.
• Legacy /metrics callers unchanged. Test helpers and CLI batch callers that invoke MetricsState::render() or MetricsState::content_type() (no Accept header) continue to receive the 0.5.15 behavior. Both helpers are now wrappers around negotiate(None).
• /api/export/report Prometheus counter export_report_requests_total continues to bump on every request, including cold-start responses, consistent with HTTP access-log conventions and identical to the 0.5.13 behavior.
• No SARIF, JSON CLI, terminal, or HTML output format change beyond the additive Report.warnings field. Existing dashboards, baselines, and SARIF integrations parse without code change.

Documentation

• docs/QUERY-API.md (EN) and docs/FR/QUERY-API-FR.md (FR) describe the 200-with-empty-envelope cold-start path.
• docs/RUNBOOK.md and FR mirror gain the distroless inspection section.

Compatibility check

• VictoriaMetrics / vmagent: vmagent does not advertise application/openmetrics-text in its scrape Accept header (per VictoriaMetrics issue #9239), but it does include the */* wildcard in its non-RFC text/plain;*/*;q=0.1 form. The new dispatch routes that header to the legacy mode, preserving exemplars on the Grafana click-through path. The substring */* detection handles both the RFC-correct text/plain, */* and the vmagent semicolon variant.
• Prometheus: Prometheus advertises application/openmetrics-text in its scrape Accept header. With the new dispatch it now receives a fully conformant OpenMetrics 1.0 body (with # EOF) on cold-start and quiet windows where 0.5.15 served plain Prometheus.
• VictoriaTraces: unaffected. The depth-boundary tests sit in JsonIngest::ingest (local file or stdin path), not in the remote jaeger-query subcommand that talks to VictoriaTraces.

Install

Prebuilt binaries (Linux amd64 / arm64, macOS arm64, Windows amd64):

curl -LO https://github.com/robintra/perf-sentinel/releases/download/v0.5.16/perf-sentinel-linux-amd64
chmod +x perf-sentinel-linux-amd64
sudo mv perf-sentinel-linux-amd64 /usr/local/bin/perf-sentinel

Linux binaries are statically linked against musl and run on any distribution (Alpine, Debian, RHEL, Ubuntu any version) regardless of glibc version, and inside FROM scratch images.

From crates.io:

cargo install perf-sentinel --version 0.5.16

Docker:

docker run --rm -p 4317:4317 -p 4318:4318 \
  ghcr.io/robintra/perf-sentinel:0.5.16 watch --listen-address 0.0.0.0

Also available on Docker Hub: robintrassard/perf-sentinel:0.5.16.

Helm (chart 0.2.19 ships 0.5.16 as its appVersion default):

helm install perf-sentinel oci://ghcr.io/robintra/charts/perf-sentinel \
  --version 0.2.19 \
  --namespace observability --create-namespace

Verify the binary against SHA256SUMS.txt:

curl -LO https://github.com/robintra/perf-sentinel/releases/download/v0.5.16/SHA256SUMS.txt
sha256sum -c SHA256SUMS.txt --ignore-missing

Full diff: v0.5.15...v0.5.16