Podtrace v0.11.0 — Kubernetes-native operator, agent, and session runtime

This is the largest release since the project began. v0.11.0 turns Podtrace from a CLI into a full Kubernetes operator with declarative PodTrace, PodTraceSession, ExporterConfig, and TracerConfig CRDs, a per-node agent DaemonSet, a session Job runtime that wires the existing --diagnose flow to Kubernetes, and a deploy/operational chart layer that gives you a working trace fleet end-to-end with helm install.

The standalone podtrace CLI is unchanged, every flag, every output, every test on the existing binary still works.

Highlights

• Four podtrace.io/v1alpha1 CRDs with admission validation:
  • PodTrace — continuous tracing over a dynamic pod selector
  • PodTraceSession — bounded "diagnose this pod for 30s" jobs
  • ExporterConfig — central exporter declaration (OTLP / Jaeger / Zipkin / Splunk / DataDog), referenced by name
  • TracerConfig — cluster-wide agent + session image and runtime knobs
• Operator with leader election and three reconcilers (TracerConfig, PodTrace, PodTraceSession). Cross-namespace cleanup via finalizers; per-PodTrace exporter bundle synced into podtrace-system for agents to consume.
• Per-node agent DaemonSet that watches PodTraces cluster-wide, merges all active CRs into one shared eBPF tracer, routes events to the right exporter per CR, and patches per-node status back via Server-Side Apply (no clobbering across nodes).
• Session Jobs that mount the operator-rendered exporter bundle and call podtrace --diagnose, three parallel artifact channels (summary file, kubelet termination message, ConfigMap/Secret upload) so the result of a session is durably visible regardless of how the Job ended.
• Helm install gives you a working fleet: chart now renders a default TracerConfig, optional Prometheus ServiceMonitor / PodMonitor, and an opt-in OTLP starter ExporterConfig.
• Hardened security posture: all systemic suppressions replaced with scoped os.Root helpers and saturating integer conversions; a overflow bug in alert thresholds fixed at the source.

What's new

Custom resources and webhooks (#69)

Four CRDs with kubebuilder validation markers, generated typed clientset under pkg/client/clientset/versioned/, and a validating webhook that enforces invariants the CRD schema can't express:

• spec.selector and spec.podRefs are mutually exclusive — exactly one is required.
• spec.exporterRef.name must resolve to an ExporterConfig in the same namespace.
• ExporterConfig.spec.type must match the populated typed sub-field (otlp: with type: otlp, etc.).

A stable pkg/tracer engine seam (Engine, Exporter, TracerBackend, Target, TargetSet) becomes the single boundary between the eBPF core and every operational mode (CLI, agent, session job). The Helm chart, multi-stage Dockerfile, and make generate | manifests | clientset | docker-build | helm-lint | helm-template | envtest workflow all land here.

Operator control plane (#70)

podtrace operator now runs a controller-runtime manager with three reconcilers:

• TracerConfigReconciler — owns the agent DaemonSet, ServiceAccount, ClusterRole/Binding, and (new in #73) a namespaced bundle Role.
• PodTraceReconciler — resolves the referenced ExporterConfig, materializes a per-CR exporter bundle ConfigMap (+ optional credential Secret) in podtrace-system, rolls per-node statuses up into top-level conditions.
• PodTraceSessionReconciler — fans the session out into one Job per node hosting matched pods, enforces an optional per-node concurrency cap, drives TTL-based cleanup.

Cross-namespace child cleanup uses a single shared finalizer; the operator runs unprivileged. End-to-end smoke verified on kind.

Per-node agent DaemonSet (#71)

The agent watches PodTraces cluster-wide, Pods on its own node, and exporter bundles in podtrace-system. Concrete behaviors:

• Multi-CR merge router — every event is dispatched to the subset of per-CR exporters whose cgroup set contains the event's CgroupID and whose filter set includes the event type. One shared eBPF pipeline serves N overlapping CRs.
• Exporter cache keyed by bundle ResourceVersion — credential rotation rebuilds the OTLP exporter; no-op reconciles don't reopen connections.
• Server-Side Apply per-node status — the field-owner is podtrace-agent-<NODE_NAME> so many agents write status.nodeStatus[] concurrently without clobbering each other.
• Health probes — /healthz (stall-window liveness) + /readyz (flips true post-cache-sync).
• Prometheus metrics — podtrace_agent_{events_exported_total, events_dropped_total, active_cgroups, active_crs, reconcile_total}, labeled per CR.

OTLP is the only concrete agent-side exporter for now; Jaeger/Zipkin/Splunk/DataDog return an explicit "not yet implemented" so a Degraded condition surfaces it on kubectl describe.

Session Jobs (#72)

PodTraceSession is the bounded-diagnose CRD. The operator now creates one Job per matched node that mounts the per-session exporter bundle at /etc/podtrace/exporter/, runs the existing podtrace --diagnose CLI with three new flags, and captures the result through three parallel channels:

• --summary-file <path> — full JSON summary on a shared emptyDir.
• --termination-message-path <path> — compact JSON on /dev/termination-log so kubelet surfaces it in Pod.Status.ContainerStatuses[…].State.Terminated.Message.
• --report-to <kind>/<ns>/<name> — self-uploads the human-readable report to a ConfigMap or Secret via the in-cluster ServiceAccount.

A new podtrace report-uploader subcommand provides a native sidecar mode for clusters that need re-upload-on-completion semantics (TracerConfig.spec.session.sidecarUploader: true).

The session Job runs as a dedicated podtrace-session ServiceAccount with a narrow per-session Role + RoleBinding in the user namespace, not the agent SA. RBAC is scoped to resourceNames of the specific report sink.

A shared pkg/exporter/bundle package now provides the canonical Payload type plus FromConfigMapData/ToConfigMapData and FromYAML/ToYAML codecs, so the operator, agent, and CLI all consume byte-identical exporter config.

End-to-end verified on kind: a session against an nginx pod produces phase: Completed, aggregated status.summary, per-Job eventCount, and a human-readable report in a user-namespace ConfigMap.

Deploy / operational chart layer (#73)

helm install podtrace deploy/charts/podtrace now produces a working trace fleet end-to-end:

• Default TracerConfig rendered from values.yaml (image, agent runtime, session caps, scheduling, BTF mode, sidecar uploader). Default-on with the operator; opt out via tracerConfig.create=false.
• ServiceMonitor + PodMonitor templates for Prometheus scrape config (operator :8080, agent :9090). Double-gated on toggle and monitoring.coreos.com/v1 API availability — enabling them on a cluster without prometheus-operator is a silent no-op rather than a hard install failure.
• Opt-in starter OTLP ExporterConfig so a fresh install can run a sample diagnose without first authoring exporter config.
• Refined metrics.serviceMonitor / metrics.podMonitor values — labels, interval, path.
• Post-install NOTES surface what was rendered and the right next-step command.

Agent RBAC narrowed: the agent ClusterRole no longer grants cluster-wide configmaps/secrets reads. Bundle reads are scoped to podtrace-system via a namespaced Role + RoleBinding the TracerConfigReconciler materializes alongside the agent ServiceAccount.

Operator RBAC: gains roles/rolebindings create/update/delete (per-session RBAC) and delete on podtrace.io CRDs (TTL-driven session cleanup — pre-existing bug fixed in passing).

ExporterConfig → PodTrace watch wired, so changes to a referenced ExporterConfig requeue every PodTrace that references it.

Documentation: full per-CRD reference docs (PodTrace, PodTraceSession, ExporterConfig, TracerConfig) plus a Chainsaw e2e suite that exercises lifecycle invariants — continuous, session, multi-CR overlap, agent restart, credential rotation, PSA enforcement.

Security hardening (#74)

• Five new helper packages that own the trust boundaries:
  • internal/safeconv — saturating integer conversions; replaces 19 G115 sites.
  • internal/procfs — os.Root-scoped reads under /proc (~25 G304 sites).
  • internal/sysfs — os.Root-scoped reads under /sys/fs/cgroup.
  • internal/ldsoconf — os.Root-scoped /etc/ld.so.conf* parsing.
  • internal/hostfs — validated boundary helpers (Stat, WalkRegular, ReadFile, WriteFile) for paths that legitimately cross os.Root's scope.
• Real correctness fixes that the suppressions had been hiding:
  • AlertWarnPct/CritPct/EmergPct are now clamped to [0, 100] at config-load time. A misconfigured PODTRACE_ALERT_WARN_PCT=4294967296 previously wrapped to 0 and made every event trip a "WARNING" alert; it now clamps to 100.
  • BPFHashMapSize and RingBufferSizeKB are clamped at the source via config.ClampUint32, with overflow-safe multiplication for the ring-buffer byte count.
  • EventResourceLimit utilization comparisons run in int end-to-end with a defensive negative-value guard.
  • string(rune(event.PID)) in internal/diagnose/tracker/trace_tracker.go (which produced a single Unicode codepoint per PID and silently broke for PIDs above 0x10FFFF) is replaced with strconv.FormatUint.
• Envtest path fix: make envtest now runs the full CRD validation suite (api/v1alpha1, operator, agent) again — three suite_test.go files were looking under deploy/charts/podtrace/templates/crds/ but the chart layout puts CRDs at deploy/charts/podtrace/crds/.

Install / upgrade

helm upgrade --install podtrace oci://ghcr.io/podtrace/charts/podtrace --version 0.1.0 \
  --namespace podtrace-system --create-namespace

The chart provisions:

• The podtrace-system namespace with PSA privileged.
• All four CRDs with crds.install/crds.keep toggles.
• Operator Deployment + RBAC + cert-manager Certificate for the validating webhook.
• A default TracerConfig (so the agent DaemonSet rolls out immediately).

The standalone podtrace CLI image works exactly as before; the operator and agent are additive.

Backwards compatibility

• The existing podtrace CLI binary is unchanged. Every flag, environment variable, output format, and test that worked on v0.10.0 still works on v0.11.0.
• The default values for AlertWarnPct/CritPct/EmergPct (80 / 90 / 95), BPFHashMapSize (4096), and RingBufferSizeKB (2048) are identical. Sane configs flow through the new clamp helpers unchanged.

Known gaps / follow-ups

• Agent-side non-OTLP exporters — Jaeger, Zipkin, Splunk, and DataDog return an explicit "not yet implemented" from the agent's BuildExporter and surface as a Degraded condition. The CLI path supports them.
• Cross-namespace NamespaceSelector evaluation — presence of spec.namespaceSelector opens cross-namespace matching today; honoring its label expressions requires a Namespace informer not yet wired.
• spec.reportRef.objectStore — the field is part of the schema so clients can adopt it ahead of the upload path going live, but sessions that set it today are rejected by the webhook.

Full diff

Full Changelog: v0.10.0...v0.11.0

What's Changed

• feat: scaffold Podtrace Kubernetes operator with v1alpha1 CRDs, validating webhooks, tracer engine seam, and Helm chart by @gma1k in #69
• feat: implement Podtrace operator with TracerConfig, PodTrace, and PodTraceSession reconcilers plus exporter-bundle sync and kind e2e smoke by @gma1k in #70
• feat: implement podtrace per-node agent DaemonSet runtime with multi-CR merge router, exporter-bundle consumption, SSA status writer, and Prometheus metrics by @gma1k in #71
• feat: add PodTraceSession Job runtime with operator-mounted exporter bundle, CLI session sinks, per-session narrow RBAC, kind-smoke verification of the full diagnose and Completed path by @gma1k in #72
• feat: ship deploy operational chart layer TracerConfig, monitoring templates, narrowed agent RBAC and the CRD documentation set with a Chainsaw e2e suite by @gma1k in #73
• chore: fix gosec/CodeQL findings via scoped fs/conv helpers, fix envtest CRD path, bump to v0.11.0 by @gma1k in #74