Release v1.24.0

This release adds new-node validation, concurrent node-partitioned event processing in the health-events-analyzer, DCGM 4.7.0 support, and opt-in KOM policies for NPD GPU system-service conditions. It also removes preflight's Kubernetes API rate limits, which is a breaking change for some configurations — please read the Upgrade Notes below.

Upgrade Notes

Preflight Kubernetes API rate-limit settings removed (#1849, #1848)

Breaking change for some configurations.

Preflight's --kube-api-qps and --kube-api-burst flags are removed, along with their Helm values, schema entries, and documentation. Preflight now takes its REST config from ctrl.GetConfig() and uses controller-runtime's connection defaults.

The default of 5 QPS was actively harmful because preflight sits on the main path of workload creation. In a measured 1,028-pod gang, throttling left 51 of 1,028 peers registered with server-side admission P99 at 24,850 ms, and the gang never recovered — the coordination ConfigMap advertised a quorum that never formed, which is worse than registering nothing. Preflight drew 0.01 cores while stalled, waiting on its own limiter rather than the API server. Removing the limit rather than re-defaulting it avoids replacing one guessed number with another, since any fixed value must be chosen against the largest gang the product will admit.

What this means for you:

Standard Helm deployments will not break. The chart no longer emits either flag, so the new behaviour reaches the deployment.
If you set qps or burst in your preflight values, the schema removal surfaces it at install time — that is deliberate, so the setting is not silently ignored. Remove those keys from your values.
If you run preflight outside Helm and pass either flag directly, the binary rejects the unknown flag and the pod enters CrashLoopBackOff with exit 2. Drop the flags from your invocation before upgrading.

Note that this removes rate-limit configurability for preflight only. The equivalent settings for fault-quarantine, node-drainer, and labeler are unaffected.

Percona version sets are now validated at render time (#1831)

mongodb-store gains a render-time guard that refuses version combinations that cannot work, following the same fail pattern as the existing oplog and auth validations. A render is refused when crVersion is more than one minor behind the operator, when major versions differ, when crVersion is ahead of the operator, or when initImage.tag differs from the operator tag (the init container runs the operator image, so a mismatch silently runs a different operator build). A new mongodb-store.psmdbVersion value lets you declare the intended operator version once and have all three checked against it.

One minor of skew is permitted deliberately, because that is Percona's documented upgrade path. If your deployment is currently in a refused state — most likely crVersion two minors behind the operator — helm upgrade will now fail rather than render. That is the intended behaviour: it surfaces a combination Percona does not support. Reconcile your Percona values before upgrading.

A companion opt-in check, mongodb-store.validateDeployedCrVersion (default false), extends this to the deployed resource (#1850). The render-time guard only compares chart values with each other, so it cannot see a live PerconaServerMongoDB still carrying the crVersion it was installed with — and lookup returns empty under helm template, leaving every ArgoCD user blind to it. When enabled, an init container on the bootstrap Job reads the deployed crVersion and applies the same rule.

Major New Features

New-Node Validation (#1821)

Adds the node-validation-controller, which batches nodes matching configured criteria and creates a single ValidationRequest covering the whole batch — so newly joined nodes are validated before they take work. Enabling it requires the validation-request-controller plus a spec.newNodeValidation block in the ValidationConfiguration:

newNodeValidation:
  condition: NewNodeValidated
  batchPeriodSeconds: 900
  criteria:
    - name: recently-joined
      expression: >-
        has(node.metadata.creationTimestamp) &&
        now() - timestamp(node.metadata.creationTimestamp) < duration("30m")
    - name: gpu-present
      expression: >-
        has(node.metadata.labels) && "nvidia.com/gpu.present" in node.metadata.labels
  newNodeTests:
    - dcgm-diag-test

Concurrent Node-Partitioned Event Processing (#1817, #1800)

The health-events-analyzer processed events serially at roughly 57 events/s. Events are now routed by node-name hash into dedicated FIFO worker channels, so events for the same node still process strictly in order while events for different nodes process concurrently. A low-water-mark tracker guarantees that out-of-order completion never checkpoints past an earlier unresolved event across restarts or crashes, and a MaxInFlight bound applies backpressure by pausing stream reads when worker queues or uncheckpointed events reach capacity. The design is recorded in ADR-056. The default remains workers: 1, so throughput is opt-in.

DCGM 4.7.0 Support (#1866, #1759)

Updates DCGM to 4.7.0 on Ubuntu 24.04 across the GPU health monitor, preflight diagnostics, and lifecycle diagnostics, with compatibility handling for older 4.x servers: thermal-margin and power-brake readings use a version-compatible API when client and server differ, and monitoring continues when an older server does not support the optional GPU serial-number request. Adds imexMonitoringEnabled (disabled by default), handling both the dedicated IMEX watch and older servers that report IMEX errors under NVLink. Recovery events are now sent for individual GPU and NVSwitch error codes rather than only when a whole watch clears. ConnectX monitoring is deliberately excluded until the GPU health monitor supports NIC discovery, identities, and health events.

Host-Native Authentication for metadata-collector (#1837, #1836)

metadata-collector can now authenticate as a host-native process while keeping hardware inventory and pod-to-GPU mapping intact, via --kubeconfig for the Kubernetes API and --kubelet-kubeconfig for kubelet HTTPS. Endpoints, credentials, and CA trust stay separate, and explicit configuration requires authenticated, verified HTTPS. In-cluster defaults are preserved and a home-directory kubeconfig is never loaded implicitly. Inventory and annotation schemas are unchanged. Recorded in ADR-058, which extends the deployment options in ADR-010 without changing publisher authentication.

Opt-In KOM Policies for NPD GPU System Services (#1385, #1382)

Ships the KOM-policy half of ADR-050 as an opt-in values overlay, values-npd-gpu-services.yaml, covering NPDFabricManagerDown and NPDFabricManagerFlapping (fatal, RESTART_BM), NPDFabricManagerNotInstalled (fatal, CONTACT_SUPPORT, because there is no unit to restart and a reboot will not install one), and NPDNvidiaPersistencedDown (non-fatal, CONTACT_SUPPORT). Each policy matches a condition published by the ADR-050 reference NPD configurations. The companion documentation PR adds the NPD plugin scripts, reference configs, and a tutorial.

Unified Platform-Connector Handler and On-Change Node Updates (#1845)

Part of ADR-052. The request handler and connector set become one code path, with Connector.ProcessBatch taking a single batch, Set fanning a batch to several connectors and failing on the first error, and BestEffort wrapping connectors whose failure or timeout must not fail the batch. Connectors and pipeline stages are sized and hardened for fleet scale. The wire contract is unchanged — same PlatformConnector gRPC service, same protobuf. Node updates are now written on change rather than unconditionally.

Configurable Node-Drainer Requeue Backoff (#1875, #1847)

The workqueue requeue backoff base was compiled in at 10 seconds, so for fast-terminating or zero-grace pods that floor dominated drain latency — 10.09 s P50 where the eviction work itself took milliseconds. The base is now a CLI flag and Helm value, keeping the 2-minute ceiling and the 10-second default. Operators with fast workloads can lower it while workloads with long grace periods keep the existing behaviour.

Configurable GPU Reset `driverRoot` (#1813)

gpu_reset.sh reaches nvidia-smi only through chroot "$DRIVER_ROOT", so that path must be a complete filesystem — binary, shared libraries, and dynamic loader. The container-side path was a Go constant, with only the host side configurable, which works against a GPU Operator driver container that bind-mounts its full root but fails against a driver root staging only driver surfaces, where the chroot fails before nvidia-smi starts. The new janitor.config.controllers.gpuReset.resetJob.driverRoot value makes the container-side path configurable too.

Metrics Additions

dcgm_health_active_events carries the error code (#1874, #1862): the gauge reported which watch and GPU had an active event but not which code raised it, so on an NVL72 fleet every active GpuNvlinkWatch series looked identical and an effective-BER breach was indistinguishable from a link-down without going to the datastore. The new error_code label uses the same string already published as the event's errorCode. Because this is a gauge, clears now zero the exact label set they set, which matters where one check can be latched by more than one code.
Opt-in rule_matched_entity_total (#1812, #1746): rule_matched_total is labelled only {rule_name, node_name}, so entity-keyed rules fire without saying which GPU, GPC, TPC, or NIC matched. The new series adds entity_type and entity_value, registered only when health-events-analyzer.ruleMatchedEntityMetricEnabled is set. rule_matched_total is unchanged. GPU UUID is never exported; PCI, GPU index, GPC, TPC, NVLINK, NIC, and NICPort are. Node-scoped rules do not emit the series.

Scrape the MongoDB Exporter From Outside the Release Namespace (#1879, #1878)

With usePerconaOperator enabled, the exporter port 9216 shared an ingress rule with the MongoDB port 27017, restricting both to the release namespace — so Prometheus deployed in a separate monitoring namespace timed out scraping with context deadline exceeded. The two ports now have separate rules: 27017 stays restricted to the release namespace so database traffic is never exposed to scrape sources, while 9216 accepts networkPolicy.additionalScrapeNamespaces and additionalScrapeRules. Both default to empty, so the release namespace is always permitted and existing installations are unaffected.

Performance

Fault-quarantine node cache pruned to the keys its rules read (#1842, #1810): v1.20.0 cleared node.status and deliberately kept the rest. That was not enough — fault-quarantine still held 17.88 GB of node cache at 100,005 nodes, or 161,516 bytes per node, because map entries rather than bytes predict the cost: clearing status drops 60% of the bytes while the remainder still costs the same 161,516 bytes held. The cache now keeps only the fields the CEL rules read plus the fixed keys fault-quarantine hardcodes.
Fault-remediation fetches live-stream events lazily (#1811, #1801): v1.19.0 fixed the cold-start path to queue only a document ID, but the live change-stream path still retained the whole decoded document as a map[string]any for as long as the item sat in the workqueue — 7.79 KB per queued event against 0.55 KB on cold start. That only matters under a backlog, which is exactly when it matters most, and component sizing is measured with an empty queue so the backlog is memory nobody accounted for. The live path now queues the ID and fetches on work.
Slinky-drainer caches only what it reads (#1823, #1822): the manager used default cache options, caching every Pod in the cluster and every Node in full, while the reconciler reads only a Pod's spec.nodeName and Slurm drain conditions, and a Node's state label and cordon-reason annotation. The Pod informer is now scoped to the Slinky namespace and both types are transformed down to those fields. Pruning the labels also quietens the controller, which watches Nodes with LabelChangedPredicate and previously woke on every unrelated NFD or GFD label change.
Sticky XID window bounded with a scalar (#1880): the sticky XID check pushed $$ROOT over an unbounded window, materialising every previous event as a full document for every document and then scanning that array — quadratic in both data volume and work. Because the three-hour bound was already inside the map, only the most recent previous sticky XID can ever satisfy it, so a scalar $max of the sticky timestamps answers the identical question. Applied across all four affected rules.

Bug Fixes & Reliability

All DCGM incident codes are preserved (#1804, #1795): every distinct error code for a GPU within a health watch is now kept, with messages grouped by watch, GPU, and code, each code published with its own remediation, and all pending codes retained in the event cache. No protobuf or configuration changes.
NVSwitch incidents are reported (#1825, #1824): NVSwitch incidents were detected but never reported.
Platform-connectors retries Kubernetes writes in order (#1747): failed Kubernetes writes are now retried before the next batch is consumed, reducing health-event loss during transient API failures. Successful writes are not replayed, and a permanent error on one write does not stop independent transient failures from retrying. The node-local queue defaults to 25 retries per write under a whole-batch deadline of one minute, configurable up to five, with API calls and backoff sharing that deadline and cancellation interrupting both. Uncertain Event writes are reconciled by stable fault name and persisted timestamp before retrying, avoiding false drops and duplicate counts.
Readiness probes report datastore connectivity (#1843, #1839): fault-quarantine, node-drainer, and fault-remediation served readiness from an unconditional 200, and because the HTTP server binds before datastore initialization completes (to protect liveness against slow database startup), pods reported Ready=True while completely disconnected from MongoDB or PostgreSQL. Readiness now reflects datastore connectivity.
XID74Reg2Bit13Set is time-bounded (#1855, #1838): it was the only shipped analyzer rule without a time-bounding first stage, so MongoDB scanned the node's entire historical event log and decoded every document into memory. It now opens with the standard 24-hour bound and ends with $limit: 1, and a new test asserts that every rule in values.yaml opens with a stage bounding generatedtimestamp.
Duplicate label in the external-MongoDB setup Job (#1844, #1833): the Job template emitted app.kubernetes.io/name twice because it appended the key after nvsentinel.labels, which already emits it. Helm's parser tolerates a duplicate mapping key so the chart rendered and published fine, but strict YAML parsers rejected the document. The resulting label set is unchanged.

Documentation

VDR review feedback (#1852): incorporates documentation feedback raised during the VDR review.

Acknowledgments

This release includes contributions from:

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @dmvevents, @giuliocalzo, and @HesandaLiyanage.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.24.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.23.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.24.0 \
  --namespace nvsentinel \
  --reuse-values

NVIDIA/NVSentinel v1.24.0 Release v1.24.0 on GitHub