Release v1.28.0

NVSentinel is now graduated.

This release makes metadata-collector and embedded-mode gpu-health-monitor leave a node during GPU Operator driver upgrades and MIG changes, lets MongoDB change-stream consumers recover from transient failures, and adds Kubernetes 1.37 support to preflight gang discovery. It also detects XIDs on MIG GPUs and fixes remediation on Amazon DocumentDB. Several behaviours change on upgrade — please read the Upgrade Notes below.

Upgrade Notes

metadata-collector and embedded gpu-health-monitor follow `nvidia.com/gpu.deploy.client` (#1988, #1868)

metadata-collector, and gpu-health-monitor in embedded-mode, now have a default required node affinity on the nvidia.com/gpu.deploy.client node label. A pod runs when the label is true or absent. The DaemonSet controller removes the pod for any other value, such as paused-for-driver-upgrade, paused-for-mig-change, or false. While the pod is off the node, embedded-mode gpu-health-monitor does not report GPU health for that node. If a driver upgrade or MIG change fails and leaves the label at a paused value, the pods stay off the node until the label is true again.

The default rule applies only when neither global.affinity nor the component's own affinity is set. global.affinity applies to every NVSentinel component, so if you set it for any reason, these DaemonSets do not get the rule. To keep the rule together with a custom affinity, add its two terms to your affinity, as shown in the metadata-collector and gpu-health-monitor configuration docs.

Remote DCGM probe hangs are store-only by default (#1984, #1978)

In the operator-service and external-hostengine modes, gpu-health-monitor reports a hung DCGM probe as a GpuDcgmConnectivityFailure event with error code DCGM_PROBE_HANG. This event used the global processing strategy (EXECUTE_REMEDIATION by default), so the default rules cordoned the node. It now follows gpu-health-monitor.dcgm.probeStoreOnly, which defaults to true, so a remote probe hang is stored and NVSentinel does not act on it. Other DCGM connectivity errors still use the global strategy. To restore remediation for remote probe hangs, set probeStoreOnly: false. That setting also enables remediation for the embedded-mode GpuDcgmUnresponsive event, which recommends RESTART_BM.

MongoDB metrics in a namespace other than `nvsentinel` (#2053, #2052)

The bundled Bitnami MongoDB metrics exporter connected to mongodb-headless.nvsentinel.svc in every namespace, so MongoDB metrics were missing when NVSentinel ran in any other namespace. The fix changes the default value of mongodb-store.mongodb.metrics.args. helm upgrade --reuse-values keeps your previous value, so the exporter still connects to the wrong namespace. If you run the bundled MongoDB in a namespace other than nvsentinel, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults.

Major New Features

GPU Operator Driver Upgrades and MIG Changes (#1988, #1868)

GPU Operator 26.7 and later uses the nvidia.com/gpu.deploy.client node label to move GPU clients off a node before it unloads the driver or changes the MIG layout. metadata-collector and embedded-mode gpu-health-monitor keep the GPUs open, but they ignored this label, so during a driver upgrade they could keep the old driver in use. Both DaemonSets now leave the node while the label holds any value other than true, and return when it is true again. Older GPU Operator releases and GPUCluster mode do not set the label, so the pods run there as before. The operator-service and external-hostengine modes of gpu-health-monitor do not open the GPUs, so they do not get the rule. See the Upgrade Notes for how the rule interacts with a custom affinity.

MongoDB Change-Stream Recovery (#2040, #2030)

A transient MongoDB failure ended change-stream processing for good: the watcher returned instead of reopening the stream. The watcher now reopens the stream from its stored resume token, and keeps trying for up to 5 minutes. The budget is wall-clock time, sized for a replica-set roll, which takes about two minutes on a 3-member Percona cluster. When the budget runs out, the stream stops and the consumer exits, so a persistent failure shows as a pod restart instead of a Ready pod that processes nothing. This applies to health-events-analyzer, fault-quarantine, node-drainer, and fault-remediation. The new change_stream_reopens_total{client,result} counter reports success, failure, and exhausted; alert on exhausted to see why a consumer restarted.

Preflight Gang Discovery on Kubernetes 1.35 and 1.37 (#2048, #2047)

Preflight's native gang discovery, the default when gangDiscovery is {}, now works on Kubernetes 1.35 and 1.37:

Kubernetes 1.37: the discoverer used the scheduling.k8s.io/v1alpha2 PodGroup, but Kubernetes 1.37 serves PodGroup only as v1beta1. Preflight failed startup validation and crash-looped, and with failurePolicy: Fail, pod creation failed in every preflight-enabled namespace. Preflight now uses the PodGroup version that the cluster prefers.
Kubernetes 1.35: preflight never detected a gang. The webhook dropped the pod's spec.workloadRef when it decoded the pod, and it could not read the field back from the API server, because the pod is not stored until admission completes. The webhook now reads the gang from the admitted pod. When spec.workloadRef.podGroupReplicaKey is set, the gang ID includes it, because Kubernetes applies the gang policy to each replica separately.

XID 154 Recovery Clears XID Conditions (#1997, #1992)

When XID 154 reports that the GPU recovery action has returned to None, syslog-health-monitor now publishes a healthy event for the same GPU entities. This clears the earlier XID conditions on that GPU. Conditions for other GPUs on the node stay in place. The documentation now lists the explicit signals that clear an XID condition, and states that a quiet period does not clear one.

Remote DCGM Probe Hangs Follow `runtimeDebounce` (#1987, #1978)

A hung remote DCGM probe was published at the probe deadline, whatever runtimeDebounce.failureThreshold said, because a hung call has no later cycle to count and liveness restarts the container first. gpu-health-monitor now publishes a remote probe hang only after DCGM has been unreachable for failureThreshold * pollIntervalSeconds. It saves the start of this window in its state file, so the window continues across liveness restarts. A successful cycle ends the window. This takes effect only when runtimeDebounce.failureThreshold is above its default of 1.

NIC Port-State Hold-Down (#1996, #1993)

A port that dropped for a few seconds was reported fatal at once, so a short fabric blip could cordon a node. The new portStateHoldDown value sets how long a port must stay unhealthy before InfiniBandStateCheck or EthernetStateCheck reports it. A port that recovers before the hold-down ends publishes nothing and increments nic_health_monitor_port_state_blips_total{node,check,device,port}. Recovery, newly seen ports, and the escalation of a port already reported unhealthy are not held. The hold-down starts again after a restart. It is off by default ("0s").

Configurable Driver Binary Path for GPU Reset (#2013, #1681)

The GPU reset Job runs every nvidia-smi call as chroot <driverRoot> nvidia-smi, so nvidia-smi must be on the default PATH inside the chroot. The GKE driver installer puts the driver in /home/kubernetes/bin/nvidia, so the reset failed with exit code 127 before it reset anything. The new janitor.config.controllers.gpuReset.resetJob.driverBinPath value names a directory inside driverRoot that holds nvidia-smi and nvidia-bug-report.sh. The reset script adds it to PATH for the chroot command only. janitor refuses at startup a value that is not an absolute, clean path without :. For the GKE driver installer:

janitor:
  config:
    controllers:
      gpuReset:
        resetJob:
          hostDriverRootPath: "/"
          driverBinPath: "/home/kubernetes/bin/nvidia/bin"

Bug Fixes & Reliability

Fault-remediation sees drained nodes on Amazon DocumentDB (#2027): on DocumentDB, fault-remediation never saw that a node was drained, so it created no remediation CR and the node stayed cordoned, with nothing in the logs. The change-stream filter read the update with $objectToArray, which DocumentDB evaluates to null in a change-stream $match. DocumentDB also reports the drain status under a different key than MongoDB. The filter now reads the status with $getField under both keys.
XIDs on MIG GPUs are detected (#2060, #2059): on MIG GPUs the driver logs XIDs as NVRM: Xid (PCI:0000:01:00 GPU-I:05): 94, .... The XID pattern required ) directly after the PCI address, so syslog-health-monitor dropped these XIDs, including XID 95, which maps to COMPONENT_RESET. The pattern now accepts the GPU-I: and GPU-CI: fields. The event uses the same PCI address as a non-MIG XID on that GPU.
Later events on a quarantined node apply their taints and cordon (#2021, #1439): an event on a node that was already quarantined applied only the labels of its matching rule-sets, not their taints or cordon. For example, a node tainted by a taint-only rule-set was not cordoned by a later fatal GPU event. Fault-quarantine now applies the taints and cordon too, and records any taint or cordon that was on the node before, so the release at the end of the quarantine session leaves it in place. This implements ADR-062.
Fault-quarantine clears faults after a GPU replacement (#2016): a replacement GPU keeps its index and PCI address but has a new GPU_UUID. The healthy event for the new GPU cleared the stored GPU and PCI entries but not the GPU_UUID entries of the old GPU, so the node stayed quarantined. A healthy GPU event now also clears the old GPU_UUID entries of a stored fault when it covers every GPU and PCI entity of that fault. v1.20.0 made the same correction for node conditions. Recovery during the cold-start replay still matches exact entity values.
Janitor GPU reset does not override labels during a driver upgrade (#2000, #1869): during a driver upgrade, GPU Operator sets the operand deploy labels to paused-for-driver-upgrade. A GPU reset on that node overwrote them with false and later true, which redeployed DCGM, the DCGM exporter, the device plugin, and GFD while the driver was still reloading. Janitor now changes a managed label only while the label holds janitor's own enabled or disabled value. For any other value, it waits and retries every 3 seconds. If the label stays held past the GPUReset timeout, the reset fails with ServiceTeardownTimeoutExceeded.
Fault-remediation skips the log collector at the attempt cap (#2046, #2045): when an equivalence group had reached maxRemediationAttempts, fault-remediation ran a full log collection for every new event before it found the group capped. With enableGpuOperatorMustGather, each event also started a cluster-wide must-gather. On one bare-metal cluster, a NIC fault that survived three reboots produced more than 90 log-collector Jobs in an hour. Fault-remediation now gives up before it starts the log collector. See the Upgrade Notes for the effect on attemptCount.
Health-events-analyzer exits when event processing stops (#2024, #2020): when its change stream closed on an error, the analyzer stayed Ready but processed nothing. It now exits with a non-zero status, and Kubernetes restarts it from the stored resume token, as fault-quarantine and node-drainer already do.
Dry run skips events that match no rule-set (#2042, #2041): in dry run, fault-quarantine treated every unhealthy event as a quarantine, whether or not a rule-set matched it. It now skips an unmatched event, as a live run does.
The circuit breaker reports its state after a restart (#2025, #2023): a breaker restored as TRIPPED from its ConfigMap did not set fault_quarantine_breaker_state, so an alert on state="TRIPPED" cleared at the first restart while the trip still held. The breaker now reports the state it restored. The gauge exports only the current state, so when the breaker is CLOSED the TRIPPED series is absent rather than 0.
MongoDB metrics exporter connects in the release namespace (#2053, #2052): see the Upgrade Notes.

Documentation

Adopters list (#2029): adds ADOPTERS.md, which lists organizations and projects that use or build on NVSentinel, and links it from the README.

Acknowledgments

This release includes contributions from:

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @atif1996, @jianzhangbjz, @michal-ii, @msnow-nv, and @ranrubin.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.28.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.27.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.28.0 \
  --namespace nvsentinel \
  --reuse-values

NVIDIA/NVSentinel v1.28.0 Release v1.28.0 on GitHub