Release v1.31.0

This release adds Akamai Cloud (LKE) support to janitor-provider and csp-health-monitor, adds IPv6 support, and lets janitor stop GPU clients outside the GPU Operator and patch the GPU reset Job. A new default fault-quarantine rule set cordons nodes for fatal health-events-analyzer events. It also fixes several PostgreSQL change-stream problems, stops a GPU reset that a retry recovers from rebooting the node, and keeps fault-quarantine running on a cluster with no GPU nodes.

Upgrade Notes

Fault-quarantine rule set for health-events-analyzer events (#2335, #2297)

health-events-analyzer publishes fatal derived events, such as MultipleRemediations and RepeatedXID74Reg0HardwareIssue. No default fault-quarantine rule set matched them, so these events set node conditions but did not cordon, drain or remediate the node. The new default rule set Health events analyzer fatal error ruleset cordons the node for a fatal health-events-analyzer event. This is a change to the default of fault-quarantine.ruleSets. helm upgrade --reuse-values keeps your previous rule sets. If you run health-events-analyzer and want these events to quarantine nodes, upgrade with --reset-then-reuse-values (Helm 3.14 or later). It keeps your own values and applies the new chart defaults. If you set your own ruleSets, copy the rule set from the v1.31.0 chart.

XID 74 rules in health-events-analyzer (#2271, #2270)

The four XID 74 solo rules took their registers and their GPU from the node's stored events, not from the incoming XID 74 event. After one matching XID 74 on a GPU, every later XID 74 on the node fired the rule, and the CONTACT_SUPPORT event could name a GPU whose registers did not match, or a GPU that was not solo. The rules now read the registers and the GPU from the incoming event. The fix is in the default value of health-events-analyzer.config. helm upgrade --reuse-values keeps your previous value. If you run health-events-analyzer, upgrade with --reset-then-reuse-values. If you set your own config, copy the four XID 74 solo rules from the v1.31.0 chart.

Major New Features

Akamai Cloud (LKE) Support (#2114, #2115)

NVSentinel now supports Akamai Cloud (Linode Kubernetes Engine, LKE):

• janitor-provider (#2114): with janitor-provider.csp.provider: akamai, janitor reboots and replaces nodes through the Akamai Cloud API. A reboot is complete when the instance is running and the kubelet reports a new boot ID. A terminate recycles an LKE node, so LKE provisions a replacement and the node pool keeps its size. An instance outside LKE is deleted. The provider acts only on a node with a linode://<id> provider ID. Set the API token Secret in janitor-provider.csp.akamai.apiTokenSecretRef. The token needs the linodes:read_write and lke:read_write scopes.
• csp-health-monitor (#2115): with csp-health-monitor.cspName: akamai, the monitor polls the Akamai account maintenance API and quarantines and drains a node before its maintenance starts. Every maintenance type quarantines the node, live_migration included, because a GPU workload cannot live-migrate. The poll interval is configToml.akamai.pollingIntervalSeconds, default 60, minimum 30. Set the token Secret in csp-health-monitor.akamaiApiTokenSecret. The token needs the account:read_only scope.

These scopes cover the whole Akamai account, so use a dedicated token for NVSentinel.

IPv6 Support (#2252)

Set global.ipv6.enabled: true to run NVSentinel on an IPv6-only or dual-stack cluster. The default is false. With the flag on, gpu-health-monitor binds its metrics server to ::, Bitnami MongoDB sets MONGODB_ENABLE_IPV6, Percona MongoDB adds net.ipv6: true to the default replica set configuration, and the in-cluster file server also listens on [::]:8080. Every node must boot with IPv6 on. Also set the GPU Operator's DCGM host engine to bind :: with dcgm.args in the GPU Operator chart, which needs GPU Operator v26.7.1 or later. If DCGM binds only 0.0.0.0, the labeler sets no DCGM version label, and gpu-health-monitor schedules no pods.

With helm upgrade --reuse-values, the Percona replica set configuration keeps its previous value. To turn on IPv6 with the Percona backend, upgrade with --reset-then-reuse-values. If you set your own mongodb-store.psmdb-db.replsets.rs0.configuration, add net.ipv6: true to it.

GPU Reset Stops GPU Clients Outside the GPU Operator (#2276, #1691)

A GPU client that runs outside the GPU Operator namespace, for example a monitoring agent, keeps the GPUs open, and nvidia-smi --gpu-reset fails with In use by another client. GPU Operator 26.7 and later sets the nvidia.com/gpu.deploy.client=true node label, and GPU clients select it. Before a GPU reset, janitor now sets this label to false when at least one GPU client pod runs on the node. It waits until the DaemonSet controller removes the GPU client pods, and sets the label back to true after the reset. NVSentinel's metadata-collector selects this label too, so it also leaves the node during the reset. The janitor ClusterRole gains get on DaemonSets.

GPU Reset Job Patch (#2324, #1782)

The new janitor.config.controllers.gpuReset.resetJob.jobTemplatePatch value is a strategic merge patch that janitor applies on top of the built-in GPU reset Job, so you can change the Job without forking janitor. janitor always sets the Job name, namespace, owner reference and node name, and the NVIDIA_GPU_RESETS variable of each container, itself. janitor refuses at startup a patch with a misspelled field, a field in the wrong case or a value of the wrong type. Init containers now also get NVIDIA_GPU_RESETS. The patch is empty by default. The janitor configuration docs give the merge rules and an example.

Mount Only the Journal Directory (#2292, #2291)

The new syslog-health-monitor.journalDirectoryHostPath value mounts only the host journal directory in regular mode, in place of its parent in journalHostPath. Use it where a vcluster syncer rewrites a /var/log host path to a per-tenant directory, which hides the journal. The directory must exist on every node. The value is empty by default.

Metrics

• metadata_collector_collection_completed_timestamp_seconds (#2157): a new gauge. It is 0 until GPU metadata collection completes, and then holds the completion time. To find a collector that is stuck, alert on metadata_collector_collection_completed_timestamp_seconds == 0 and time() - process_start_time_seconds > 300.
• fault_quarantine_get_total_nodes_retry_attempts is removed (#2273, #2272): fault-quarantine no longer retries a GPU node count of 0. Remove dashboards and alerts that use this metric.
• External remediation metrics (#2161): nvsentinel_external_remediation_total{phase="closed"} gains the result value node_deleted, for a request whose Node is missing, terminating or replaced. nvsentinel_external_remediation_open is now set to 0 or 1, and janitor restores it after a restart.

Bug Fixes & Reliability

• A GPU reset that a retry recovers does not reboot the node (#2254, #2279): the reset Job runs gpu_reset.sh again after a failed run, and every run wrote its result to syslog. When the first run failed and the retry reset the GPU, syslog-health-monitor had already requested a reboot for the first failure. The reset Job now reports a failed reset only from its last run. When the last run fails, the node still reboots.
• Node patches retry after a connection error (#2267, #2266): a node patch whose connection failed before the response, for example when the API server restarted, was not retried. In fault-quarantine, this was the cordon of a fatal event, so the event was checkpointed and the node stayed schedulable until fault-quarantine restarted. The node patches of fault-quarantine, the labeler and lifecycle-manager now retry connection resets, unexpected EOFs, lost HTTP/2 connections and refused connections, with the existing backoff of 10 attempts.
• Fault-quarantine runs on a cluster with no GPU nodes (#2273, #2272): the circuit breaker treated a GPU node count of 0 as a cache sync failure, retried, and then made fault-quarantine exit, so fault-quarantine crash-looped. With no GPU nodes, the breaker now reports CLOSED. If a cordon is recorded in the window while the breaker counts 0 GPU nodes, the breaker trips, because fault-quarantine.gpuNodeLabel then does not match the nodes that fault-quarantine cordons.
• An escalated recommended action stays in the quarantine annotation (#2163, #2162): when a later event escalated the RecommendedAction of a stored event, fault-quarantine overwrote some keys of the older compound event without compacting the stored events. The older event still listed those entities, so a JSON round trip of the annotation could restore the older action. Fault-quarantine now compacts the stored events whenever it overwrites an entry.
• PostgreSQL change stream (#2231, #2153, #2250, #2249, #2265, #2264): with the PostgreSQL datastore:
  • The event filter did not support $exists. Every event that was not EXECUTE_REMEDIATION logged Unsupported operator, and fault-quarantine and health-events-analyzer dropped events with no processing strategy, which MongoDB delivers. The filter now supports $exists.
  • When the changed_at lookup of an acknowledged event failed, the watcher saved the current time as its resume position, so a restart skipped events. The watcher now returns an error and keeps its position.
  • The cursor moved only to a row with a higher id. A row with a higher id but an earlier changed_at made every later poll return the same rows, and with 100 or more such rows the consumer stopped. The cursor now moves forward in (changed_at, id) order.
• Datastore clients wait for MongoDB to start (#2245, #2140): the chart set MONGODB_PING_TIMEOUT_TOTAL_SECONDS to 30, so platform-connectors, fault-quarantine, node-drainer and fault-remediation gave up after about 35 seconds while MongoDB started, and restarted. The chart no longer sets it, so these components use the store-client default of 300 seconds. The same setting also sets the MongoDB server selection timeout, the resume token store retry window and the change stream recovery pass, which also change from 30 to 300 seconds.
• External remediation requests stay open until the repair completes (#2125, #2072, #2161): janitor started the ExternalRemediationRequest TTL at the first reconcile. With the default TTL of 336h, a repair that took longer than 14 days lost its release taint and its nvsentinel.dgxc.nvidia.com/managed=false label, and NVSentinel took the node back during the repair. The TTL now starts when the request completes, so the TTL never deletes an open request. janitor also checks the Node UID before it applies or cleans up a request, matches the release taint by key, effect and request name, and keeps taints that operators or other requests own. The new CompletedRequestDeleted and OwnerNodeDeleted events report when a completed request is deleted and when a Node deletion closes an open request.
• Generic reboot Jobs end with their attempt (#2178, #2177, #2300): janitor-provider deleted its reboot Job only when the node came back with a new boot ID. When janitor gave up on a RebootNode, the Job stayed, its pod could reboot the node later, and a leftover pod in ImagePullBackOff failed each later RebootNode for that node. janitor-provider now deletes the old reboot Jobs before it starts an attempt and when an attempt fails, and reads only the pods of the current Job. When the kubelet rejects the reboot pod at admission, for example as Evicted under disk pressure, the attempt now fails at the next readiness check instead of at the reboot timeout.
• XID detection continues past invalid UTF-8 (#2269, #2268): the name= field of a kernel XID line can hold bytes that are not valid UTF-8. The health event built from such a line could not be encoded, so the publish failed, and the XID check stopped at that line on every poll, restarts included. syslog-health-monitor now replaces such bytes with U+FFFD.
• The post-reboot scan reads the current boot (#2298, #2277): the scan that reads the journal entries written before syslog-health-monitor started compared the journal _BOOT_ID, which has no dashes, with the boot ID in /proc, which has dashes. Nothing matched, so the scan skipped every entry of the current boot, and XIDs logged before the monitor started were missed. The comparison now ignores the field name and the dashes.
• XID 74 derived events name the reporting GPU (#2271, #2270): see the Upgrade Notes.
• metadata-collector memory (#2251, #2176): the pod device mapper decoded the whole kubelet /pods response every 30 seconds. On nodes with large finished pods, metadata-collector exceeded its memory limit and was OOM-killed in a loop. It now decodes one pod at a time, so its memory no longer grows with the size of the pod specs.
• metadata-collector serves metrics during collection (#2157): metadata-collector started its metrics server only after GPU metadata collection finished, so a collector blocked on a hung driver served no metrics. The metrics server now starts first. See Metrics for the new completion gauge.
• metadata-collector permission for fine-grained kubelet authorization (#2328, #2323): the metadata-collector ClusterRole had no get on nodes/pods, so under fine-grained kubelet authorization, each /pods poll made a denied access review before the kubelet fell back to nodes/proxy. The ClusterRole now grants get on nodes/pods.
• nccl-loopback preflight check (#2259, #2258, #2261, #2260): the check reported a fatal NCCL_TEST_ERROR against healthy nodes in two cases. On RTX Pro 6000 (sm_120), the image had no sm_120 kernels, and it now builds them. When the workload set NCCL_DEBUG=INFO, NCCL log lines broke the result table, and the check now sends the NCCL log to stderr.

Security & Infrastructure

• CVE fixes (#2337): the gpu-reset image installs the latest Ubuntu security fixes for curl and its libraries, and the log-collector image moves to kubectl v1.35.10.

Documentation

• Metrics reference (#2326, #2280): docs/METRICS.md now lists 92 metrics that were missing across 16 components, and corrects 12 entries, including the _total suffix of 11 gpu-health-monitor counters and the drain_scope label of node_drainer_events_processed_total.
• datastore_connected (#2243): the metrics reference and the HELP text now say that the gauge becomes 1 when the datastore watcher starts, and stays 1 for the life of the process. To detect a datastore problem, alert on pod readiness (kube_pod_status_ready) and restarts.
• GangScheduling gate on Kubernetes 1.37 (#2319): Kubernetes 1.37 removed the GangScheduling feature gate, and kube-scheduler does not start with an unknown gate. The README and values-preflight-kube.yaml now ask for the gate only on Kubernetes 1.35 and 1.36.

Chores

• E2E conflict retries (#2202, #2138, #2310, #2307): E2E test helpers failed with HTTP 409 when NVSentinel components updated the same Node. The new helpers.ConflictBackoff tries 10 times in about 7.5 seconds, and replaces retry.DefaultRetry in all 30 places in tests/. The cordon-patch connection-reset test now accepts every retryable error, including an unexpected EOF over HTTP/2.
• Network retries in CI (#2242, #2246, #2142): one failed network call failed a CI job. CI now runs apt-get, go install, pip install and the other network downloads that failed jobs through scripts/retry.sh, up to 3 times, and the Dockerfiles retry go mod download.
• Image builds on native runners (#2285, #2283): Container Build Validation built the arm64 images in QEMU emulation, and the preflight-nccl-loopback job reached its 45-minute limit. Each architecture now builds on its own runner, and the two builds run in parallel.
• Contributor guidance (#2253, #2120): AGENTS.md adds the rules that the CI reliability work showed, for example how to retry conflicts in E2E tests, how to show that a flake is fixed, and how to list a new E2E test in tests/e2e-shards.yaml.

Acknowledgments

This release includes contributions from:

• @akhil9tiet
• @blacksoxx
• @cbaenziger
• @cbumb
• @dcandido-dot
• @deesharma24
• @eexwhyzee
• @glassharbor
• @jianzhangbjz
• @JSJANG552
• @jtschelling
• @karanaggarwal1
• @kdotmanoj
• @lalitadithya
• @nsd9696
• @patelpayal
• @pcbeingused333
• @Saibernard
• @sumedh-aerram
• @TheVicky1
• @XRFXLP
• @yuminn-k
• @ZeXi0308

Thank you to everyone who contributed code, testing, documentation, design reviews, and feedback! Special thanks to first-time contributors @cbaenziger, @dcandido-dot, @eexwhyzee, @glassharbor, @JSJANG552, @karanaggarwal1, @nsd9696, @patelpayal, @pcbeingused333, @sumedh-aerram, @TheVicky1, and @yuminn-k.

Container Images

See versions.txt for the full list of container images and versions.

Helm Chart

Install with:

helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.31.0 \
  --namespace nvsentinel \
  --create-namespace

To upgrade from v1.30.0 — review the Upgrade Notes above first:

helm upgrade nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version v1.31.0 \
  --namespace nvsentinel \
  --reuse-values