NVIDIA/dcgm-exporter 4.9.0

on GitHub

Tested component versions: DCGM 4.7.0, go-dcgm v1.4712.2, go-nvml v0.13.4-1,
Go 1.27.3, and NVIDIA distroless CC v4.2.0.

DCGM Exporter 4.9.0 is tested with DCGM 4.7.0 and 4.7.1 and ships with DCGM 4.7.0. In DCGM 4.7.1, some requests between libdcgm and the host engine fail with an API version mismatch when one side is 4.7.1 and the other is an earlier version. If you use DCGM 4.7.1, run a 4.7.1 host engine with a 4.7.1 libdcgm.

4.9.0 renames default metrics and labels, changes some units, and changes the configuration and Helm values formats. Read the upgrade checklist in the DCGM Exporter 4.9.0 release notes before upgrading from 4.8.4.

Metrics

• Update to DCGM 4.7.0 and go-dcgm v1.4712.2.
• Reject malformed DCGM responses that used to crash the exporter or produce wrong values.
• Rename the 23 default and profiling-preset fields that DCGM 4.7 deprecates to their new DCGM 4.7 names. Eleven of them also change unit: clocks from MHz to hertz, energy from millijoules to joules, utilization from a percentage to a 0 to 1 ratio, GPU memory from MiB to bytes, and NVLink throughput from MB/s to bytes per second.
• Rename the driver-version label on every metric from DCGM_FI_DRIVER_VERSION to DCGM_FI_SYSTEM_DRIVER_VERSION. Helm installs did not have this label in 4.8.4 and now get it.
• Collect DCGM_FI_DEV_FB_CAPACITY_BYTES (total GPU memory in bytes) by default.
• Deprecate dcp-metrics-included.csv and 1.x-compatibility-metrics.csv. The exporter logs a warning at startup when either file is used. Use default-counters.csv instead.
• Remove commented-out rows that use deprecated or old field names from the shipped CSV files. The remaining commented-out rows list optional fields under their DCGM 4.7 names.
• Add optional fields in hertz, bytes, seconds, and 0 to 1 ratios for measurements that the default metric list does not include.
• For Kubernetes pods, add up per-process GPU utilization and GPU memory use under the new DCGM_FI_DEV_GPU_UTIL_RATIO and DCGM_FI_DEV_FB_USED_BYTES names as well as the old names. A pod's utilization ratio can be above 1.0.
• Update the bundled Grafana dashboard to the new metric names and units.
• Add optional DCGM_EXP_FIELD_POLICY_* gauges that report each DCGM policy's state, thresholds, current value, violation count, and last trigger time.
• Report XID count and total metrics for each selected MIG GPU instance, its compute instances, and its parent GPU (#751 by @sylvesterkaczmarek; fixes #750, reported by @erikkressner).
• When the exporter monitors MIG instances (the default device selection), report GPU-wide fields such as power, temperature, clocks, and PCIe once per physical GPU instead of repeating them for each MIG instance. These metrics have no MIG instance labels and carry pod labels only when one pod owns the whole GPU, so adding them up across instances no longer counts a GPU more than once. The clock-event count and total metrics work the same way. Addresses #561, reported by @taejune, and #353, reported by @frittentheke.
• Set NVIDIA_MIG_MONITOR_DEVICES=all in container images so DCGM can collect profiling fields on MIG-enabled GPUs and their instances. This fills in the default PCIe TX/RX and DRAM utilization metrics, which the 4.8.4 image left blank on MIG-enabled GPUs.
• Reject raw fields that cannot be exported safely, such as bitmasks, supported-clock lists, and status fields that can read as a misleading zero after a DCGM error. Reject help text that is not valid UTF-8.
• Keep the physical link numbers on GPU NVLink metrics, including gaps where links are inactive (fixes #753, reported by @v0ropaev).
• Apply --replace-blanks-in-model-name to GPU NVLink metrics as well as GPU metrics.
• Report IMEX domain recovery incidents with health_error_code="DCGM_FR_GPU_RECOVERY_IMEX_DOMAIN".
• Add optional DCGM_EXP_SYSTEM_HEALTH_STATUS for host-level IMEX health. It needs DCGM 4.7.1 with IMEX health-watch support, so it does not work with the bundled DCGM 4.7.0.
• Document that cumulative XID and clock-event totals restart from zero when the collectors are rebuilt, for example after a CSV or SIGHUP reload, a DRA pool change, or GPU recovery.
• Report a clock-event count of zero when no clock-event reasons are active.

Enrichment

• Add true, false, and auto settings for Kubernetes, container, DRA, and HPC labels.
• Keep every pod label value when two label names become the same after conversion to Prometheus names, by adding stable _conflictN suffixes (#755 by @git-jxj; fixes #754).
• Require a non-empty allowlist when pod labels are turned on; in 4.8.4 an empty allowlist included all labels. Use .* to include all labels.
• Keep metrics for unallocated GPUs when DRA and virtual-GPU mapping are both on.
• Keep GPU pod labels off CPU, NVSwitch, and NVSwitch-link metrics, including with MIG, virtual GPUs, and DRA (#762 by @v0ropaev).
• Trim spaces from HPC job IDs, and ignore blank lines and the 0 that marks a finished job. Idle GPUs no longer get hpc_job="0".
• Read container labels from /var/run/docker.sock by default, and detect Docker-compatible sockets that do not work.
• Keep serving metrics when an HPC job mapping file or directory cannot be read, and add job labels once a missing directory appears.
• Reject symbolic links and hard links in HPC job mappings, read only the mapping files for exported GPUs, and add limits on file size, job ID size, and jobs per file (--hpc-job-mapping-max-file-bytes, --hpc-job-mapping-max-job-id-bytes, --hpc-job-mapping-max-jobs-per-file).
• Retry DRA setup every minute after a failure, warn when the kubelet socket is missing, and log each repeated label warning at most once every 5 minutes.
• Cache MIG device lookups, including failed ones, between scrapes (addresses #599, reported by @jaeeyoungkim).
• Drop stale process-to-pod mappings when a process ID is reused.
• Rename a pod label called hpc_job to pod_label_hpc_job when HPC job labels are on, so it does not clash with the job label.

Configuration

• Replace the preview version 2 YAML format from 4.8.4 with the final version 2 format: keys are now snake_case, metric lists go under metrics[], collection settings can be set per field, and overrides pick fields with match. Rewrite preview version 2 files (camelCase keys and a collections block); they fail at startup. Invalid values are rejected at startup.
• Deprecate version 1 YAML configuration files. The exporter logs a warning when it loads one; support ends in 4.10. Version 1 files that set sources now fail at startup; use version 2 sources.dcgm.detect_bind_unbind instead.
• Add configuration-file keys for server, collection, logging, device, and workload-label settings that were only available as flags.
• Match DCGM field names in watch groups and overrides by field ID, so a field's new name and its old aliases select the same field.
• Reload the metric CSV file when a ConfigMap mounted as a volume is updated, including when the file is replaced in one step.
• Include the ConfigMap namespace, name, and key in errors about malformed CSV from --configmap-data (#761 by @spicysharkco).
• Require metric files to be regular files no larger than 8 MiB.

Command-line interface

• Group flags by topic in --help, and add offline config show, config validate, and config explain commands to print, check, and explain the settings the exporter would use.
• Deprecate --configmap-data (-m). It still works, and the exporter logs a warning when it or its environment variable is set.
• Warn when an XID or clock-event count window is longer than the history DCGM keeps.
• Reject badly formed durations, HTTP timeouts of zero or less, and invalid watch-retention settings at startup.
• Require TLS or basic authentication in the web policy before Go runtime profiling can be turned on.
• Check collection intervals, listen addresses, pod-label selection, and conflicting settings before startup.
• Reject the embedded DCGM log (--enable-dcgm-log) when a remote host engine is set.
• Name the flag, environment variable, or file key in startup errors.
• Report all configuration problems found at startup, not just the first.

Runtime and recovery

• Start and reload faster when many NVLink or CPU-core metrics are selected.
• Report profiling, NVSwitch, and CPU fields once DCGM has sampled them; the first scrape after startup or a reload may not include them.
• Add a separate five-minute timeout for cleaning up DCGM watches during reload, shutdown, and GPU bind/unbind. Change it with sources.dcgm.cleanup_timeout or --dcgm-cleanup-timeout.
• Exit with an error after 6 failed GPU recovery retries, waiting 1 to 30 seconds between them; 4.8.4 retried forever. Stop waiting forever while the GPU watcher starts or stops.
• Skip finding CPUs, NVSwitches, and other non-GPU devices when none of their selected fields are supported.
• Keep event metrics working after reloads, and keep the current metrics when finding devices again fails.
• Start when some GPUs are detached, and keep physical GPU numbers in P2P metrics.
• Use DCGM's MIG profile names when the driver lacks the optional NVML function for looking them up (#758 by @WeAreResilience; fixes #757, reported by @kkohtaka).
• Find NVSwitches through NVLink status when DCGM rejects the entity-list API version.
• When an older DCGM hostengine refuses fields, name those fields in the error and say how to fix it.
• Warn when the exporter has to identify a GPU using older DCGM fields or a placeholder UUID.
• Explain how to check the driver libraries and the Kubernetes RuntimeClass when NVML cannot be found.
• Fix hangs when forwarding large DCGM log messages (fixes NVIDIA/DCGM#315, reported by @evanjpw).
• Exit with a nonzero status after fatal DCGM scrape errors, failed GPU bind/unbind recovery, and HTTP listener failures.
• Add Content-Security-Policy and X-Frame-Options headers to the landing and health pages, keeping any headers set in the web policy.
• Create debug dump directories and files that only their owner can read.
• Have dump cleanup delete only dump files the exporter named and unfinished .tmp files.
• Limit how often dumps are written, how large each file is, their total size, and how many there are (--dump-min-interval, --dump-max-file-bytes, --dump-max-total-bytes, --dump-max-files).

Helm, containers, and packages

• Ship /etc/dcgm-exporter/config.yaml in images and packages and load it by default. It holds the same settings as the 4.8.4 command-line defaults.
• Use the Helm config value as the exporter's configuration, and add a configMap block (existingName, key) for using your own ConfigMap instead. Exporter settings move out of service, kubernetes, kubernetesDRA, and debugDump into config.
• Mount the chart's configuration, metric list, and web-policy files in separate directories under /etc/dcgm-exporter/chart/. Running pods pick up metric-list changes without a restart. A change to the chart's config restarts the pods automatically; if you use your own ConfigMap (configMap.existingName), restart the pods yourself after editing it.
• Give each Helm release its own metrics ConfigMap, and remove the Role and RoleBinding that let the exporter read ConfigMaps.
• Use default-counters.csv as the chart's default metric list, so Helm installs also collect DCGM_FI_DEV_FB_RESERVED_BYTES and get the DCGM_FI_SYSTEM_DRIVER_VERSION label.
• Set Kubernetes labels to auto in the chart instead of always on.
• Deprecate Helm customMetrics; use config.metrics[].fields instead. When the chart manages config, reject custom metric lists that would be ignored, and flags or environment variables that override the metric file.
• Set the Pod and ResourceSlice read permissions and service-account token mounting from the pod-label and DRA settings in config.
• Require the configured listen port to match service.port, and reject overrides that would not match the chart's resources.
• Mount TLS and basic-auth web policies from a Secret, and set the web-policy path in the generated configuration.
• Store Helm debug dumps in a pod emptyDir volume capped by debugDump.sizeLimit (default 512Mi) instead of a directory on the host. Dumps are lost when the pod is replaced.
• Add a Helm startupProbe, on by default, that gives the exporter up to five minutes to start without delaying the readiness and liveness probes (#749 by @moleculargastronomy).
• Request 100Mi of ephemeral storage by default.
• Add an optional NetworkPolicy for incoming traffic to the metrics port, in the chart and as a raw YAML example.
• Reject invalid Helm rolling-update settings when the chart is rendered.
• Keep Helm arguments as strings, so values such as 1000 are not turned into numbers (#746 by @git-jxj).
• Remove app.kubernetes.io/version from the selectors in the raw YAML DaemonSet, Service, and ServiceMonitor.
• Have the systemd unit set DCGM_EXPORTER_CONFIG_FILE instead of passing -f.
• Use the distroless image for all release tags: 4.9.0 (the Helm default), 4.7.0-4.9.0, the older-style alias 4.7.0-4.9.0-distroless, and latest.
• Stop publishing the Ubuntu-based image. Use the distroless image.
• Pin all DCGM packages to the same build, and use DCGM 4.7.0 package revision 2 in container images.
• Tighten file ownership and permissions in the container; update Go to 1.27.3 and refresh dependencies and base images.
• Build release artifacts with Bazel, and use Task and Mise for build and development commands. The Makefile is removed.

OpenTelemetry (OTLP)

DCGM Exporter 4.9.0 includes the first code for OpenTelemetry Protocol (OTLP) export, but nothing turns it on yet. A future release will add initial OTLP support. OpenTelemetry Support in DCGM Exporter gives an overview of the feature and will be updated as it develops.