Agent

Prelude

Release on: 2025-07-10

• Please refer to the 7.68.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Bump the Python version to 3.12.11
• Change how attribute precedence is handled. All fields are now evaluated across both span and resource attributes, using the following order of precedence (from highest to lowest):
  • datadog.* span attributes
  • datadog.* resource attributes
  • standard span attributes
  • standard resource attributes

New Features

• Add a port of the Windows integrations-core Python network check to Go. This version is disabled by default but can be enabled with use_networkv2_check in your configuration.
• Add support for Autodiscovery for RDS Postgres and MySQL databases.
• Windows: Add remote certificate collection for the Windows Certificate Store integration.
• Add a System Probe module that will collect software inventory data from the host.
• Added logs.truncated and associated aggregate tags into /comp/core/agenttelemetry/impl/config.go
• Workload protection (CWS) can now generate events based on the setsockopt syscall
• Added a new logs.truncated metric to the Agent that reports the number of logs truncated before being sent. This metric helps monitor log volume loss due to truncation and is tagged by service and source for better visibility.

Enhancement Notes

• The agent configcheck --verbose command and flares now include a section that lists all collected configurations, both matched and unmatched. This addition aids debugging by revealing which configurations the Agent has detected.
• Adds in newly supported ap2.datadoghq.com site to the MSI's GUI menu.
• Individual integrations can now set their own auto multiline configurations, including adding custom samples for logs specific to that integration.
• Allows RDS autodicovery to work with an empty tag list. If an empty tag list is provided, the autodiscovery will not filter instances based on tags, allowing all RDS instances to be discovered.
• OpenTelemetry instrumentation scope attributes are now converted into log attributes.
• Introduce a new sample configuration file, application_monitoring.yaml, to support the Hands Off config feature. This file is automatically placed under /etc/datadog-agent/ on Linux systems only. Users can manually edit the file to apply application monitoring configurations.
• Agents are now built with Go 1.24.4.
• ecs_cluster_name is added as a global tag when running on EC2.
• Improve the memory efficiency of obfuscator key generation.
• In OTLP metrics ingestion, the instrumentation_scope_metadata_as_tags option is now enabled by default. This means scope attributes are now added as tags to metrics. If you have too many unique values for instrumentation scope attributes, this may cause cardinality issues. To mitigate this, you can disable the behavior by setting datadog.metrics.instrumentation_scope_metadata_as_tags to false.
• Orchestrator manifests will now be published with all tags present in their metadata counterparts.
• Single Step Instrumentation now uses the Python tracer major version 3 by default.
• Refactor the logs-agent auditor to utilize a more testable architecture.
• Add Kind, ApiVersion, and NodeName to manifests. Add HostName to CollectorManifest.
• Sensitive text from custom resources is now scrubbed from the manifest. If a field is sensitive, all values within that field are automatically redacted, ensuring that sensitive data is not exposed even in nested structures.
• Update registry writer to not write atomically when Agent runs on ECS Fargate to reduce memory leak.
• Updated Windows container image labels to align with Linux image labels for better OCI compliance. Added standard Open Container Initiative (OCI) labels including image source, revision, and version information.

Bug Fixes

• APM: Fix an issue where the trace-agent could panic during shutdown trying to obfuscate a SQL payload.
• APM: Fix an issue where trace-agent could panic with "send on closed channel" during shutdown.
• Prevent Logs Agent registry entries from being removed prematurely when the log source is still active.
• Fixed TCP retransmit counts by excluding TCP keep-alive packets. Also fixed potential IRQL corruption and memory corruption related to IPv6 filters.
• APM: Reduce the log level of APM Traces Received log message to debug. These values are available via metrics so this log is mostly just noisy.
• Factor dependent services into the timeout when stopping the Agent service on Windows. Operations such as the stop-service Agent subcommand and remote updates now wait longer for the Agent and its subservices to stop before reporting an error.
• Fixed debug log message for detected locally defined servers in NTP check.
• Fixes a panic in the checks collector that occasionally occurs when the Agent is shutting down.
• Fixes Python integrations not being persisted after Agent uninstall. Enables persisting integration during fleet updates.
• Fixes multiline stacktraces being split up into separate logs when serverless-init is installed in-process.
• Windows Agent remote updates now submit the remote config task state to the backend. This reduces the time it takes for a remote update to complete.
• Windows Agent installer now uses absolute path to msiexec.exe instead of PATH lookup, improving installation reliability
• Fixes telemetry reporting in the Agent Install Script for Windows PowerShell on hosts using PowerShell version less than 6 and without Internet Explorer installed, such as on a Server Core installation.
• The Datadog Installer service on Windows is now set to manual start. This prevents alerts from tools that monitor automatically started services, such as the Windows Server Manager Dashboard.
• Fix a bug that resulted in some Orchestrator Kubernetes manifests losing the configured "extraTags".
• Fix how the Live Process and Live Containers sets the hostname when running in an Agent that is running in AWS Fargate
• Applies SQL obfuscation logic to OpenTelemetry db semantics. Specifically, db.statement and db.query.text values will be obfuscated along with resource name and sql.query, according to obfuscation settings in the Agent config:

  datadog-agent/pkg/config/config_template.yaml

  Lines 1226 to 1364 in 1768f80

  	## @param obfuscation - object - optional
  	## Defines obfuscation rules for sensitive data.
  	## See https://docs.datadoghq.com/tracing/setup_overview/configure_data_security/#agent-trace-obfuscation
  	#
  	# obfuscation:
  	# credit_cards:
  	## @param DD_APM_OBFUSCATION_CREDIT_CARDS_ENABLED - boolean - optional
  	## Enables obfuscation rules for credit cards. Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_CREDIT_CARDS_LUHN - boolean - optional
  	## Enables a Luhn checksum check in order to eliminate false negatives. Disabled by default.
  	# luhn: false
  	## @param DD_APM_OBFUSCATION_CREDIT_CARDS_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - client_id
  	#
  	# elasticsearch:
  	## @param DD_APM_OBFUSCATION_ELASTICSEARCH_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "elasticsearch". Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_ELASTICSEARCH_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - client_id
  	## @param DD_APM_OBFUSCATION_ELASTICSEARCH_OBFUSCATE_SQL_VALUES - boolean - optional
  	## The set of keys for which their values will be passed through SQL obfuscation
  	# obfuscate_sql_values:
  	# - val1
  	#
  	# opensearch:
  	## @param DD_APM_OBFUSCATION_OPENSEARCH_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "opensearch". Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_OPENSEARCH_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - client_id
  	## @param DD_APM_OBFUSCATION_OPENSEARCH_OBFUSCATE_SQL_VALUES - boolean - optional
  	## The set of keys for which their values will be passed through SQL obfuscation
  	# obfuscate_sql_values:
  	# - val1
  	#
  	# http:
  	## @param DD_APM_OBFUSCATION_HTTP_REMOVE_QUERY_STRING - boolean - optional
  	## Enables obfuscation of query strings in URLs
  	# remove_query_string: false
  	## @param DD_APM_OBFUSCATION_HTTP_REMOVE_PATHS_WITH_DIGITS - boolean - optional
  	## If enabled, path segments in URLs containing digits are replaced by "?"
  	# remove_paths_with_digits: false
  	#
  	# memcached:
  	## @param DD_APM_OBFUSCATION_MEMCACHED_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "memcached". Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_MEMCACHED_KEEP_COMMAND - boolean - optional
  	## If enabled, the full command for the query will be kept, including any lookup
  	## keys that could be present. The value for storage commands will still be
  	## redacted if Memcached obfuscation is enabled.
  	# keep_command: false
  	#
  	# mongodb:
  	## @param DD_APM_OBFUSCATION_MONGODB_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "mongodb". Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_MONGODB_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - document_id
  	## @param DD_APM_OBFUSCATION_MONGODB_OBFUSCATE_SQL_VALUES - object - optional
  	## The set of keys for which their values will be passed through SQL obfuscation
  	# obfuscate_sql_values:
  	# - val1
  	#
  	# redis:
  	## @param DD_APM_OBFUSCATION_REDIS_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "redis". Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_REDIS_REMOVE_ALL_ARGS - boolean - optional
  	## When true, replaces all arguments of a redis command with a single "?". Disabled by default.
  	# remove_all_args: false
  	#
  	# valkey:
  	## @param DD_APM_OBFUSCATION_VALKEY_ENABLED - boolean - optional
  	## Enables obfuscation rules for spans of type "valkey". Enabled by default.
  	# enabled: true
  	#
  	## @param DD_APM_OBFUSCATION_VALKEY_REMOVE_ALL_ARGS - boolean - optional
  	## When true, replaces all arguments of a valkey command with a single "?". Disabled by default.
  	# remove_all_args: false
  	#
  	## @param DD_APM_OBFUSCATION_REMOVE_STACK_TRACES - boolean - optional
  	## Enables removing stack traces to replace them with "?". Disabled by default.
  	# remove_stack_traces: false
  	#
  	# sql_exec_plan:
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_ENABLED - boolean - optional
  	## Enables obfuscation rules for JSON query execution plans. Disabled by default.
  	# enabled: false
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - id1
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_OBFUSCATE_SQL_VALUES - boolean - optional
  	## The set of keys for which their values will be passed through SQL obfuscation
  	# obfuscate_sql_values:
  	# - val1
  	#
  	# sql_exec_plan_normalize:
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_NORMALIZE_ENABLED - boolean - optional
  	## Enables obfuscation rules for JSON query execution plans, including cost and row estimates.
  	## Produces a normalized execution plan. Disabled by default.
  	# enabled: false
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_NORMALIZE_KEEP_VALUES - object - optional
  	## List of keys that should not be obfuscated.
  	# keep_values:
  	# - id1
  	## @param DD_APM_OBFUSCATION_SQL_EXEC_PLAN_NORMALIZE_OBFUSCATE_SQL_VALUES - boolean - optional
  	## The set of keys for which their values will be passed through SQL obfuscation
  	# obfuscate_sql_values:
  	# - val1
  	# cache:
  	## @param DD_APM_OBFUSCATION_CACHE_ENABLED - boolean - optional
  	## Enables caching obfuscated statements. Currently supported for SQL and MongoDB queries.
  	## Enabled by default.
  	# enabled: true
  	## @param DD_APM_OBFUSCATION_CACHE_MAX_SIZE - integer - optional - default: 5000000
  	## The maximum size of the cache in bytes. The maximum allowed resource length is 5000.
  	## Datadog stores a minimum of 1000 queries (5000000 / 5000) by default.
  	# max_size: 5000000
  	## @sql_obfuscation_mode - string - optional - default: ""
  	## @env DD_APM_SQL_OBFUSCATION_MODE - string - optional - default: ""
  	## Obfuscator mode for SQL queries.
  	## Leave empty to use the default obfuscator.
  	## Set to "obfuscate_only" to obfuscate the query with the new `sqllexer` obfuscator.
  	## If you use DBM, set to "obfuscate_and_normalize" to obfuscate and normalize the query for better APM/DBM correlation.
  	#
  	# sql_obfuscation_mode: ""

• Ensure serverless deployments send logs with gzip compression.
• Fix a rare panic that can occur when a log is unable to be written to a TCP-based unreliable endpoint.
• Fixed a bug where the system.cpu.num_cores metric could be incorrect on certain Windows platforms.
• Fixed Windows container image metadata to properly include build timestamps and version information.

Other Notes

• Add Origins for DuckDB, Keda and Supabase
• Add metric origins for the Windows Certificate Store integration.
• Add metric origins for new integrations.
• SystemD units are now written by .deb and .rpm package scripts during the installation process. They were previously part of the package archive. We do not expect this change to affect users.

Datadog Cluster Agent

Prelude

Released on: 2025-07-10 Pinned to datadog-agent v7.68.0: CHANGELOG.

New Features

• The admission controller can now enable kubelet API logging in the injected agent sidecar.

Enhancement Notes

• Added a new metric to expose the ksm kube_cronjob_status_last_successful_time metric. The name of the metric is kubernetes_state.cronjob.duration_since_last_successful.
• Single Step Instrumentation now uses the Python tracer major version 3 by default.

Bug Fixes

• Stop sending telemetry associated with a DatadogMetric when the object is deleted.
• Fix a bug in the Kubernetes State Metrics (KSM) check where custom resource metrics were incorrectly named using the kubernetes_state.customresource.<name> pattern instead of the intended kubernetes_state_customresource.<prefix>_<name> format.
• Fixes a bug in the admission controller webhook that caused volume mounts to be skipped when other webhooks injected init containers after our own volume mounts had been added.
• Properly take into account the timeZone field of the CronJob objects in the kubernetes_state.cronjob.on_schedule_check service check.