Agent

Prelude

Release on: 2022-12-12

• Please refer to the 7.41.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Troubleshooting commands in the Agent CLI have been moved to the diagnose command. troubleshooting metadata_v5 command is now diagnose show-metadata v5 and troubleshooting metadata_inventory is diagnose show-metadata inventory.
• Journald launcher can now create multiple tailers on the same journal when config_id is specified. This change enables multiple configs to operate on the same journal which is useful for tagging different units. Note: This may have an impact on CPU usage.
• Upgrade tracer_agent debugger proxy to use logs intake API v2 for uploading snapshots
• The Agent now defaults to TLS 1.2 instead of TLS 1.0. The force_tls_12 configuration parameter has been removed since it's now the default behavior. To continue using TLS 1.0 or 1.1, you must set the min_tls_version configuration parameter to either tlsv1.0 or tlsv1.1.

New Features

• Added a required infrastructure to enable protocol classification for Network Performance Monitoring in the future. The protocol classification will allow us to label each connection with a L7 protocol. The features requires Linux kernel version 4.5 or greater.
• parse the snmp configuration from the agent and pass it to the integrated snmpwalk command in case the customer only provides an ip address
• The Agent can send its own configuration to Datadog to be displayed in the Agent Configuration section of the host detail panel. See https://docs.datadoghq.com/infrastructure/list/#agent-configuration for more information. The Agent configuration is scrubbed of any sensitive information and only contains configuration you’ve set using the configuration file or environment variables.
• Windows: Adds support for Windows Docker "Process Isolation" containers running on a Windows host.

Enhancement Notes

• APM: All spans can be sent through the error and rare samplers via custom feature flag error_rare_sample_tracer_drop. This can be useful if you want to run those samplers against traces that were not sampled by custom tracer sample rules. Note that even user manual drop spans may be kept if this feature flag is set.
• APM: The trace-agent will log failures to lookup CPU usage at error level instead of debug.
• Optionally poll Agent and Cluster Agent integration configuration files for changes after startup. This allows the Agent/Cluster Agent to pick up new integration configuration without a restart. This is enabled/disabled with the autoconf_config_files_poll boolean configuration variable. The polling interval is configured with the autoconf_config_files_poll_interval (default 60s). Note: Dynamic removal of logs configuration is currently not supported.
• Added telemetry for the "container-lifecycle" check.
• On Kubernetes, the "cluster name" can now be discovered by using the Node label ad.datadoghq.com/cluster-name or any other label key configured using to the configuration option: kubernetes_node_label_as_cluster_name
• Agents are now built with Go 1.18.8.
• Go PDH checks now all use the PdhAddEnglishCounter API to ensure proper localization support.
• Use the windows_counter_refresh_interval configuration option to limit how frequently the PDH object cache can be refreshed during counter initialization in golang. This replaces the previously hardcoded limit of 60 seconds.
• [netflow] Add disable port rollup config.
• The OTLP ingest endpoint now supports the same settings and protocol as the OpenTelemetry Collector OTLP receiver v0.61.0.
• The disable_file_logging setting is now respected in the process-agent.
• The process-agent check [check-name] command no longer outputs to the configured log file to reduce noise in the log file.
• Logs a warning when the process agent cannot read other processes due to misconfiguration.
• DogStatsD caches metric metadata for shorter periods of time, reducing memory usage when tags or metrics received are different across subsequent aggregation intervals.
• The agent CLI subcommands related to Windows services are now consistent in use of dashes in the command names (install-service, start-service, and so on). The names without dashes are supported as aliases.
• The Agent now uses the V2 API to submit series data to the Datadog intake by default. This can be reverted by setting use_v2_api.series to false.

Deprecation Notes

• APM: The Rare Sampler is now disabled by default. If you wish to enable it explicitly you can set apm_config.enable_rare_sampler or DD_APM_ENABLE_RARE_SAMPLER to true.

Bug Fixes

• APM: Don't include extra empty 'env' entries in sampling priority output shown by agent status command.

• APM: Fix panic when DD_PROMETHEUS_SCRAPE_CHECKS is set.

• APM: DogStatsD data can now be proxied through the "/dogstatsd/v1/proxy" endpoint and the new "/dogstatsd/v2/proxy" endpoint over UDS, with multiple payloads separated by newlines in a single request body. See https://docs.datadoghq.com/developers/dogstatsd#setup for configuration details.

• APM - remove extra error message from logs.

• Fixes an issue where cluster check metrics would be sometimes sent with the host tags.

• The containerd check no longer emits events related with pause containers when exclude_pause_container is set to true.

• Discard aberrant values (close to 18 EiB) in the container.memory.rss metric.

• Fix Cloud Foundry CAPI Metadata tags injection into application containers.

• Fix Trace Agent's CPU stats by reading correct PID in procfs

• Fix a potential panic when df outputs warnings or errors among its standard output.

• The OTLP ingest is now consistent with the Datadog exporter (v0.56+) when getting a hostname from OTLP resource attributes for metrics and traces.

• Make Agent write logs when SNMP trap listener starts and Agent receives invalid packets.

• Fixed a bug in the workloadmeta store. Subscribers that asked to receive only unset events mistakenly got set events on the first subscription for all the entities present in the store. This only affects the container_lifecycle check.

• Fix missing tags on the kubernetes_state.cronjob.complete service check.

• In kubernetes_state_core check, fix the labels_as_tags feature when the same Kubernetes label must be turned into different Datadog tags, depending on the resource:

  labels_as_tags:
  daemonset:
  first_owner: kube_daemonset_label_first_owner

  deployment:
  first_owner: kube_deployment_label_first_owner

• Normalize the EventID field in the output from the windowsevent log tailer. The type will now always be a string containing the event ID, the sometimes present qualifier value is retained in a new EventIDQualifier field.

• Fix an issue where the security agent would panic, sending on a close channel, if it received a signal when shutting down while all components were disabled.

• Fix tokenization of negative numeric values in the SQL obfuscator to remove extra characters prepended to the byte array.

Datadog Cluster Agent

New Features

• Add Namespace collection in the orchestrator check and enable it by default.

Enhancement Notes

• Improves performance of the Cluster Agent admission controller on large pods.