Agent

Prelude

Release on: 2023-01-23

• Please refer to the 7.42.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Downloading and installing official checks with agent integration install is no longer supported for Agent installations that do not include an embedded python3.

New Features

• Adding the kube_api_version tag to all orchestrator resources.

• Kubernetes Pod events generated by the kubernetes_apiserver can now benefit from the new cluster-tagger component in the Cluster-Agent.

• APM OTLP: Added compatibility for the OpenTelemetry Collector's datadogprocessor to the OTLP Ingest.

• The CWS agent now supports rules on mount events.

• Adding a configuration option, exclude_ec2_tags, to exclude EC2 instance tags from being converted into host tags.

• Adds detection for a process being executed directly from memory without the binary present on disk.

• Introducing agent sampling rates remote configuration.

• Adds support for secret_backend_command_sha256 SHA for the secret_backend_command executable. If secret_backend_command_sha256 is used, the following restrictions are in place:

  • Value specified in the secret_backend_command setting must be an absolute path.

  - Permissions for the datadog.yaml config file must disallow write access by users other than ddagentuser or Administrators on Windows or the user running the Agent on Linux and macOS. The agent will refuse to start if the actual SHA256 of the secret_backend_command executable is different from the one specified by secret_backend_command_sha256. The secret_backend_command file is locked during verification of SHA256 and subsequent run of the secret backend executable.

• Collect network devices topology metadata.

• Add support for AWS Lambda Telemetry API

• Adds three new metrics collected by the Lambda Extension

  `aws.lambda.enhanced.response_latency`: Measures the elapsed time in milliseconds from when the invocation request is received to when the first byte of response is sent to the client.

  `aws.lambda.enhanced.response_duration`: Measures the elapsed time in milliseconds between sending the first byte of the response to the client and sending the last byte of the response to the client.

  `aws.lambda.enhancdd.produced_bytes`: Measures the number of bytes returned by a function.

• Create cold start span representing time and duration of initialization of an AWS Lambda function.

Enhancement Notes

• Adds both the StartTime and ScheduledTime properties in the collector for Kubernetes pods.
• Add an option (hostname_trust_uts_namespace) to force the Agent to trust the hostname value retrieved from non-root UTS namespaces (Linux only).
• Metrics from Giant Swarm pause containers are now excluded by default.
• Events emitted by the Helm check now have "Error" status when the release fails.
• Add an annotations_as_tags parameter to the kubernetes_state_core check to allow attaching Kubernetes annotations as Datadog tags in a similar way that the labels_as_tags parameter does.
• Adds the windows_counter_init_failure_limit option. This option limits the number of times a check will attempt to initialize a performance counter before ceasing attempts to initialize the counter.
• [netflow] Expose collector metrics (from goflow) as Datadog metrics
• [netflow] Add prometheus listener to expose goflow telemetry
• OTLP ingest now uses the minimum and maximum fields from delta OTLP Histograms and OTLP ExponentialHistograms when available.
• The OTLP ingest endpoint now reports the first cumulative monotonic sum value if the timeseries started after the Datadog Agent process started.
• Added the workload-list command to the process agent. It lists the entities stored in workloadmeta.
• Allows running secrets in the Process Agent on Windows by sandboxing secret_backend_command execution to the ddagentuser account used by the Core Agent service.
• Add process_context tag extraction based on a process's command line arguments for service monitoring. This feature is configured in the system-probe.yaml with the following configuration: service_monitoring_config.process_service_inference.enabled.
• Reduce the overhead of using Windows Performance Counters / PDH in checks.
• The OTLP ingest endpoint now supports the same settings and protocol as the OpenTelemetry Collector OTLP receiver v0.64.1
• The OTLP ingest endpoint now supports the same settings and protocols as the OpenTelemetry Collector OTLP receiver v0.66.0.

Deprecation Notes

• Removes the install-service Windows agent command.
• Removes the remove-service Windows agent command.

Security Notes

• Upgrade the wheel package to 0.37.1 for Python 2.
• Upgrade the wheel package to 0.38.4 for Python 3.

Bug Fixes

• APM: Fix an issue where container tags weren't working because of overwriting an essential tag on spans.
• APM OTLP: Fix an issue where a span's local "peer.service" attribute would not override a resource attribute-level service.
• On Windows, fixes a bug in the NPM network driver which could cause a system crash (BSOD).
• Create only endpoints check from prometheus scrape configuration when prometheus_scrape.service.endpoint option is enabled.
• Fix how Kubernetes events forwarding detects the Node/Host.
  • Previously Nodes' events were not always attached to the correct host.
  • Pods' events from "custom" controllers might still be not attached to a host if the controller doesn't set the host in the source.host event's field.
• APM: Fix SQL parsing of negative numbers and improve error message.
• Fix a potential panic when df outputs warnings or errors among its standard output.
• Fix a bug where a misconfig error does not show when hidepid=invisible
• The agent no longer wrongly resolves its hostname on ECS Fargate when requests to the Fargate API timeout.
• Metrics reported through OTLP ingest now have the interval property unset.
• Fix a PDH query handle leak that occurred when a counter failed to add to a query.
• Remove unused environment variables DD_AGENT_PY and DD_AGENT_PY_ENV from known environment variables in flare command.
• APM: Fix SQL obfuscator parsing of identifiers containing dollar signs.

Other Notes

• JMXFetch upgraded to 0.47.2
• Bump embedded Python3 to 3.8.16.

Datadog Cluster Agent

New Features

• Supports the collection of custom resource definition and custom resource manifests for the orchestrator explorer.

Enhancement Notes

• Collects Unified Service Tags for the orchestrator explorer product.