Release Notes

Upgrade Notes

• google-adk: With google-adk 2.9.0 and later, tool call spans use the resource name FunctionTool._call_tool_async (one leading underscore) instead of FunctionTool.__call_tool_async, following a rename in google-adk itself. Update any monitor or dashboard that matches on the old resource name.

• kafka: Adds support for Python 3.14.

New Features

• azure_durable_functions: Adds tracing for orchestration function invocations and, when Durable Functions distributed tracing V2 is enabled, connects durable orchestration and activity spans to the originating trace without requiring OpenTelemetry configuration.

• tracing: Adds DD_TRACE_STATS_CARDINALITY_LIMIT (default 7000), DD_TRACE_STATS_RESOURCE_CARDINALITY_LIMIT (default 1024), DD_TRACE_STATS_HTTP_ENDPOINT_CARDINALITY_LIMIT (default 512), DD_TRACE_STATS_PEER_TAGS_CARDINALITY_LIMIT (default 512) and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT (default 100) to bound the number of distinct trace metrics aggregation keys tracked per time bucket when client-side stats computation is enabled. Values beyond a limit are aggregated together under a sentinel value.

• trio: Adds native trace-context support for Trio tasks and worker threads on Linux. The integration is enabled by default and can be disabled with DD_TRACE_TRIO_ENABLED=false.

• LLM Observability: The agent argument of LLMObs.annotate() and LLMObs.annotation_context() now accepts name, instructions, model, model_settings and tools alongside version, and reports them as the agent's manifest on agent spans for manual annotation.

• LLM Observability: Introduces tag-based head sampling via the DD_LLMOBS_SAMPLING_RULES environment variable, which mirrors the format of DD_TRACE_SAMPLING_RULES. See the documentation for details.

• LLM Observability: Managed chat prompts can now include named message placeholders that ManagedPrompt.format() replaces with caller-provided text or tool message lists in their authored position.

• openfeature: adds safe direct Event Platform intake fallback for exposure and flag-evaluation events when the agentless provider cannot use the local Agent's EVP proxy route. Direct requests use API-key authentication and preserve at-most-once delivery for ambiguous local failures.

• openfeature: By default, feature flag evaluation data sent to Datadog now uses hashed user identifiers and omits evaluation context attributes to limit sensitive data collection. Enabling full evaluation data for an environment includes the original identifiers and attributes to help investigate flag usage and unexpected behavior.

• LLM Observability: Managed prompts now expose version-owned JSON configuration through ManagedPrompt.config and accept it through the config argument when creating prompts or versions.

Security Issues

• tracing: Fixes a security vulnerability, as described in GHSA-3633-r77q-h3v3.

Bug Fixes

• AI Guard: Fixes an issue where model responses were never evaluated for LangChain chat model and LLM calls, leaving unsafe model output neither reported nor blocked. Such calls now run a second evaluation covering the model's answer, including any tool calls it requested, which raises AIGuardAbortError when the verdict is DENY or ABORT. A tool call decided by an instrumented agent is still evaluated only once. Streamed LangChain responses are not covered by this change.

• LLM Observability: Fixes an issue where a LangChain model response blocked by AI Guard resulted in LLM Observability spans with empty output and no token usage.

• LLM Observability: resolves an issue where a streamed Bedrock InvokeModelWithResponseStream request that ended in an error or was cancelled part-way produced a span with no output annotation. The text received before the failure is now tagged on the span.

• LLM Observability: resolves an issue where a streamed Bedrock ConverseStream request that ended in an error or was cancelled part-way produced a span that was not marked as an error, making an interrupted request indistinguishable from a successful one.

• CI Visibility: Code Coverage report uploads now apply [paths] aliases from .coveragerc to SF: lines before uploading, fixing duplicate file entries for installed packages.

• dynamic instrumentation: Fixes an issue where sensitive values were not redacted when the name used -, ., @, or $.

• LLM Observability: Fixes an issue where abandoned or disconnected streamed LLM requests can leave spans unfinished, causing memory to grow in long-running web workers.

• AI Guard: Fixes an issue where network.client.ip and ai_guard.network.client.ip contain an IP resolved from proxy headers instead of the peer address of the incoming connection. These tags are omitted when the peer address is unavailable.

• AI Guard: Fixes an issue where credentials embedded in DD_AI_GUARD_ENDPOINT could be written to debug logs, exception messages and span error tags when an evaluation failed to reach the AI Guard service. Any user info in the endpoint is now removed before the request is sent, and the endpoint is redacted from failure messages. The underlying transport error is no longer chained onto the raised AIGuardClientError, which now reports that error's type and redacted message in its own message instead.

• CI Visibility: Fixes an issue where collecting per-test code coverage across threads could terminate a test run with an internal error or attribute coverage to the wrong test.

• CI Visibility: Fixes an issue where code coverage path aliases from the active coverage configuration were not applied to LCOV source filenames when the mapped destination files did not exist on disk. This caused uploaded coverage reports to retain separate installed-package paths for the same repository file instead of deduplicating them to the canonical repository path.

• internal: Fixes incorrect telemetry being collected when hitting cardinality limits during stats computation.

• google-adk: Fixes an issue where applications using google-adk 2.9.0 and later fail to start when tracing is enabled, or start normally but report no tool call or code execution spans. Tool calls and code execution are now traced on all supported google-adk versions.

• grpc: Fixes an issue where creating synchronous or asynchronous gRPC servers with interceptors=None raises a TypeError when tracing is enabled.

• Code Security (IAST): Fixes an issue where calling join on a bytes or bytearray separator with an element of the wrong type crashes the process instead of raising TypeError.

• AAP: Fixes false-positive and duplicate unvalidated redirect reports in Flask applications when a request parameter matches the response Location header.

• Code Security (IAST): fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text containing a lone surrogate character.

• LLM Observability: Fixes an issue where formatting managed text or chat prompts consumes a closing JSON object brace immediately after a single-brace placeholder, producing malformed JSON. Braces surrounding placeholders are now preserved.

• LLM Observability: Fixes an issue where agent attribution was propagated to downstream services from spans that had no agent ancestor. #19531

• profiling: A bug where Lock profiling would not associate samples with spans and traces when PyTorch profiling was enabled has been fixed.

• internal: Fixes an issue on macOS where starting a subprocess can deadlock while a hostname lookup is in progress.

• tracing: Fixes an issue where traces can be dropped when flushing while the tracer is shutting down.

• openfeature: Fixes an issue where flag evaluations return local defaults and the provider remains not ready when Remote Configuration completes its first poll during application startup.

• profiling: Fixes an issue where an unexpected error while uploading a profile stopped the profiler from uploading any further profiles for the lifetime of the process.

• profiling: An issue where unexpected errors while stopping the profiler prevented the profiler from stopping completely has been fixed.

• CI Visibility: Fixes an issue where Auto Test Retries do not retry tests that use pytest-timeout thread mode and terminate a pytest-xdist worker.

• tracing: Fixes an issue where the Python garbage collector stops and memory grows without limit when DD_RUNTIME_METRICS_ENABLED is true.

• CI Visibility: Fixes an incompatibility between vcrpy and Test Optimization that prevents test traces from being uploaded to Datadog.

• LLM Observability: An issue where where streamed LLM requests could leave spans unfinished and cause memory to grow in long-running processes has been fixed.

• LLM Observability: Fixes an issue where gen_ai.* attributes on APM spans could produce a duplicate LLM Observability span when LLM Observability was disabled.

• LLM Observability: Fixes an issue where a traced stream opened as a context manager through a stream manager could close before any chunks were read if the caller omitted as, and where errors raised while closing the wrapped stream were missing from the span.

• langchain: Fixes an issue where discarding an unused traced stream could end an enclosing AI Guard evaluation early.

• LLM Observability: Fixes an issue where APM spans annotated with gen_ai attributes could cause a duplicate LLM Observability span to be created.

• LLM Observability: This fix resolves an issue where cancelled or failed OpenAI streamed requests produced no LLM Observability span at all. These requests now produce an LLM span carrying the input messages, whatever output arrived before the stream ended, and estimated token metrics.

• tracing: Fixes a native memory leak on Linux when a thread exits while an OpenTelemetry thread context is still attached. Publication is enabled by default and can be disabled by setting DD_TRACE_OTEL_CTX_ENABLED to false.

• Profiling: A crash that could occur in applications that use signal handlers to recover from crashes has been fixed.

Other Changes

• tracing: Reduces trace encoding time in the default Datadog trace exporter by approximately 18–42% in benchmarks, depending on the trace workload.