DataDog/dd-trace-py v4.15.0rc2

4.15.0rc2

on GitHub

Estimated end-of-life date, accurate to within three months: 05-2027

See the support level definitions for more information.

Upgrade Notes

• openfeature
  • Reports strings, numbers, booleans, datetimes, and null values from evaluation context; unsupported values are no longer converted to strings and are omitted without affecting flag resolution.
• AAP
  • Upgrades libddwaf to version 2.1.0.

New Features

• tracing
  • Adds a discard boolean field to DD_TRACE_SAMPLING_RULES rules to fully drop trace chunks which were rejected during sampling. See the DD_TRACE_SAMPLING_RULES configuration docs for details.
  • add consistent probability sampling support for distributed traces in mixed Datadog - OpenTelemetry environment
  • Adds OpenTelemetry consistent probability sampling state (ot.th and ot.rv) to spans exported through OTLP when OTEL_TRACES_EXPORTER=otlp is configured, allowing downstream collectors to preserve and extrapolate sampling decisions.
  • Runtime metrics now include CPython GC collection counts and stop-the-world pause time and max pause when DD_RUNTIME_METRICS_ENABLED is true.
  • Adds OpenTelemetry thread-context synchronization for applications using uvloop on Linux with Python 3.13 and earlier. It rides on the asyncio integration, is enabled by default, and can be disabled with DD_TRACE_OTEL_CTX_ENABLED=false or DD_TRACE_ASYNCIO_ENABLED=false.
• LLM Observability
  • botocore: The Amazon Bedrock integration now captures tool definitions, tool calls, and tool results on InvokeModel spans for Anthropic models, matching the Converse API.
  • Adds finish_reason to LLM spans to indicate the reason the model stopped generating for the openai, litellm, and anthropic integrations.
  • APM spans for LLM and agent operations now carry gen_ai.* attributes, so model, provider, application, conversation, and token usage can be searched, faceted, and monitored directly in APM.
  • The OpenAI Realtime integration now traces each conversation turn as a span tree instead of a single llm span.
• openfeature
  • Limits the amount and nesting depth of evaluation context data reported with flag evaluations; data outside these limits is omitted without affecting flag resolution.
• CI visibility
  • Adds DD_CIVISIBILITY_DYNAMIC_ATR_ENABLED to enable dynamic Auto Test Retries budgets based on test duration, instead of the flat per-test retry limit. Optionally configure the five duration-based budgets with DD_CIVISIBILITY_DYNAMIC_ATR_BUCKETS (five comma-separated integers in [1, 20]), corresponding to the Early Flake Detection duration buckets.
• uWSGI
  • added support for normal, non-lazy mode, provided threads are enabled and the py-call-uwsgi-fork-hooks option is set.
• anyio
  • Adds native trace-context support for synchronous FastAPI handlers running in AnyIO worker threads on Linux. The integration is enabled by default and can be disabled with DD_TRACE_ANYIO_ENABLED=false.
• kafka
  • Adds support for kafka in the DD_TRACE_PROPAGATION_AS_SPAN_LINKS environment variable. When enabled together with distributed tracing, the kafka.consume span starts a new trace and relates to every consumed message's producer via span links, instead of continuing a single producer's trace. This is disabled by default.
• profiling
  • DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED now defaults to true, so the heap profiler also tracks allocations in the Mem domain on Python 3.12+.
• AAP
  • Adds the build_py option --no-bundle-libddwaf, set through the [build_py] section of setup.cfg, which builds the package without a bundled libddwaf. The library is then loaded from the system at import time (libddwaf.so.2, then libddwaf.so, resolved by the dynamic linker, and required to be 2.x). Intended for Linux distribution packages; default builds bundle libddwaf as before.

Bug Fixes

• AAP
  • Fixes an issue where exceptions raised by urllib3 requests include an Exploit Prevention frame in their traceback, which caused ordinary application errors to be misattributed to Datadog. Enabling urllib3 APM tracing, which is off by default, still contributes a frame of its own.
• crashtracking
  • fix crash reports being sent to the wrong intake when DD_AGENTLESS_ENABLED is set. The errors-intake upload was incorrectly reusing the same endpoint configured for the telemetry crash-report path, sending it to the telemetry intake host instead of the dedicated errors-intake host.
  • Strips the bootstrap directory from the inherited PYTHONPATH before spawning the receiver, so it stays an uninstrumented script. Empty PYTHONPATH entries (the current working directory) are preserved so the receiver can still import ddtrace when it is loaded from the working directory.
• LLM Observability
  • botocore: This fix resolves an issue where the Amazon Bedrock InvokeModel integration only captured the first content block of an Anthropic response, so output messages were empty or truncated when a response began with a tool_use or thinking block.
  • Fixes an issue where span and evaluation metric events can be lost when intake temporarily returns a server error.
  • Fixes an issue where enabling LLM Observability together with DD_APM_TRACING_ENABLED=false discarded every trace, so AI Guard, App and API Protection, Code Security and Software Composition Analysis reported no spans while running in standalone mode. These traces are now sent, and APM billing stays opted out.
  • This fix resolves an issue where reasoning content from streamed google_genai responses was merged into the output message instead of being captured as a separate reasoning message.
• CI Visibility
  • Fixes an issue where the pytest plugin entered a code coverage collection context around every test even when code coverage was disabled, which reported coverage as active while no coverage was being collected and, on Python 3.12 and later, restarted sys.monitoring events once per test.
  • Fixes an issue where collecting coverage could abort the session with a TypeError.
  • Fixes ValueError: I/O operation on closed file logging tracebacks at the end of pytest sessions when standard stream handlers point to streams closed by pytest. Tracer logs continue to reach healthy root handlers, including file output and log forwarding.
  • Fixes an issue where test events can be reported without their test session when running pytest-xdist with the legacy pytest plugin.
• botocore
  • EventBridge PutEvents now propagates Data Streams Monitoring (DSM) context when DD_DATA_STREAMS_ENABLED is enabled.
• AI Guard
  • Fixes an issue where transport and unexpected internal evaluation failures were reported as client_error, making them difficult to diagnose.
• ASM
  • Fixes an issue where Exploit Prevention does not evaluate SSRF rules for calls to webbrowser.open, or for calls to urllib.request.urlopen that pass the URL positionally or go through a custom opener installed with urllib.request.install_opener.
  • Fixes an issue where exceptions raised by urllib.request.urlopen and http.client requests include an internal Datadog frame in their traceback when Exploit Prevention is enabled, which caused ordinary application errors to be misattributed to the tracer.
• opentelemetry
  • Fixes an issue where empty OTEL_* environment variables override ddtrace defaults instead of being treated as unset.
• httplib
  • Fixes an issue where an outgoing request blocked by Exploit Prevention leaves the http.client.request span unfinished and active, so it is never sent and every later span on that thread is parented to it.
• internal
  • Fixes a memory leak that causes memory usage to grow over time in long-running processes, when instrumentation applied to a function fails while entering, returning from or unwinding that function. Each such call retained per-call state, and in some cases the frame of the instrumented function.
  • Fixes race conditions after process forks that can cause telemetry recording to stall or background workers to restart incorrectly when multiple threads resume concurrently or forks are nested.
  • Fixes an issue where Python-forked child processes can emit incorrect telemetry lifecycle events or block while cleaning up an inherited telemetry worker.
• mysql
  • Fixes an issue where tracing and SQL-injection detection or blocking are skipped for database connections created with mysql.connector.Connect().
• rq
  • Fixes an issue where the rq.worker.perform_job span was not created when jobs were processed by rq.SimpleWorker on rq >= 2.7.
• selenium
  • Fixes a memory leak where memory usage grows over the lifetime of a browser test session, proportionally to the number of WebDriver.get, WebDriver.quit and WebDriver.close calls made.
• tracing
  • Fixes an issue where switching a product into standalone mode at runtime with tracer.configure(apm_tracing_disabled=True) did not apply the standalone sampling settings, so traces were sent without the one trace per minute limit and without the _dd.apm.enabled=0 metric that opts them out of APM billing.
  • Fixes an issue where the datadog.runtime_id resource attribute on OTLP trace and trace-metrics exports was a randomly generated value instead of the tracer's runtime id, preventing those exports from being correlated with the other signals emitted by the same process.
• dynamic instrumentation
  • Fixes an issue where Symbol Database uploads could be disabled in the first worker when multiple processes fork concurrently.
• telemetry
  • Fixes an issue where an extra, redundant telemetry request was sent at process shutdown, adding avoidable latency to application teardown. The final telemetry payload is now sent as a single request.
• mcp
  • Resolves an error logged when a module named mcp is importable but is not the MCP SDK. Patching is now skipped instead of raising.
• profiling
  • Stack collection now honors configured frame limits, reducing profiler CPU and memory overhead for deep Python call stacks.

Other Changes

• CI Visibility
  • Reduces the cost of the source-location lookup the pytest plugin performs once per test, which is most noticeable on large suites of fast tests.