DataDog/dd-trace-py v4.11.0rc1

4.11.0rc1

on GitHub

Upgrade Notes

• flask: Requests served under a non-empty WSGI SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.

• flask: API endpoint discovery now reports every HTTP method the framework serves — including Werkzeug's auto-added HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].

New Features

• tracing: collect the x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.

• Database Monitoring (DBM) propagation supports dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.

• ai_guard: add AI Guard evaluation support to the Anthropic SDK Messages instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client (covering Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.

• aiokafka: Adds kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.

• ASM: This introduces the _dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.

• ASM: Extends _dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).

• LLM Observability: The claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.

• aws_durable_execution_sdk_python: Adds distributed tracing for durable workflows across suspend/resume cycles, so a workflow that pauses and resumes in a later invocation appears as a single connected trace. See aws_durable_execution_sdk_python for details and opt-out configuration.

• AAP: API endpoint discovery now covers Flask sub-applications mounted via werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.

• google_cloud_pubsub: This introduces Data Streams Monitoring (DSM) context propagation for the Google Cloud Pub/Sub integration. Producer publish operations inject the DSM pathway context into message attributes, and subscriber callbacks extract it and record a consume checkpoint. To enable, set DD_DATA_STREAMS_ENABLED=true.

• LLM Observability: automatically tags spans and experiments with git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.

• LLM Observability: experiment evaluators and summary evaluators can now return a MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.

• LLM Observability: Add LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.

• LLM Observability: task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.

• profiling: The fast stack profiler optimisation is now enabled by default.

• Profiling: The heap profiler can now track allocations made through PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.

• ray: Added DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.

• CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).

  Example:

  from pydantic_evals.evaluators import EqualsExpected import logging

  from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter

  handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)

  while True:
  job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)

  handler.close()

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.

• aiokafka: Collects the Kafka cluster ID when tracing aiokafka producers and consumers and forwards it to Data Streams Monitoring, matching the behavior of the confluent_kafka integration.

• aap: Adds instrumentation telemetry distributions for WAF and RASP execution durations, including waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.

• azure_cosmos: This fix resolves an issue where cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.

• CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.

• tracing: Exclude wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.

• DD_KOMBU_SERVICE, DD_JINJA2_SERVICE, DD_DJANGO_DATABASE_SERVICE, DD_CELERY_PRODUCER_SERVICE, DD_CELERY_WORKER_SERVICE, and DD_DJANGO_CACHE_SERVICE now override the service tag on their respective spans. Previously they were ignored in favor of the DD_<INTEGRATION>_SERVICE_NAME form.

• Fixed a bug in the kombu integration where programmatically setting config.kombu["service"] before patching was applied to producer spans but not consumer spans. Both now read from the same service config source.

• falcon: Fixes an issue where a None value for req.uri_template caused a TypeError when constructing the route in the Falcon tracing middleware.

• ASM: Fixes body inspection silently skipping requests whose Content-Type header carried a parameter (most commonly charset=utf-8).

• Fixed an issue that could have caused some instrumented code to fail to execute correctly when the original function had keyword arguments passed in as a cell variable.

• CI Visibility: Truncates string meta tag values to 5000 characters when encoding test-cycle payloads, preventing rejected payloads caused by tag values that exceed the backend maximum length.

• LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where a span's error message showed an uncategorized unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.

• LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where sequential ClaudeSDKClient.query() calls produced nested agent spans instead of separate root traces. Agent spans are now finalized eagerly when a ResultMessage is observed, rather than relying on the wrapping async generator's finally block to execute.

• dynamic instrumentation: Fixes debugger argument/local captures misclassifying keyword-only arguments and dropping *args / **kwargs for any function that declares keyword-only parameters alongside variadic arguments.

• IAST: This fix resolves an issue where IAST could report a false positive vulnerability against a request whose input did not actually contain tainted data. A concurrent or still-open request holding a tainted value could leak its taint into the current request's checks. Queries are now scoped to the calling request's slot.

• langchain, botocore: This fix resolves an issue where auto-instrumented langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.

• LLM Observability: Fixes an issue where reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.

• profiling: An issue where monkey patching in gevent silently drops all lock profiling samples, caused by gevent monkey-patching threading lock primitives in place after the profiler had already patched them.

• ASM: Fixes multipart body parts whose Content-Type header carried parameters being dropped.

• Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first execution.

• internal: This fix resolves an issue where calling awake() on a PeriodicThread after it had been stopped would block forever. awake() now returns immediately in that case.

• internal: Fix a memory leak where reference cycles through PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.

• profiling: Fixes a memory leak in native frame tracking caused by unbounded native call-site metadata growth.

• pydantic_ai: Fixes APM span naming so the operation name is the generic category (pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.

• SCA: This fix resolves an issue where unresolved runtime reachability targets could accumulate across Software Composition Analysis updates, causing resident memory usage to grow over time.

• tracing: Fixes the subprocess integration silently spans for any shell command whose env-var assignment contained = in the value. tracing: The subprocess integration now correctly scrubs env-var values for env-var names containing digits.

• dynamic instrumentation: fixes an issue where the Symbol Database uploader sends empty payloads on a recurring timer.

• CI Visibility: Fixes spurious git warnings when running tests with pytest-xdist. Simultaneous git fetch --update-shallow calls now retry with exponential back-off on .git/shallow.lock contention, skip the fetch entirely once a sibling worker has already unshallowed the repository, and run under the C locale so the contention check is robust against non-English git messages. git merge-base is deferred until after unshallowing so the required commits are available locally.

• Flare: Fixes a bug where the live tracer configuration was mutated when redacting the API key before writing the flare diagnostic file.

• LLM Observability: Fixed an issue where _current_trace_context did not propagate _dd.p.llmobs_parent_id into the trace context metadata. Without this value, downstream consumers (e.g. dd-trace-go) could not locate the active LLMObs parent span and would create orphaned child traces instead of correctly continuing the parent trace.

• LLM Observability: Fixes an issue where document content on embedding and retrieval spans bypassed the span processor. LLMObsSpan now exposes input_documents and output_documents for mutation or removal before export.

• LLM Observability: Fixes provider mis-attribution on openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.

• Profiling: This fixes an issue where thread pool worker spans were not correctly being linked to their profiles

• profiling: A bug where the Profiler could crash when Python's fault handler utility was enabled has been fixed.

• profiling: fixed an issue where the stack sampler over-counted total CPU time for gevent workloads.

• profiling: A rare crash that could happen after fork in fork-based applications has been fixed.

• profiling: The Profiler now flushes the last profile on SIGINT and SIGTERM signals.

• tracing: Fixes a bug where running ddtrace-run caused a traceback on keyboard interrupt.

• kafka: Fixes a crash in the confluent-kafka integration where internal list_topics calls triggered librdkafka background metadata-refresh tasks.

Other Changes

• opentelemetry: Adds OTEL_METRIC_EXPORT_TIMEOUT and OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to the allowlist of recognized OpenTelemetry environment variables. Previously, setting any of these variables produced an inaccurate OpenTelemetry configuration ... is not supported by Datadog warning at startup, even though the variables are valid OpenTelemetry SDK settings.

• LLM Observability: when LLMObs is enabled in agentless mode (Datadog Agent not reachable or with DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.

• profiling: Each uploaded profile now exposes the effective profiler configuration under info.profiler.settings on the upload event, so profiles can be filtered by individual settings.

• ray: Cache host name and Ray runtime invariants (gcs_address, namespace, version, etc.) at first access. Per-call lookups (socket.gethostname(), ray.get_runtime_context().get_*()) are amortized across the thread lifetime, reducing overhead for high-frequency Ray task and actor-method instrumentation.