DataDog/dd-trace-py v4.13.0rc1

4.13.0rc1

on GitHub

Upgrade Notes

• LLM Observability:
  • Deprecates the ml_app argument and DD_LLMOBS_ML_APP environment variable, which will be removed in the next major version of ddtrace. Use agent_service and DD_SERVICE instead to group traces belonging to a root agentic service (i.e. previously known as ML application).

Deprecation Notes

• AI Guard:

  • The public SDK has moved from the ddtrace.appsec.ai_guard package to the top-level ddtrace.aiguard package. Importing from ddtrace.appsec.ai_guard is deprecated, now emits a ddtrace.DDTraceDeprecationWarning, and will be removed in 5.0.0. As an alternative, update imports such as from ddtrace.appsec.ai_guard import AIGuardClient to from ddtrace.aiguard import AIGuardClient. The Strands Agents classes are the exception: import them from ddtrace.aiguard.integrations.strands (AIGuardStrandsPlugin and AIGuardStrandsHookProvider are not re-exported from the top-level package).

• pytest:

  • The legacy pytest plugin (ddtrace/contrib/internal/pytest) is deprecated and will be removed in ddtrace 5.0.0. This plugin is active only when DD_PYTEST_USE_NEW_PLUGIN is explicitly set to false or 0. The new pytest plugin (ddtrace/testing/internal/pytest) is already the default. To migrate, remove the DD_PYTEST_USE_NEW_PLUGIN environment variable from your configuration.

• LLM Observability:

  • The label parameter of LLMObs.get_prompt() and LLMObs.refresh_prompt() is deprecated. Set DD_ENV instead; the prompt version is resolved for that environment.
  • Managed prompts no longer expose deprecated deployment label metadata. #19047

New Features

• Tracing:

  • Trace and span IDs are now generated more safely in AWS Lambda MicroVM environments, reducing the risk of duplicate IDs. No configuration is required. #18017
  • Adds OpenTelemetry thread-context publication on Linux, enabling compatible profiling tools to correlate samples with the active trace and span.
  • Adds DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans created through OpenTelemetry APIs and for OpenTelemetry Protocol (OTLP) trace export.

• AI Guard:

  • This introduces per-integration kill-switch environment variables, extending the existing DD_AI_GUARD_OPENAI_ENABLED switch to the Anthropic and LangChain auto-instrumentations: DD_AI_GUARD_ANTHROPIC_ENABLED and DD_AI_GUARD_LANGCHAIN_ENABLED (both true by default). Set either to false to disable AI Guard for that specific provider/framework without affecting other integrations or requiring a tracer version rollback.

• AAP:

  • Add support for the DD_APPSEC_AGENTIC_ONBOARDING environment variable. Its value is reported verbatim in configuration telemetry, letting Datadog record that a service was onboarded through the agentic onboarding solution.

• LLM Observability:

  • Added DD_BOTOCORE_BEDROCK_RESOLVE_INFERENCE_PROFILE (default false). When enabled, Bedrock calls using an application-inference-profile ARN as modelId resolve the underlying foundation model, so the span reports the real model and cost populates. Requires DD_LLMOBS_ENABLED=true and bedrock:GetInferenceProfile IAM permission.
  • This introduces capture of session metadata (session id, user id, and app name) from the Google ADK Session object on LLM Observability spans.
  • Spans generated under an agent are now automatically attributed to their nearest agent ancestor, both within a single process and across service boundaries. This enables the Datadog backend to group and display spans by the agent that produced them.
  • Added agent_service as a preferred alias for the ml_app argument across SDK methods (LLMObs.enable(...), span constructors, decorators, submit_evaluation(...), get_spans(...), and publish_evaluator(...)). When provided, agent_service takes precedence over the deprecated ml_app argument and over service. When neither agent_service nor ml_app is set, application identity continues to resolve from a propagated parent value, then DD_LLMOBS_ML_APP, then DD_SERVICE.
  • Auto-instrumented LLM integrations now automatically link Datadog Managed Prompts retrieved with LLMObs.get_prompt() and rendered with .format() to the resulting LLM span.
  • LLMObs.get_prompt() now resolves environment-scoped prompt versions and supports targeting for A/B testing. Set DD_ENV to select the version for that environment. The signature is now keyword-only after prompt_id and accepts targeting_key (sticky bucketing) and **attributes (targeting rules). #18127
  • Adds a version argument to LLMObs.get_prompt() for retrieving an exact prompt version. #19050
  • This introduces support for attaching images to messages on LLM spans. Input and output messages annotated via LLMObs.annotate now accept an optional image_parts field, a list of images each with a mime_type and either inline base64 content or an attachment_key. This lets images be collected and rendered in LLM Observability.
  • The OpenAI integration now automatically instruments the Realtime API (client.realtime.connect(...), both sync and async). Each conversation turn is captured as an llm span with the user and assistant transcripts, audio, token usage, and any tool calls (function and MCP), grouped per connection so the whole conversation is traced end to end. Realtime instrumentation can be disabled on its own, without affecting the rest of the OpenAI integration, by setting DD_OPENAI_REALTIME_ENABLED=false.
  • Adds prompt management methods to LLMObs for creating, updating, deleting, and listing prompts and prompt versions in the prompt registry. #18186
  • Adds the env_ids argument to prompt creation and version update methods, allowing prompt versions to be deployed to feature-flag environments.
  • session_id now propagates to following spans in a trace. If session_id is set explicitly on a span, this will take precedence over the inherited or propagated value.
  • Adds support for submitting LLM Observability spans for synchronous and asynchronous streaming chat completion calls using the mistralai client.

• CI visibility:

  • Adds DD_CODE_COVERAGE_FLAGS to group and filter uploaded code coverage reports.

• Crashtracking:

  • Unhandled Python exceptions are now reported through the crashtracker, providing crash reports with exception type, message, and Python stack trace in addition to the existing native signal-based crash reporting.

• Stats computation:

  • Enables obfuscation by default. Stats computation itself remains enabled by default only on Python 3.14 and above.

• Profiling:

  • Exception profiling is now enabled by default when the profiler is active on Python 3.12+. Exception profiling samples raised exceptions to help identify error hotspots in your application. Exception profiling is not available on Python < 3.12 and free-threaded builds.
  • Exception profiling now captures the full Python call stack at the point of the raise, instead capturing the exception traceback at the point where the exception is handled. This provides richer context including the complete call chain leading to the exception, consistent with how wall-time and CPU profiles capture stacks.

• openfeature:

  • Adds experimental APM span enrichment for the Feature Flagging and Experimentation (FFE) OpenFeature provider. When DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED=true, the provider attaches feature-flag evaluation metadata (ffe_flags_enc, ffe_subjects_enc, and ffe_runtime_defaults) to the root APM span so traces and errors can be filtered by active flag variant. This is off by default and, when disabled, adds no per-span overhead.
  • The experimental OpenFeature provider now reports aggregated server-side flag evaluations to Datadog (the EVP flagevaluation track), lighting up server-side flag-evaluation observability. The existing feature_flag.evaluations OpenTelemetry metric behavior is unchanged. The new emission is gated by DD_FLAGGING_EVALUATION_COUNTS_ENABLED (default enabled).

• mistralai:

  • Adds support for tracing synchronous and asynchronous streaming chat completion calls using the mistralai client.

• Profiling:

  • Work offloaded by an asyncio task to a ThreadPoolExecutor is now labeled with the originating task.
  • Native frames collected by the profiler are now attributed to third-party code in the code provenance view, rather than appearing as application code.

Bug Fixes

• aiohttp:

  • Fixes a TypeError: 'tuple' object does not support item assignment raised when passing headers as a sequence of (key, value) tuples (aiohttp's LooseHeaders type) to request methods such as get or post.

• aws_durable_execution_sdk_python:

  • This fix resolves an issue where a replayed durable operation that had failed permanently reported an incorrect aws.durable.replayed value and an aws.durable.operation_attempt one higher than its original execution.

• CI Visibility:

  • This fix resolves an issue where the Datadog UI showed No extra information available when viewing details about why a test was skipped by Intelligent Test Runner, due to missing ITR correlation ID on test or suite events.
  • Fixed an issue on Python 3.12+ where enabling coverage report upload (DD_CIVISIBILITY_CODE_COVERAGE_REPORT_UPLOAD_ENABLED=true) alongside Test Impact Analysis caused per-test coverage data to be silently empty. The two features can now be enabled together.
  • Fixes an issue where pytest suites using suite-level Intelligent Test Runner skipping could still skip individual tests when the suite contains an unskippable test marker.

• LLM Observability:

  • This fix resolves a token double-counting issue in the Claude Agent SDK integration. When the Claude Agent SDK split a single model turn into multiple AssistantMessage chunks that share a message_id, the integration emitted one llm span per chunk and counted the turn's token usage multiple times. Chunks sharing a message_id are now merged into a single llm span with the usage counted once. This applies to claude-agent-sdk 0.1.51 and later, which expose message_id; earlier versions retain the previous behavior.
  • Fixes an issue where the LangChain integration failed to submit traced chat model stream spans that completed without yielding any chunks.
  • This fix resolves an issue where the google_adk integration dropped an agent span (logging missing span kind in span context) and orphaned its child spans when a Gemini response part could not be parsed. The span kind is now set before input/output extraction, so a parsing failure degrades to empty input/output instead of dropping the span.
  • This fix resolves an issue where the Google GenAI and google_adk integrations rendered unhandled Gemini response parts (such as inline_data, file_data, empty parts, or thought-signature parts) as a confusing Unsupported file type placeholder. These parts now produce a concise summary or empty content.
  • Fixes LLMObs.submit_evaluation() incorrectly tagging evaluations with source:otel based on the global DD_TRACE_OTEL_ENABLED flag. The tag is now controlled by the is_otel field on the ExportedLLMObsSpan dictionary: True for OTel gen.ai instrumented spans, False for Datadog LLMObs spans. LLMObs.export_span() sets this field automatically.
  • Resolves an issue where OpenRouter requests made through the LiteLLM integration produced no span when the OpenAI integration was also enabled.
  • Resolve an issue where the Pydantic AI integration failed to submit spans with JSON serialization errors when an agent's model_settings contained non-JSON-serializable provider sentinel values.

• Tracing:

  • When APM tracing is disabled, all spans are now tagged as APM-disabled instead of only the local root span, so inferred services are no longer metered as having APM enabled.
  • Fixes an issue in applications that fork where the security agent could read stale process metadata, preventing security signals from being correlated with APM traces.

• flask:

  • Fix span resource remaining as the raw URL path when a worker is killed before start_response runs (e.g. gunicorn --timeout). The route pattern is now populated from request.url_rule during preprocess_request.

• Code Security (IAST):

  • Fixes an issue where taint tracking could abort the Python process when old-style % string formatting handled tainted text containing literal IAST evidence marker delimiters.
  • This fix scopes copy_ranges_from_strings and copy_and_shift_ranges_from_strings to the active request slot, matching the scoped taint read path. Previously these copy helpers resolved the taint map by scanning all request slots, so a concurrent or still-open request could capture the derived taint and the current request would miss the transformed tainted input.
  • A crash occurring when using uvloop has been fixed.
  • A an issue where container values could not go unscanned has been fixed.
  • Resolve a weak-hash false positive reported on ddtrace's own code when IAST and the live debugger (Symbol Database) are both enabled.

• langchain:

  • Fixes an issue where a LangChain-wrapped Google GenAI (Gemini) chat model produced two llm-kind LLM Observability spans for a single call when the langchain and google_genai integrations were both enabled, causing downstream cost and token metrics to be double-counted.

• openfeature:

  • This fix resolves an issue where the feature flagging provider failed to load with openfeature-sdk 0.10.0 or later.

• AAP:

  • Reduces noisy warning logs from exploit prevention (RASP) checks running outside of a request context.

• Fixed an issue that could have caused the internal symbol DB component to restart in nested fork child processes, potentially leading to high memory usage.

• Fix crashes in uwsgi worker when exiting from SIGTERM.

• openai_agents:

  • This fix resolves an issue where the integration stopped capturing the agent manifest on openai-agents >= 0.8.0. The integration now instruments those versions, restoring agent manifest capture.

• profiling:

  • Fixes the stack profiler tearing down an alternate signal stack it did not allocate (for example one installed by the application or the crash tracker), which could disable that component's fault handling.
  • Fixes the stack profiler disabling another component's alternate signal stack at thread teardown when that stack had replaced the profiler's own.
  • Fixes inconsistent handling of DD_PROFILING_MAX_FRAMES by clamping it before stack collection so samples stay within the backend's 600-location limit.

Other Changes

• botocore:
  • Improves the performance of AWS payload tagging by compiling each JSONPath redaction expression once and reusing it, instead of rebuilding the JSONPath parser and re-parsing every redaction path on each instrumented AWS API call. This reduces CPU overhead when DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING or DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING is enabled.