DataDog/dd-trace-py v4.9.0

4.9.0

on GitHub

Upgrade Notes

• AI Guard: ddtrace.appsec.ai_guard.AIGuardAbortError now derives from ddtrace.internal._exceptions.DDBlockException (a BaseException subclass) instead of Exception. This brings AI Guard block decisions in line with how ASM blocks are surfaced and prevents a generic except Exception: in user code from silently swallowing a block.

• settings: Legacy environment variable names registered as aliases in the configuration registry now also work when set via local or fleet stable config files, not just shell environment variables. #17958

• ray: Adds DD_TRACE_RAY_IGNORED_ACTORS configuration to exclude specific Ray actor methods from instrumentation. Set DD_TRACE_RAY_IGNORED_ACTORS='{"ActorA": ["method1"], "ActorB": "*"}' to leave matching methods or actors uninstrumented while continuing to trace other Ray actor methods. Matching is based on actor class name only.

Deprecation Notes

• Tracing: DD_TRACE_INFERRED_SPANS_ENABLED is deprecated and will be removed in 5.0.0. Use DD_TRACE_INFERRED_PROXY_SERVICES_ENABLED instead. The old environment variable continues to work but emits a DDTraceDeprecationWarning when set.

New Features

• aws_durable_execution_sdk_python: Add tracing support for the aws-durable-execution-sdk-python library. Instruments @durable_execution workflows and DurableContext operations (step, invoke, wait, wait_for_condition, wait_for_callback, create_callback, map, parallel, run_in_child_context) to generate spans.

• LLM Observability: Adds step spans to the Claude Agent SDK integration. Each inference cycle is now represented by a step container span with an llm child span for the model call and tool child spans for any tool invocations.

• tracing: Adds a centralized supported-configurations.json registry of all supported DD_* and OTEL_* environment variables, following the same schema used by other Datadog tracing libraries. Accesses to unregistered environment variables now produce a debug log to help identify typos or unsupported configuration options.

• AI Guard: Copies anomaly-detection attributes from the local root (service-entry) span onto every ai_guard span: ai_guard.http.useragent, ai_guard.http.client_ip, ai_guard.network.client.ip, ai_guard.usr.id and ai_guard.usr.session_id.

• AI Guard: When DD_AI_GUARD_ENABLED=true is set and an ai_guard span is created during a request, the tracer now populates http.client_ip and network.client.ip on the service-entry (local root) span, mirroring the behavior used for Application Security. If AI Guard does not run during the request, no client IP tags are added. DD_TRACE_CLIENT_IP_ENABLED is ignored once AI Guard reports, and DD_TRACE_CLIENT_IP_HEADER continues to override header resolution.

• ai_guard: add AI Guard evaluation support to the OpenAI SDK chat completions instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client, and evaluation is automatically skipped when a framework integration (LangChain, Strands Agents) is already evaluating the same call.

• code origin for spans: The code origin for spans feature has been enabled by default.

• code origin: attach code origin information to the first span generated by a function wrapped with tracer.wrap.

• openfeature: This introduces a configurable initialization timeout for DataDogProvider. The timeout controls how long initialize() waits for configuration before returning, and defaults to 10 seconds. Set it via the DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS environment variable or the init_timeout constructor parameter.

• CI Visibility: This introduces Jenkins custom parent ID propagation, which enables Datadog to correlate tests run from Jenkins with their Jenkins jobs and pipelines.

• LLM Observability: Adds an optional cost_tags argument to LLMObs.annotate() and LLMObs.annotation_context(). Pass a list of tag keys (already set via tags or annotated previously on the same span) to have them attached to the cost and token metrics generated from LLM and embedding spans, which can help breaking down spend by team, project, org, or any custom dimension.

• LLM Observability: Adds support for an optional version (string) field on each tool definition dictionary passed to LLMObs.annotate() via the tool_definitions parameter.

• profiling: Add DD_PROFILING_LOCK_EXCLUDE_MODULES config to skip lock profiling for framework-internal locks. Excluded locks remain native with zero profiling overhead. Set it to a comma-separated list of module prefixes (e.g., django.db,sqlalchemy.pool,urllib3).

• LLM Observability: Bedrock Agent orchestration step events (model invocations, tool/action group calls, knowledge base lookups, guardrails, rationales) are now emitted as APM child spans of the Bedrock Agent <agent_id> span when LLM Observability is enabled, with the same LLMObs payload shape as before.

Bug Fixes

• tracing: Exclude wrapt==2.2.0 from the supported dependency range to avoid a regression that breaks wrapped C descriptors.

• ai_guard: This fix resolves a conflict between ddtrace.auto and strands when imported in the same file, which left Strands hooks silently disabled. The Strands integration now loads lazily on first attribute access so its event class identities match those the agent dispatches.

• appsec: Adds telemetry metrics instrum.user_auth.missing_user_login and instrum.user_auth.missing_user_id when Django auth events cannot resolve the expected identity fields, enabling detection of misconfigured user model field mappings.

• AAP: This fix resolves an issue where the usr.session_id tag was missing from the entry span of authenticated follow-up Django requests when automatic user instrumentation was enabled. They now also carry usr.session_id, matching other authenticated user-tagging paths.

• azure_cosmos: This change removes the http.status_code tags from Azure CosmosDB spans and replaces them with the use of the db.response.status_code metric. For customers using ddtrace v4.8.0 and relying on the http.status_code tag of cosmosdb.query spans, this is a breaking change.

• CI Visibility: Fixes an issue in the pytest plugin where a malformed log call emitted a --- Logging error --- traceback to stderr during Attempt to Fix retries, polluting pytest output and contributing to spurious test failures.

• CI Visibility: Fixes an IndexError in retry bookkeeping that occurred when a test's teardown phase failed. The error produced --- Logging error --- tracebacks in stderr, which could pollute test output and cause spurious test failures during retries. #17863

• CI Visibility: Fixes a regression where setting DD_TEST_MANAGEMENT_ENABLED=0 was not honored by the new pytest plugin, causing Test Management features such as quarantining, disabling tests, and Attempt to Fix to remain enabled.

• CI Visibility: Fixes code coverage instrumentation on Python 3.13, 3.14, and 3.15. Resolves lost per-test line data caused by: sys.monitoring callbacks running in a snapshot context where ContextVar changes are not visible (Python 3.14+); empty modules emitting no LINE events (Python 3.13+); and ProcessPoolExecutor child coverage not being propagated to the parent context. Also fixes a stale-data bug where child process executable lines could inflate coverage denominators after stop_coverage() was called before join().

• datastreams: Demotes the retry limit exceeded submitting pathway stats log from ERROR to WARNING and removes the multi-line traceback from the record. This message fires when the processor cannot reach the agent within its 1-second timeout; the dropped 10 seconds of DSM data is auto-recovered on the next flush.

• LLM Observability: Fixes a concurrency bug in the Bedrock Agent integration where concurrent invoke_agent calls could orphan or cross-attribute spans due to shared class-level state. Per-invocation state is now used.

• LLM Observability: This fix resolves an issue where text wrapped in Bedrock Converse guardContent content blocks was rendered as [Unsupported content type: guardContent] in traces, dropping the user's input.

• Fixed an issue that could have caused some instrumented code to fail to execute correctly when the original function had keyword arguments passed in as a cell variable.

• CI Visibility: Fixes an issue where tests marked as attempt-to-fix could have failures hidden when they were also quarantined or disabled.

• django: Stop tagging async view and middleware spans as errored on routine ASGI cancellations (e.g. client disconnects on streaming responses), a regression introduced in 4.8.0rc4. Cancellation still propagates; the span just finishes without error.type='asyncio.exceptions.CancelledError'.

• django: Fixes DD_DJANGO_DATABASE_SERVICE and DD_DJANGO_DATABASE_SERVICE_NAME, which were previously generated as DD_DJANGO-DATABASE_SERVICE and DD_DJANGO-DATABASE_SERVICE_NAME. The hyphenated names were invalid POSIX identifiers and unusable from most shells. Hyphens in integration names are now normalized to underscores when building env var names. The old hyphenated names are preserved as aliases for backward compatibility. #17952

• django: API endpoint discovery now supports Django sub-applications mounted with django.urls.include(...). Endpoints are reported with their full URL path including the parent prefix — for example, a view served at /api/users/ is now reported as /api/users/ instead of losing the /api/ prefix.

• django: API endpoint discovery now reports the correct HTTP methods for views decorated with @require_http_methods combined with another decorator such as @csrf_exempt; the declared methods are reported instead of a generic wildcard entry.

• telemetry: tolerate malformed installed distribution metadata so a single bad dist-info entry no longer floods stderr with repeated tracebacks.

• langchain, botocore: This fix resolves an issue where auto-instrumented langchain_aws.ChatBedrockConverse spans reported an opaque inference-profile ARN identifier as the model name when an inference profile was used. base_model_id which represents the underlying foundation model is now checked first when extracting model names, and the botocore Bedrock integration reads the resolved base model from a shared in-process cache populated by langchain so the same resolution applies to the underlying bedrock-runtime span.

• LLM Observability: This fix resolves an issue where running an experiment with a dataset whose records had null metadata caused the summary evaluator to crash with a TypeError while preparing evaluator inputs.

• LLM Observability: Changes the default model_name and model_provider of LLM and embedding spans from custom to unknown if not provided or empty. This applies to both auto-instrumented spans and manual instrumentation via LLMObs.llm() / LLMObs.embedding() and the @llm / @embedding decorators.

• profiling: Fixes an issue where the lock profiler silently stopped capturing lock events when running under ddtrace-run with gevent installed.

• LLM Observability: The OpenAI integration now preserves assistant message content when tool_calls are present on the same message. #17760

• openfeature: This fix resolves an issue where DataDogProvider.initialize() returned before configuration was received, causing the OpenFeature SDK to mark the provider as ready to serve evaluations too early and flag evaluations to silently return default values. The provider now waits for configuration before returning.

• openfeature: Fixes targeting key handling in the OpenFeature provider. None targeting key is now correctly passed to the native evaluator instead of being coerced to empty string. Flags that don't require a targeting key (static, rule-based) now evaluate successfully without one, matching the Datadog provider spec. Additionally, the Rust binding now correctly maps TargetingKeyMissing errors from libdatadog instead of returning a generic error code.

• tracing: Fixes an issue where the svc.auto process tag produced garbled values such as python_-m_unittest when a process was launched with the full command as a single sys.argv[0] string (e.g. from a Docker ENTRYPOINT, a process manager, or a subprocess call with an unsplit command). The correct module or script name is now extracted in these cases. #17764

• profiling: A crash that could happen in child processes after fork has been fixed.

• profiling: A rare crash caused by the memory allocation profiler has been fixed.

• RemoteConfig: Fixed an issue where deleted remote configurations were not applied, causing stale settings to persist.

• tracing: This fix resolves a memory leak where reference cycles through a span's properties were invisible to Python's cyclic garbage collector and accumulated proportionally to traced call volume.

• starlette: This fix resolves an issue where passing middleware=None caused application startup to fail when Starlette tracing was enabled.

• wsgi: This fix resolves an issue where the http.url tag on inbound request spans contained the WSGI mount prefix twice (for example /admin/admin/users instead of /admin/users) when the application was served behind werkzeug.middleware.dispatcher.DispatcherMiddleware or any other in-process mount that preserves the original request URI in RAW_URI / REQUEST_URI while also setting SCRIPT_NAME.

• tracing: A crash that occurred when exiting a gevent application with DD_TRACE_DEBUG=1 has been fixed.

• langchain: Strips interface identifiers (e.g. chat, llm) and path prefixes (e.g. models/) when extracting the model_provider and model_name, so reported values identify the actual provider and model name rather than the LangChain interface or API resource path.

• llmobs: fixes child spans created within an experiment task not inheriting the dataset_id tag. Previously only dataset_name was propagated via baggage to child spans; dataset_id is now propagated as well, making dataset, project, and experiment context (name and ID) consistent across all spans in an experiment trace.

• Profiling: This fixes a bug where uploaded profiles would not have a linked span post fork

• profiling: A rare crash happening on systems with small stack sizes when profiling asyncio code has been fixed.

• propagation: Limits parsing of the W3C tracestate header during tracecontext extraction to 32 list-members and 512 UTF-8 bytes, consistent with the W3C Trace Context specification (https://www.w3.org/TR/trace-context/). Extra list-members and trailing whole entries that would exceed the byte budget are ignored, so unusually large headers no longer expand unbounded work during extraction. The Datadog dd= list-member is preferred: it is kept when present (including when it appears late in the header or alone exceeds the byte cap), and other vendors are dropped first. List-members longer than DD_TRACE_TRACESTATE_ITEM_MAX_CHARS (128) characters are removed first when trimming by list-member count or byte budget, so shorter vendor entries are kept when possible.

• Fixed a startup deadlock when using snowflake-connector-python >= 4.4.0 with DD_TRACE_SNOWFLAKE_ENABLED=true.

• tracing: This change fixes an issue in which svc_src is set to m in cases where service matches the _default_service of an active integration config. In such cases, the intended behavior is that it svc_src is equal to service. #17712

• tracing: Parsing incoming baggage HTTP headers now respects DD_TRACE_BAGGAGE_MAX_ITEMS [default 64] and DD_TRACE_BAGGAGE_MAX_BYTES [default 8192], consistent with baggage injection. Previously, extraction could retain every comma-separated entry regardless of those limits. The tracer drops excess pairs and records truncation telemetry when limits apply.

Other Changes

• profiling: The ECHION_ALT_VM_READ_FORCE configuration flag has been removed and support for the associated feature has been dropped.