DataDog/dd-trace-py v4.12.0rc1

4.12.0rc1

on GitHub

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Upgrade Notes

• LLM Observability
  • when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.
• ASM
  • This upgrades libddwaf to 2.0.0.

New Features

• AI Guard
  • This introduces the DD_AI_GUARD_OPENAI_ENABLED environment variable (true by default) as a per-provider kill switch. Set it to false to disable AI Guard auto-instrumentation of the OpenAI SDK without affecting other providers or requiring a tracer version rollback.
  • add AI Guard support for LangChain 1.0+ agents built with create_agent.
  • Added opt-in evaluation of streamed OpenAI Chat Completions and Responses via DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED (default off); streams are buffered and evaluated before any chunk is delivered, raising AIGuardAbortError on a block.
  • This introduces AI Guard standalone mode. When APM tracing is disabled (DD_APM_TRACING_ENABLED=false), traces produced by AI Guard are still sent to Datadog, kept with USER_KEEP sampling priority and the AI Guard decision maker so that they can be attributed to AI Guard, while APM host billing is not triggered.
  • Add opt-in buffered stream response evaluation for AI Guard on Anthropic streaming calls. When DD_AI_GUARD_ANALYZE_STREAM_RESPONSES_ENABLED=true (default false), all streamed chunks are buffered, the full assistant response is evaluated by AI Guard, and chunks are replayed to the caller only after an ALLOW verdict. On DENY/ABORT, AIGuardAbortError is raised and zero chunks are delivered. When the flag is off (the default), streaming behavior is unchanged.
• AAP
  • Adds the normalized HTTP route span tag (_dd.appsec.normalized_route) for Tornado, following RFC-1103 and the existing FastAPI, Starlette, Django, and Flask implementations. The tag is emitted on every request span that already carries http.route when the API Security feature is active. Tornado routes using named capturing groups ((?P<name>...)) produce named parameters (e.g. /users/{id}); positional groups produce auto-numbered placeholders (e.g. {param1}). Optional trailing-slash patterns (/?) are treated as not declaring a trailing slash per RFC-1103 rule 1. #18398
• AppSec:
  • Detect OpenAI LLM usage within API endpoints. When AppSec is enabled and an OpenAI chat.completions, completions, or responses call is made during a web request, the WAF address server.business_logic.llm.event is emitted with the provider and model name. This enables WAF rules to tag the trace with appsec.events.llm.call.provider and appsec.events.llm.call.model, surfacing LLM-backed endpoints in the API Endpoint Catalog. Supports both sync and async OpenAI clients, Azure OpenAI (engine kwarg), and streaming. Only the first LLM call per request is recorded (endpoint-level detection, not per-invocation).
• aws_durable_execution_sdk_python
  • Add aws.durable.operation_attempt tag to aws.durable.step and aws.durable.wait_for_condition spans. The tag is 0 for the original attempt and 1, 2, etc. for each subsequent retry.
• ray_serve
  • Adds instrumentation support for ray[serve]>=2.47.1. When Ray is instrumented, ddtrace now traces Serve HTTP and gRPC requests, deployment routing, deployment method execution, and calls made with DeploymentHandle.remote(), with distributed trace context propagated across Serve deployments. See the Ray documentation for more information. #18076
• LLM Observability
  • This introduces support for attaching audio to messages on LLM spans. Input and output messages annotated via LLMObs.annotate now accept an optional audio_parts field, a list of audio segments each with a mime_type and either inline base64 content or an attachment_key. This lets audio (for example, speech-to-text and text-to-speech payloads) be collected and rendered in LLM Observability.
  • Added a sample_rate argument to LLMObs.enable() to configure the proportion of LLMObs traces to sample (between 0.0 and 1.0) in code. This takes precedence over the DD_LLMOBS_SAMPLE_RATE environment variable.
  • Add SyncExperiment.rerun_evaluators() to re-run evaluators on the stored task outputs from a previous experiment run without re-executing the task function. Reads from experiment.result (set by run() or pull()) and returns a new SyncExperiment with fresh evaluations while preserving original span IDs, trace IDs, inputs, and outputs via replay span copies linked back to the originals via parent_experiment_span_id. The missing_task_strategy parameter controls behavior when prior rows contain errors: "raise" (default) raises immediately, "skip" omits the row, and "retry" re-executes the task for that row.
  • LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.
  • The OpenAI integration now captures input and output audio on audio chat completions (any audio-capable Chat Completions model, e.g. gpt-audio, gpt-audio-mini, gpt-4o-audio-preview) as audio_parts on LLM span messages. Note: audio in streamed responses (stream=True) is not captured yet.
  • Adds support for DD_LLMOBS_SAMPLE_RATE, which controls the proportion of LLM Observability spans (between 0.0 and 1.0, defaults to 1.0) that are retained. This does not affect APM span retention or the accuracy of LLM Observability metrics such as token usage, and cost.
  • Adds span metadata to the data exposed to the span processor registered with LLMObs.enable(span_processor=...) or LLMObs.register_processor(...). Previously only input and output were exposed; the processor can now read and modify metadata to redact or remove sensitive values that integrations record there.
  • Adds support for submitting LLM Observability spans for mistralai client chat and embedding calls.
• profiling
  • implement heap live count data collection
  • The adaptive sampling mechanism has been revamped to better match the needs of real-life workloads.
  • The PyTorch profiler now properly shows nested frames in flame graphs.
• mistralai
  • Adds tracing support for the mistralai client chat and embedding calls.
• pytorch
  • Adds a pytorch.rank lifetime span for PyTorch distributed training. The span opens at init_process_group and closes at destroy_process_group or process exit. Tags include rank, world_size, framework (DDP / FSDP / DeepSpeed), launcher, torch.distributed.backend, and training_job_id (resolved from launcher environment variables). When running under Ray Train, ray.train.run_name, ray.submission_id, and ray.metadata.* are also applied. Enable with DD_PATCH_MODULES=pytorch:true.
• This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence, DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.
• CI Visibility
  • The DD_TEST_MANAGEMENT_ATTEMPT_TO_FIX_RETRIES environment variable is now respected by the ddtrace.testing internal pytest plugin. When set to a valid integer, it overrides the retry count returned by the Datadog API for "attempt to fix" tests.

Bug Fixes

• LLM Observability:
  • Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).
  • Resolves an issue in the openai integration where streamed chat completion spans under-reported output_tokens and total_tokens for OpenAI-compatible providers that emit a cumulative usage object on every streamed chunk.
  • Fixes an issue where the span writer ignored its configured request timeout (_DD_LLMOBS_WRITER_TIMEOUT, default 5 seconds) and instead used the 2 second connection default, causing intermittent TimeoutError and dropped span events on high-latency connections to the agent or intake.
  • This fix resolves an issue where the OpenAI and Anthropic integrations did not record the model output on the LLM Observability span when AI Guard blocked the request after the model call completed. The model response is now captured even though the span is marked as errored by the block.
  • Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.
  • Resolves an issue where Azure OpenAI calls through the LiteLLM integration tagged model_name with the request deployment name (e.g. my-deployment) instead of the canonical model returned by the API (e.g. gpt-4o-2024-08-06). The integration now prefers the response model for the azure and azure_text providers.
  • Resolves an issue where spans annotated with metadata containing non-string keys (e.g. int/float/bool) could be dropped during ingestion. Metadata keys are now stringified before encoding.
  • Resolves an issue where the OpenAI integration recorded unset request parameters (OpenAI SDK's Omit/NotGiven sentinel values) as noise in LLM span metadata.
  • Resolve an issue where tool results parsed from a streamed OpenAI chat completion could be attributed to the preceding input message instead of the streamed output.
  • Resolves an issue where non-string tag values passed to LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.
• AI Guard:
  • This fix resolves an issue where the Anthropic integration dropped document content blocks, which could cause evaluation to be skipped for prompts whose only content was a document. Document text is now scanned and binary documents leave a placeholder so evaluation still runs.
  • This fix resolves an issue where the LiteLLM proxy guardrail allowed a per-request block dynamic parameter, supplied in the incoming request body, to disable the operator-configured blocking policy and let a denied request proceed in monitor mode. Request-supplied parameters can now only strengthen blocking (turning monitor mode into blocking); they can no longer weaken an enabled blocking policy.
  • This fix resolves an issue where the Strands integration silently dropped non-text Bedrock content (images, documents, video), which could cause evaluation to be skipped for prompts or tool results with no text.
• langchain
  • Fix AI Guard blocked requests (DENY/ABORT) not appearing in APM and LLM Observability.
• aiokafka
  • Fixes kafka.partition and kafka.message_offset span tags being incorrectly set to -1. Consumer spans no longer misreport 0 (e.g. single-partition topics or the first message in a partition) due to falsy evaluation. Producer spans and any span where the value is genuinely unknown now omit these tags entirely rather than emitting a misleading -1.
• anthropic
  • Fixes LLM Observability spans reporting model_provider as unknown for the AnthropicVertex and AnthropicBedrock clients. The provider is now resolved to google and amazon respectively.
• AAP
  • Resolve an issue where some request bodies could be skipped during analysis depending on the Content-Type header for the Django, Tornado, Flask, and FastAPI integrations.
  • This fix resolves an issue where the target URL of a redirect followed internally by urllib3 was not inspected for SSRF and API Security (API10).
• ASM
  • This fix resolves an issue where user blocking could be bypassed when calling ddtrace.contrib.trace_utils.set_user if neither ddtrace.appsec.trace_utils nor the user tracking SDK had been imported. The user blocking decision is now always enforced when AppSec is enabled.
  • This fix resolves an issue where bytes commands passed to os.system, os.spawn*, and subprocess (for example os.system(b"...") or subprocess.run(b"...", shell=True)) bypassed the Exploit Prevention (RASP) shell and command injection checks. bytes commands are valid on POSIX systems and are now passed to the security checks like str commands.
  • This fix resolves an issue where the Django session id collected by automatic user activity tracking was not hashed in anonymization mode. It is now hashed, consistently with the user id and login.
• aws_durable_execution_sdk_python
  • This fix resolves an issue where the aws.durable.operation_attempt span tag was reported off by one for replayed durable operations, causing a resumed (replayed) execution to record a different attempt number than the original execution for the same operation. Replayed operations now report the same 0-indexed attempt number as a fresh execution.
• CI visibility
  • Fixes a ValueError: tool 1 is already in use error that occurred on Python 3.12+ when running pytest with coverage.py or pytest-cov alongside ddtrace. Both tools previously competed for sys.monitoring.COVERAGE_ID (slot 1). ddtrace now tries tool slots in priority order (4, 3, 1), falling back gracefully if all candidate slots are occupied.
  • Fixes a ResourceWarning: unclosed socket that occurred when using the pytest plugin. The background writer thread's thread-local HTTP connection was not closed on thread exit.
• database monitoring
  • Resolved a memory leak that grew with query volume when DD_DBM_PROPAGATION_MODE was set to full, service, or dynamic_service.
• elasticsearch
  • Fix zombie spans on TransportError. When a request failed, the generator coroutine holding the open span was not explicitly closed, leaving the span open until Python GC ran — producing spans with grossly inflated durations on non-CPython runtimes (e.g. PyPy). Both the sync and async wrappers now call coro.close() immediately on the error path. Affects Elasticsearch and OpenSearch integrations.
• tracing
  • Resolves an issue where wrapping an async generator on Python 3.11 through 3.14 raises a TypeError: object NoneType can't be used in 'await' expression error. This occurs when the generator body awaits a coroutine that suspends to the event loop before its first yield. Python 3.9 and 3.10 are not affected.
  • This fix resolves an issue where _dd.svc_src was incorrectly set in some cases.
  • This fix resolves an issue where traces buffered before os.fork() could be sent twice, once by the parent and once by the child.
  • Fixes a race condition where extra service names could be silently dropped from Remote Configuration /v0.7/config payloads in multi-threaded applications (e.g. uWSGI).
  • Applying @tracer.wrap() to an async generator now forwards sent values, thrown exceptions, and close requests to the underlying generator, so it behaves like the unwrapped generator in all cases. Previously the wrapper only relayed values during forward iteration, so sent values were dropped and try/finally cleanup was skipped whenever the generator was closed early or received a thrown exception.
  • Fixes a TypeError: cannot pickle '_thread.lock' raised when serializing ddtrace-instrumented objects with cloudpickle. Fork-safe locks and events (ddtrace.internal.forksafe.Lock / Event) are now pickle-safe, and unpickling reconstructs a fresh instance.
  • A rare crash happening on versions of CPython prior to 3.12 has been fixed.
  • Fixes network.client.ip to contain the actual TCP peer IP address (i.e. REMOTE_ADDR) rather than the resolved HTTP client IP (which could originate from forwarding headers such as X-Forwarded-For). This affects the Flask, Django, ASGI (FastAPI/Starlette), and Tornado integrations. The resolved HTTP client IP continues to be reported under http.client_ip, which is the value used by App and API Protection (AAP) for IP-based threat detection and blocking.
• botocore
  • Fixes a bug in Data Streams Monitoring where the pathway context of an SQS message was not extracted when the message body was valid JSON, silently breaking the data stream for non-SNS messages with a JSON payload. The _datadog message attribute is now read from the top-level SQS MessageAttributes first, and the message body is only parsed for SNS-to-SQS notifications.
  • Kinesis PutRecords now propagates trace context to each record in the batch. This enables trace context propagation to correctly map to each individual record, as well as enabling the correct tracking of messages through Data Streams Monitoring. Previously, only the first record would ever have been tracked.
  • aiobotocore: Resolves an issue where outbound AWS SDK requests to strict-signature endpoints (e.g. AppSync, API Gateway with IAM authorization) were rejected with The request signature we calculated does not match the signature you provided when distributed tracing was enabled.
• celery
  • Fixes Celery worker and producer spans ignoring DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED, causing spans to always use celery-worker/celery-producer instead of the configured global service name.
• Django
  • This fix resolves an issue where async function-based middleware attributes, such as process_exception, were not preserved, causing Django to skip exception handlers under ASGI.
• fastapi
  • Fixes span resource names for requests on nested routers being set to only the leaf path segment instead of the full composed path. Affects FastAPI >= 0.137.
  • Fixes span resource names being duplicated when both the FastAPI and Starlette patches are active. Affects FastAPI >= 0.137.
• grpc
  • Fix streaming grpc.aio client spans intermittently reporting error.message as "Internal error from Core" instead of the application-set abort details when a server aborts the RPC. #18015
• Code Security (IAST)
  • This fix resolves an issue where instrumenting re.finditer eagerly consumed the entire match iterator to taint every match, which removed the laziness of re.finditer and allowed a request-controlled input with many matches to allocate excess memory. Matches are now tainted lazily as the application iterates over them.
  • Fixed an issue where sensitive content could be left unredacted in vulnerability reports when a sensitive value overlapped a tainted range.
  • A crash that could happen at interpreter teardown has been fixed.
• litellm
  • Fix AttributeError when metadata=None is passed to traced litellm.completion() calls. #18033
• sqlalchemy
  • Fixes duplicate SQLAlchemy event listeners when trace_engine() is called repeatedly for the same engine.
• code origin
  • fix deepcopying of view functions performed by frameworks like cadwyn, when code origin is enabled.
  • fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.
• core
  • This fix prevents periodic thread restarts from blocking application code from resuming in forked children.
• google_genai
  • Fix TypeError when Google GenAI returns None for optional response fields (token counts or candidates) introduced in google-genai>=2.6.0.
• kafka
  • Fixes an issue where Data Streams Monitoring would incorrectly record consume checkpoints for error messages.
• otel
  • Resolves an issue where OpenTelemetry OTLP metrics export requests to the collector were incorrectly traced as spans.
• profiling
  • Reduces native heap memory used by the stack collector in gevent and asyncio workloads.
  • Rare crashes that could happen post-fork in fork-based applications have been fixed.
• protobuf
  • This fix resolves an issue where Data Streams Monitoring schema tags were missing from protobuf spans due to incorrect patching of generated message classes.
• redis
  • Fixes missing out.host and server.address tags on spans produced by redis.cluster.RedisCluster and redis.asyncio.cluster.RedisCluster (redis-py >=4.1). These clients do not expose a connection_pool attribute, so connection metadata was never extracted and the Datadog inferred-entity feature fell back to the generic peer.db.system: redis identifier instead of resolving to the specific cluster hostname. Spans now include out.host and server.address derived from the cluster's startup node, enabling the service map to display a host-specific Redis entity.
• remoteconfig
  • Fixed a configuration being reported as acknowledged before the product received it. The state is now set to acknowledged only after the configuration has been delivered to the product, and to error only when the payload is malformed and cannot be deserialized.
• starlette
  • Fix high-cardinality span resources for CORS preflight requests. When CORSMiddleware handled an OPTIONS request, the span resource was set to the raw URL path (e.g. OPTIONS /users/123) instead of the route template (e.g. OPTIONS /users/{user_id}). The route pattern is now resolved via the app's route tree when CORSMiddleware short-circuits before the Starlette router runs. Fixes #14321.

Other Changes

• profiling
  • Reduces allocator churn in the native stack collector by reusing the cycle-detection set across stack-unwind calls.
  • Reduces memory and allocation overhead for applications with many default-named asyncio tasks or gevent greenlets by avoiding storing generated task and greenlet names in the profiler's internal string table. #18437