Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.
Upgrade Notes
- claude_agent_sdk: Tool span resource names have changed from the tool name (e.g.
Read,Bash) toclaude_agent_sdk.tool. The specific tool name is still available in the span name (e.g.claude_agent_sdk.tool.Read). Users relying on tool resource names should update them accordingly.
- ray: Adds
DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED(default:False) configuration to control Ray submission tracing. SetDD_TRACE_RAY_SUBMISSION_SPANS_ENABLED=trueto tracetask.submitandactor_method.submitspans. Leave it unset to trace only execution spans. See Ray integration documentation for more details.
New Features
- AI Guard: Add
DD_AI_GUARD_BLOCKenvironment variable. Defaults toTrue, which means the blocking behavior configured in the Datadog AI Guard UI (in-app) will be honored. Set toFalseto force monitor-only mode locally: evaluations are still performed butAIGuardAbortErroris never raised, regardless of the in-app blocking setting.
- AI Guard response objects now include a dict field
tag_probswith the probabilities for each tag.
- CI Visibility: Adds Bazel offline execution support with two modes: manifest mode (
DD_TEST_OPTIMIZATION_MANIFEST_FILE), which reads settings and test data from pre-fetched cache files without network access; and payload-files mode (DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES), which writes test event, coverage, and telemetry payloads as JSON files instead of sending HTTP requests. Both modes can be used independently or together.
- LLM Observability: Captures individual LLM spans for each Claude model turn within a Claude Agent SDK session. Each LLM span captures the input messages, output messages, model name, and token usage metrics (for claude_agent_sdk >= 0.1.49).
- AAP: This adds Application Security support for FastAPI and Starlette applications using mounted sub-applications (via
app.mount()). WAF evaluation, path parameter extraction, API endpoint discovery, andhttp.routereporting now correctly account for mount prefixes in sub-application routing.
- google_cloud_pubsub: This adds tracing for Google Cloud Pub/Sub admin operations on topic, subscription, snapshot, and schema management methods.
- google_cloud_pubsub: Adds support for Google Cloud Pub/Sub push subscriptions. When a push subscription delivers a message via HTTP, the integration now creates an inferred
gcp.pubsub.receivespan that captures subscription and message metadata. UseDD_GOOGLE_CLOUD_PUBSUB_PROPAGATION_AS_SPAN_LINKSto control whether the inferred span becomes a child of the producer trace or starts a new trace with the producer context attached as a span link (default:False).
- LLM Observability: Add
ExperimentRun.as_dataframe()to convert experiment run results into apandas.DataFramewith a two-level MultiIndex on columns. Each top-level group (input,output,expected_output,evaluations,metadata,error,span_id,trace_id) maps to the first index level. Dict-valued fields are flattened one level deep; scalar fields use an empty string as the sub-column name. Each evaluator gets its own column containing the full evaluation result dict. Requirespandasto be installed (pip install pandas).
- LLM Observability: Adds an
eval_scopeparameter toLLMObs.submit_evaluation()(one of"span"(default) or"trace"). Useeval_scope="trace"to associate an evaluation with an entire trace by passing the root span context.
- LLM Observability: Adds
LLMObs.get_spans()to retrieve span events from the Datadog platform API (GET /api/v2/llm-obs/v1/spans/events). Supports filtering by trace ID, span ID, span kind, span name, ML app, tags, and time range. Results are auto-paginated. RequiresDD_API_KEYandDD_APP_KEY.
- profiling: Profiles generated from fork-based servers now include a
process_typetag with the valuemainorworker.
- tracing: Support for making the default span name for
@tracer.wrapinclude the class name has been added. For now, this is opt-in and can be enabled by settingDD_TRACE_WRAP_SPAN_NAME_INCLUDE_CLASS=true. The new naming will become the default in the next major release.
Bug Fixes
- CI visibility: This fix resolves issues where CI provider metadata could omit pull request base branch and head commit details or report incorrect pull request values for some providers.
- AAP: This fix resolves an issue where Application and API Protection (AAP) was incorrectly reported as an enabled product in internal telemetry for all services by default. Previously, registering remote configuration listeners caused AAP to be reported as activated even when it was not actually enabled. This had no impact on customers as it only affected internal telemetry data. AAP is now only reported as activated when it is explicitly enabled or enabled through remote configuration.
- asgi: Fixed an issue caused network.client.ip and http.client_ip span tags being missing for FastAPI.
- iast: A crash has been fixed.
- lambda: Fixes a spurious
Unable to create shared memorywarning on every AWS Lambda cold start.
- LLM Observability: Fixes an issue where an APM_TRACING remote configuration payload that did not include an
llmobssection would disable LLM Observability on services where it had been enabled programmatically viaLLMObs.enable(). Services that enabled LLM Observability via theDD_LLMOBS_ENABLEDenvironment variable were unaffected. The handler now only changes LLM Observability state when the remote configuration payload explicitly carries anllmobs.enableddirective.
- LLM Observability: Fixes a circular import in
ddtrace.llmobs._writerwhenanthropic,openai, andbotocoreis installed.
- Prevent potential crashes when the client library fails to restart a worker thread due to hitting a system resource limit.
- internal: This fix resolves an issue where reading unknown attributes from
ddtrace.internal.process_tagscaused aKeyErrorinstead of raising anAttributeError.
- rq: Fixes compatibility with RQ 2.0. Replaces the removed
Job.get_id()with thejob.idproperty, and handlesJob.get_status()now raisingInvalidJobOperationfor expired jobs (e.g.result_ttl=0) instead of returningNone. #16682
- tornado: Fixes an issue where routes inside a nested Tornado application were matched in reverse declaration order, causing a catch-all pattern to win over a more-specific route defined before it. This resulted in incorrect
http.routetags on spans.
- tornado: The
http.routetag is now populated for routes whose regex cannot be reversed by Tornado (e.g. patterns containing non-capturing groups such as(?:a|b)). Capturing groups are still rendered as%s, consistent with Tornado's own route format, while non-capturing constructs are kept verbatim.
- telemetry: This fix resolves an issue where unhandled exceptions raised by
importlib.metadataduring interpreter shutdown (for example, when Gunicorn workers exit uncleanly after a failed startup) causedupdate_imported_dependenciesto surface errors throughsys.excepthook. Failures while discovering dependencies for theapp-dependencies-loadedtelemetry payload are now logged at debug level and swallowed so they no longer propagate out of the dependency-reporting path.
- profiling: Fixes noise caused by the profiler attempting to load its native module even when profiling was disabled,
- profiling: A race condition which could make asyncio code raise exceptions at exit has been fixed.
- remote_config: This fix resolves an issue where brief Datadog Agent connection errors could drop Remote Configuration polls, causing products such as Dynamic Instrumentation to temporarily appear disabled.
- LLM Observability: Change the default
model_providerandmodel_nameto"unknown"from"custom"when a model did not match any known provider prefix in the Google GenAI, VertexAI, and Google ADK integrations.
- LLM Observability: This fix resolves tracing issues for pydantic-ai >= 1.63.0 where tool spans and agent instructions were not being properly captured. This fix adds tracing to the
ToolManager.execute_tool_callmethod for newer versions of the library to resolve this issue.
Other Changes
- remote config: Removes noisy warning log that was being emitted when an unsupported agent config payload was received.
- ASM: Update default security rules to 1.18.0. Notably, this adds business logic event coverage for Stripe auto-instrumentation and expands WAF rule coverage (ZipSlip detection, file upload with double extension, broader header scanning, and expanded XXE detection).