Upgrade Notes
- LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the
v0.4trace API version (v0.5cannot carry the LLMObs span data). SettingDD_TRACE_API_VERSION=v0.5with LLMObs enabled logs a warning and downgrades tov0.4. No user action is required.
New Features
- LLM Observability: LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.
- This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence,
DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=truecan be added to the environment.
Bug Fixes
- LLM Observability: Fixes agentless export dropping data on the
us3,us5,ap1, andap2Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).
- tracing: Fixes a race condition where extra service names could be silently dropped from Remote Configuration
/v0.7/configpayloads in multi-threaded applications (e.g. uWSGI).
- code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.
- LLM Observability: Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.
- IAST: A crash that could happen at interpreter teardown has been fixed.
- protobuf: This fix resolves an issue where Data Streams Monitoring schema tags were missing from protobuf spans due to incorrect patching of generated message classes.