Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
• [ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
• [BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)