Breaking Changes

• [CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
• [CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1 #4893 (@javiermolinar)
  The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver
• [CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
• [CHANGE] BREAKING CHANGE Remove tempo serverless #4599 (@electron0zero)
  Following config options are no longer valid, please remove them if you are using these in your tempo config:

  querier:
      search:
          prefer_self: <int>
          external_hedge_requests_at: <duration>
          external_hedge_requests_up_to:  <duration>
          external_backend: <string>
          google_cloud_run: <string>
          external_endpoints: <array>
  

  Tempo serverless related metric tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total and tempo_feature_enabled are being removed.
• [CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant.
  Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
• [CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)

Changes

• [CHANGE] Update query range error message #4929 (@joey-grafana)
• [CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
• [CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
• [CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
• [CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
• [CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
• [CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
• [CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
• [CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
• [CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
• [CHANGE] Update to go 1.24.3 #5110 (@stoewer)
• [CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
• [CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)

Features

• [FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
• [FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
• [FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
• [FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)

Enhancements

• [ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. #4668 (@carles-grafana)
  configurable via the throughput_bytes_slo field, and it will populate op="traces" label in slo and throughput metrics.
• [ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
• [ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
• [ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
• [ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
• [ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
• [ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
• [ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
• [ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
• [ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
• [ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
• [ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
• [ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
• [ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
• [ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
• [ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
• [ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
• [ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
• [ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
• [ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
• [ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
• [ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
• [ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
• [ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
• [ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
• [ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
• [ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
• [ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
• [ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
• [ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
• [ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
• [ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
• [ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
• [ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
• [ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
• [ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers #5150 (@electron0zero)

Bugfixes

• [BUGFIX] Choose a default step for a gRPC streaming query range request if none is provided. #4546 (@joe-elliott)
  Correctly copy exemplars for metrics like | rate() when gRPC streaming.
• [BUGFIX] Make comparison to nil symmetric #4869 (@stoewer)
• [BUGFIX] Fix behavior for queries like {.foo && true} and {.foo || false} #4855 (@stoewer)
• [BUGFIX] Fix performance bottleneck and file cleanup in block builder #4550 (@mdisibio)
• [BUGFIX] Add object name to cache key in ReadRange #4982 (@joe-elliott)
• [BUGFIX] TraceQL incorrect results for additional spanset filters after a select operation #4600 (@mdisibio)
• [BUGFIX] TraceQL metrics incorrect results for queries with multiple filters that reside in non-dedicated columns that also group by the same variable #4887 (@mdisibio)
• [BUGFIX] TraceQL results caching bug for floats ending in .0 #4539 (@carles-grafana)
• [BUGFIX] Rhythm: fix sorting order for partition consumption #4747 (@javiermolinar)
• [BUGFIX] Rhythm: fix block builder to not reuse a block ID if it was already flushed, to prevent read errors #4872 (@mdisibio)
• [BUGFIX] Rhythm: fix ingester to still clear old blocks when not flushing to object storage #5005 (@mdisibio)
• [BUGFIX] Fix rare panic during compaction. #4915 (@joe-elliott)
• [BUGFIX] Fix metrics streaming for all non-trivial metrics #4624 (@joe-elliott)
• [BUGFIX] Fix starting consuming log #4539 (@javiermolinar)
• [BUGFIX] TraceQL: Fix behavior of {} >> { span.attr-that-doesnt-exist != "foo" } #5007 (@joe-elliott)
• [BUGFIX] Rhythm - fix adjustment of the start and end range for livetraces blocks #4746 (@javiermolinar)
• [BUGFIX] Return the operand as the only value if the tag is already filtered in the query #4673 (@mapno)
• [BUGFIX] Fix flaky test #4787 #4995 (@javiermolinar)
• [BUGFIX] Fix flaky ingester test #4846 (@javiermolinar)
• [BUGFIX] Include cost attribution when converting from default config to legacy one #4787 (@javiermolinar)
• [BUGFIX] Fix memcached settings for docker compose example #4346 (@ruslan-mikhailov)
• [BUGFIX] Update memcached to respect cancelled context to prevent panic #5041 (@joe-elliott)
• [BUGFIX] Fix frontend cache key generation for TraceQL Metrics queries to prevent collisions. #5017 (@joe-elliott)
• [BUGFIX] Fix setting processors in user configurations overrides via API #4741 (@ruslan-mikhailov)
• [BUGFIX] Fix panic on startup #4744 (@ruslan-mikhailov)
• [BUGFIX] Fix intrinsic tag lookups dropped when max tag lookup response size is exceeded #4784 (@mdisibio)
• [BUGFIX] Use canonical bytesize when parsing partition.id in the distributor #5033 (@javiermolinar)
• [BUGFIX] Measure the write latency on append errors #5034 (@javiermolinar)
• [BUGFIX] Correctly cache frontend jobs for query range (TraceQL Metrics). #4771 (@joe-elliott)
• [BUGFIX] Fix error propagation in the SyncIterator. #5045 (@joe-elliott)
• [BUGFIX] Apply the artificial latency even on pushTrace errors #5153 (@javiermolinar)
• [BUGFIX] Various edge case fixes for query range (TraceQL Metrics) #4962 (@ruslan-mikhailov)
• [BUGFIX] Fix mixin to include otlp_v1_traces http write route #5072 (@mdisibio)
• [BUGFIX] Fix TempoBlockListRisingQuickly alert grouping. #4876 (@mapno)
• [BUGFIX] Reset SkipMetricsGeneration before reuse. #5117 (@flxbk)
• [BUGFIX] Fix metrics generator host info processor overrides config. #5118 (@rlankfo)
• [BUGFIX] Fix metrics generator target_info to skip attributes with no name to prevent downstream errors #5148 (@mdisibio)
• [BUGFIX] Fix for queried number of exemplars (TraceQL Metrics) #5115 (@ruslan-mikhailov)