Breaking Changes
This release deprecates some internal data formats from prerelease versions of Tempo. If upgrading from Tempo v0.6.0 or earlier, then see the special upgrade instructions below. Tempo v0.7.0 and later have no compatibility issues or special instructions.
Tempo v0.6.0 and earlier used block formats v0 and v1, which are being deprecated, and support for these blocks will be removed in the next release. To resolve this you must first upgrade to Tempo 0.7.0+ (latest 1.1 is recommended) which introduces the supported v2 block format. Tempo will write all new blocks as v2, and it must continue running until all v0 and v1 blocks are gone (either deleted due to retention, or compacted). Block versions can be checked using the tempo-cli list blocks command.
New Features and Enhancements
[FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
[FEATURE] Added a tenant index to reduce bucket polling. #834 (@joe-elliott)
[ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@JosephWoodward)
[ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
[ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
[ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@JosephWoodward)
[ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@JosephWoodward)
[ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
[ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
[ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
[ENHANCEMENT] Add ability to control bloom filter caching based on age and/or compaction level. Add new cli command list cache-summary. #805 (@annanay25)
[ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
[ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)
[ENHANCEMENT] Add -use-otel-tracer to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). #842 (@kvrhdn)
[ENHANCEMENT] Add /services endpoint. #863 (@kvrhdn)
[ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
[ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
Bug Fixes
[BUGFIX] Allow only valid trace ID characters when decoding #854 (@zalegrala)
[BUGFIX] Queriers complete one polling cycle before finishing startup. #834 (@joe-elliott)
[BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
[BUGFIX] Cortex upgrade to fix an issue where unhealthy compactors can't be forgotten #878 (@joe-elliott)
Other Changes
[CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 #841 (@aknuds1)
[CHANGE] Change example default tempo port from 3100 to 3200 #770 (@MurzNN)
[CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
[CHANGE] Docker images are now prefixed by their branch name #828 (@jvrplmlmn)