Breaking Changes

This release contains a number of small breaking changes. They will likely have no impact on your deployment, but it should be noted that due to a change in the API between the query-frontend and querier there may be a temporary read outage during deployment.

• [CHANGE] BREAKING CHANGE Drop support for v0 and v1 blocks. See 1.1 changelog for details #919 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Consolidate status information onto /status endpoint #952 @zalegrala)
  The following endpoints moved.
  /runtime_config moved to /status/runtime_config
/config moved to /status/config
/services moved to /status/services
• [CHANGE] BREAKING CHANGE Change ingester metric ingester_bytes_metric_total in favor of ingester_bytes_received_total #979 (@mapno)
• [CHANGE] Renamed CLI flag from --storage.trace.maintenance-cycle to --storage.trace.blocklist_poll. This is a BREAKING CHANGE #897 (@mritunjaysharma394)
• [CHANGE] BREAKING CHANGE Support partial results from failed block queries #1007 (@mapno)
  Querier GET /querier/api/traces/<traceid> response's body has been modified
  to return tempopb.TraceByIDResponse instead of simply tempopb.Trace. This will cause a disruption of the read path during rollout of the change.
• [CHANGE] BRREAKING CHANGE Change the metrics name from cortex_runtime_config_last_reload_successful to tempo_runtime_config_last_reload_successful #945 (@kavirajk)

New Features and Enhancements

• [FEATURE] Add ability to search ingesters for traces #806 (@mdisibio @kvrhdn @annanay25)
• [FEATURE] Add runtime config handler #936 (@mapno)
• [FEATURE] Add ScalableSingleBinary operational run mode #1004 (@zalegrala)
• [ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
• [ENHANCEMENT] Added "search blocks" cli option. #972 (@joe-elliott)
• [ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
• [ENHANCEMENT] Add support to tempo workloads to overrides from single configmap in microservice mode. #896 (@kavirajk)
• [ENHANCEMENT] Updated config defaults to reflect better capture operational knowledge. #913 (@joe-elliott)

  ingester:
    trace_idle_period: 30s => 10s  # reduce ingester memory requirements with little impact on querying
    flush_check_period: 30s => 10s
  query_frontend:
    query_shards: 2 => 20          # will massively improve performance on large installs
  storage:
    trace:
      wal:
        encoding: none => snappy   # snappy has been tested thoroughly and ready for production use
      block:
        bloom_filter_false_positive: .05 => .01          # will increase total bloom filter size but improve query performance
        bloom_filter_shard_size_bytes: 256KiB => 100 KiB # will improve query performance
  compactor:
    compaction:
      chunk_size_bytes: 10 MiB => 5 MiB  # will reduce compactor memory needs
      compaction_window: 4h => 1h        # will allow more compactors to participate in compaction without substantially increasing blocks
  

• [ENHANCEMENT] Make s3 backend readError logic more robust #905 (@wei840222)
• [ENHANCEMENT] Add gen index and gen bloom commands to tempo-cli. #903 (@annanay25)
• [ENHANCEMENT] Implement trace comparison in Vulture #904 (@zalegrala)
• [ENHANCEMENT] Compression updates: Added s2, improved snappy performance #961 (@joe-elliott)
• [ENHANCEMENT] Add support for vulture sending long running traces #951 (@zalegrala)
• [ENHANCEMENT] Shard tenant index creation by tenant and add functionality to handle stale indexes. #1005 (@joe-elliott)
• [ENHANCEMENT] Support partial results from failed block queries #1007 (@mapno)
• [ENHANCEMENT] Add new metric tempo_distributor_push_duration_seconds #1027 (@zalegrala)
• [ENHANCEMENT] Add query parameter to show the default config values and the difference between the current values and the defaults. #1045 (@MichelHollands)
• [ENHANCEMENT] Adding metrics around ingester flush retries #1049 (@dannykopping)
• [ENHANCEMENT] Performance: More efficient distributor batching #1075 (@joe-elliott)
• [ENHANCEMENT] Include tempo-cli in the release #1086 (@zalegrala)

Bug Fixes

• [BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
• [BUGFIX] Fix "magic number" errors and other block mishandling when an ingester forcefully shuts down #937 (@mdisibio)
• [BUGFIX] Fix compactor memory leak #806 (@mdisibio)
• [BUGFIX] Set span's tag span.kind to client in query-frontend #975 (@mapno)
• [BUGFIX] Fixes tempodb_backend_hedged_roundtrips_total to correctly count hedged roundtrips. #1079 (@joe-elliott)

Other Changes

• [CHANGE] update jsonnet alerts and recording rules to use job_selectors and cluster_selectors for configurable unique identifier labels #935 (@kevinschoonover)
• [CHANGE] Add troubleshooting language to config for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size when handling large traces #1023 (@thejosephstevens)