Breaking Changes

• (#1478) In order to build advanced visualization features into Grafana we have decided to change our spanmetric names to match OTel conventions. This way any functionality added to Grafana will work whether you use Tempo, Grafana Agent or the OTel Collector to generate metrics. Details in the span metrics documentation.
• (#1556) Jsonnet users will need to specify ephemeral storage requests and limits for the metrics generator.
• (#1481) Anonymous usage reporting has been added. Distributors and metrics generators will now require permissions to object storage equivalent to compactors and ingesters. This feature is enabled by default but can be disabled easily.
• (#1558) Deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds have been removed in favor of tempodb_backend_request_duration_seconds.

Changes

• [CHANGE] metrics-generator: Changed added metric label instance to __metrics_gen_instance to reduce collisions with custom dimensions. #1439 (@joe-elliott)
• [CHANGE] Don't enforce max_bytes_per_tag_values_query when set to 0. #1447 (@joe-elliott)
• [CHANGE] Add new querier service in deployment jsonnet to serve /status endpoint. #1474 (@annanay25)
• [CHANGE] Swapped out Google Cloud Functions serverless docs and build for Google Cloud Run. #1483 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Change spanmetrics metric names and labels to match OTel conventions. #1478 (@mapno)
  Old metric names:

traces_spanmetrics_duration_seconds_{sum,count,bucket}

New metric names:

traces_spanmetrics_latency_{sum,count,bucket}

Additionally, default label span_status is renamed to status_code.

• [CHANGE] Update to Go 1.18 #1504 (@annanay25)
• [CHANGE] Change tag/value lookups to return partial results when reaching response size limit instead of failing #1517 (@mdisibio)
• [CHANGE] Change search to be case-sensitive #1547 (@mdisibio)
• [CHANGE] Relax Hedged request defaults for external endpoints. #1566 (@joe-elliott)

  querier:
    search:
      external_hedge_requests_at: 4s    -> 8s
      external_hedge_requests_up_to: 3  -> 2
  

• [CHANGE] BREAKING CHANGE Include emptyDir for metrics generator wal storage in jsonnet #1556 (@zalegrala)
  Jsonnet users will now need to specify a storage request and limit for the generator wal.

    _config+:: {
      metrics_generator+: {
        ephemeral_storage_request_size: '10Gi',
        ephemeral_storage_limit_size: '11Gi',
      },
    }

• [CHANGE] Two additional latency buckets added to the default settings for generated spanmetrics. Note that this will increase cardinality when using the defaults. #1593 (@fredr)
• [CHANGE] Mark log_received_traces as deprecated. New flag is log_received_spans.
  Extend distributor spans logger with optional features to include span attributes and a filter by error status. #1465 (@faustodavid)

Features

• [FEATURE] Add parquet block format #1479 #1531 #1564 (@annanay25, @mdisibio)
• [FEATURE] Add anonymous usage reporting, enabled by default. #1481 (@zalegrala)
  BREAKING CHANGE As part of the usage stats inclusion, the distributor will also require access to the store. This is required so the distirbutor can know which cluster it should be reporting membership of.
• [FEATURE] Include messaging systems and databases in service graphs. #1576 (@kvrhdn)

Enhancements

• [ENHANCEMENT] Added the ability to have a per tenant max search duration. #1421 (@joe-elliott)
• [ENHANCEMENT] metrics-generator: expose max_active_series as a metric #1471 (@kvrhdn)
• [ENHANCEMENT] Azure Backend: Add support for authentication with Managed Identities. #1457 (@joe-elliott)
• [ENHANCEMENT] Add metric to track feature enablement #1459 (@zalegrala)
• [ENHANCEMENT] Added s3 config option insecure_skip_verify #1470 (@zalegrala)
• [ENHANCEMENT] Added polling option to reduce issues in Azure blocklist_poll_jitter_ms #1518 (@joe-elliott)
• [ENHANCEMENT] Add a config to query single ingester instance based on trace id hash for Trace By ID API. (1484)[https://github.com//pull/1484] (@sagarwala, @bikashmishra100, @ashwinidulams)
• [ENHANCEMENT] Add blocklist metrics for total backend objects and total backend bytes #1519 (@ie-pham)
• [ENHANCEMENT] Adds tempo_querier_external_endpoint_hedged_roundtrips_total to count the total hedged requests #1558 (@joe-elliott)
  BREAKING CHANGE Removed deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds in favor of tempodb_backend_request_duration_seconds. These metrics
  have been deprecated since v1.1.
• [ENHANCEMENT] Add tags option for s3 backends. This allows new objects to be written with the configured tags. #1442 (@stevenbrookes)
• [ENHANCEMENT] metrics-generator: support per-tenant processor configuration #1434 (@kvrhdn)
• [ENHANCEMENT] Include rollout dashboard #1456 (@zalegrala)
• [ENHANCEMENT] Add SentinelPassword configuration for Redis #1463 (@zalegrala)
• [ENHANCEMENT] Add support for time picker in jaeger query plugin. #1631 (@rubenvp8510)

Bugfixes

• [BUGFIX] Fix nil pointer panic when the trace by id path errors. #1441 (@joe-elliott)
• [BUGFIX] Update tempo microservices Helm values example which missed the 'enabled' key for thriftHttp. #1472 (@hajowieland)
• [BUGFIX] Fix race condition in forwarder overrides loop. 1468 (@mapno)
• [BUGFIX] Fix v2 backend check on span name to be substring #1538 (@mdisibio)
• [BUGFIX] Fix wal check on span name to be substring #1548 (@mdisibio)
• [BUGFIX] Prevent ingester panic "cannot grow buffer" #1258 (@mdisibio)
• [BUGFIX] metrics-generator: do not remove x-scope-orgid header in single tenant modus #1554 (@kvrhdn)
• [BUGFIX] Fixed issue where backend does not support root.name and root.service.name #1589 (@kvrhdn)
• [BUGFIX] Fixed ingester to continue starting up after block replay error #1603 (@mdisibio)