Breaking Changes

Config

[CHANGE] BREAKING CHANGE Use snake case on Azure Storage config #1879 (@faustodavid)
Example of using snake case on Azure Storage config:
```
# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:
```

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)

query_frontend:
  query_shards:                  // removed. use trace_by_id.query_shards
querier:
    query_timeout:               // removed. use trace_by_id.query_timeout
compactor:
    compaction:
        chunk_size_bytes:        // renamed to v2_in_buffer_bytes
        flush_size_bytes:        // renamed to v2_out_buffer_bytes
        iterator_buffer_size:    // renamed to v2_prefetch_traces_count
ingester:
    use_flatbuffer_search:       // removed. automatically set based on block type
storage:
  traces:
    wal:
        encoding:                // renamed to v2_encoding
        version:                 // removed and pinned to block.version
    block:
        index_downsample_bytes:  // renamed to v2_index_downsample_bytes
        index_page_size_bytes:   // renamed to v2_index_page_size_bytes
        encoding:                // renamed to v2_encoding
        row_group_size_bytes:    // renamed to parquet_row_group_size_bytes

[CHANGE] BREAKING CHANGE Remove search_enabled and metrics_generator_enabled. Both default to true. #2004 (@joe-elliott)
[CHANGE] BREAKING CHANGE Parquet is the new default block version #1678
To continue using the v2 backend set:
```
storage:
  trace:
    block:
      version: v2
```

Jsonnet

[CHANGE] Delete TempoRequestErrors alert from mixin #1810 (@zalegrala)
- BREAKING CHANGE Any jsonnet users relying on this alert should copy this into their own environment.

Metrics Generator

[CHANGE] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
BREAKING CHANGE Custom dimensions colliding with intrinsic dimensions will be prefixed with __.

Changes

[CHANGE] Increase default values for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size from 4MB to 16MB #1688 (@mapno)
[CHANGE] Update Go to 1.19 #1665 (@ie-pham)
[CHANGE] Update alpine image version to 3.16. #1784 (@zalegrala)

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott)
Defaults updated:

query_frontend:
  max_oustanding_per_tenant: 2000
  search:
      concurrent_jobs: 1000
      target_bytes_per_job: 104857600
      max_duration: 168h
      query_ingesters_until: 30m
  trace_by_id:
      query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

Features

[FEATURE] TraceQL Phase 1 support. A new query language for traces!
[FEATURE] Parquet backend is GA and default. By default Tempo will create Parquet blocks to enhance search performance
and to provide users with their data in an open format. See breaking changes above for a note on how to continue using older backends.

[FEATURE] Add generic forwarder and implement otlpgrpc forwarder #1775 (@Blinkuu)
New config options and example configuration:

# config.yaml
distributor:
  forwarders:
    - name: "otel-forwarder"
      backend: "otlpgrpc"
      otlpgrpc:
        endpoints: ['otelcol:4317']
        tls:
          insecure: true

# overrides.yaml
overrides:
  "example-tenant-1":
    forwarders: ['otel-forwarder']
  "example-tenant-2":
    forwarders: ['otel-forwarder']

Enhancements

[ENHANCEMENT] Add /status/usage-stats endpoint to show usage stats data #1782 (@electron0zero)
[ENHANCEMENT] Add TLS support to jaeger query plugin. #1999 (@rubenvp8510)
[ENHANCEMENT] Collect inspectedBytes from SearchMetrics #1975 (@electron0zero)
[ENHANCEMENT] Add zone awareness replication for ingesters. #1936 (@manohar-koukuntla)

# use the following fields in _config field of jsonnet config, to enable zone aware ingester
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

[ENHANCEMENT] Add new data-type aware searchtagvalues v2 api #1956 (@mdisibio)
[ENHANCEMENT] Filter namespace by cluster in tempo dashboards variables #1771 (@electron0zero)
[ENHANCEMENT] Exit early from sharded search requests #1742 (@electron0zero)
[ENHANCEMENT] Avoid running tempodb pool jobs with a cancelled context #1852 (@zalegrala)
[ENHANCEMENT] Add config flag to allow for compactor disablement for debug purposes #1850 (@zalegrala)
[ENHANCEMENT] Identify bloom that could not be retrieved from backend block #1737 (@AlexDHoffer)
[ENHANCEMENT] tempo: check configuration returns now a list of warnings #1663 (@frzifus)
[ENHANCEMENT] Make DNS address fully qualified to reduce DNS lookups in Kubernetes #1687 (@electron0zero)
[ENHANCEMENT] Return 200 instead of 206 when blocks failed is < tolerate_failed_blocks. #1725 (@joe-elliott)
[ENHANCEMENT] Add GOMEMLIMIT variable to compactor jsonnet and set the value to equal compactor memory limit. #1758 (@ie-pham)
[ENHANCEMENT] Add capability to configure the used S3 Storage Class #1697 (@amitsetty)
[ENHANCEMENT] cache: expose username and sentinel_username redis configuration options for ACL-based Redis Auth support #1708 (@jsievenpiper)
[ENHANCEMENT] metrics-generator: expose span size as a metric #1662 (@ie-pham)
[ENHANCEMENT] Set Max Idle connections to 100 for Azure, should reduce DNS errors in Azure #1632 (@electron0zero)
[ENHANCEMENT] Add PodDisruptionBudget to ingesters in jsonnet #1691 (@joe-elliott)
[ENHANCEMENT] Add a cli command to convert a block to the current parquet schema. #1707 (@joe-elliott)
[ENHANCEMENT] metrics-generator: filter out older spans before metrics are aggregated #1612 (@ie-pham)
[ENHANCEMENT] Add hedging to trace by ID lookups created by the frontend. #1735 (@mapno)
New config options and defaults:

query_frontend:
  trace_by_id:
    hedge_requests_at: 5s
    hedge_requests_up_to: 3

[ENHANCEMENT] Vulture now has improved distribution of the random traces it searches. #1763 (@rfratto)
[ENHANCEMENT] Add TLS support to the vulture #1874 (@zalegrala)
[ENHANCEMENT] metrics-generator: extract status_message field from spans #1786, #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: make intrinsic dimensions configurable and disable status_message by default #1960 (@stoewer)
[ENHANCEMENT] distributor: Log span names when distributor.log_received_spans.include_all_attributes is on #1790 (@suraciii)
[ENHANCEMENT] metrics-generator: truncate label names and values exceeding a configurable length #1897 (@kvrhdn)
[ENHANCEMENT] Convert last few Jsonnet alerts with per_cluster_label #2000 (@Whyeasy)
[ENHANCEMENT] New tenant dashboard #1901 (@mapno)
[ENHANCEMENT] Upgrade opentelemetry-proto submodule to v0.18.0 Internal types are updated to use scope instead of instrumentation_library.
This is a breaking change in trace by ID queries if JSON is requested. #1754 (@mapno)

Bugfixes

[BUGFIX] Stop distributors on Otel receiver fatal error#1887 (@rdooley)
[BUGFIX] New wal file separator '+' for the NTFS filesystem and backward compatibility with the old separator ':' #1700 (@kilian-kier)
[BUGFIX] Honor caching and buffering settings when finding traces by id #1697 (@joe-elliott)
[BUGFIX] Correctly propagate errors from the iterator layer up through the queriers #1723 (@joe-elliott)
[BUGFIX] Make multitenancy work with HTTP #1781 (@gouthamve)
[BUGFIX] Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned #1799 (@mdisibio)
[BUGFIX] tempo-mixin: tweak dashboards to support metrics without cluster label present #1913 (@kvrhdn)
[BUGFIX] Fix docker-compose examples not running on Apple M1 hardware #1920 (@stoewer)
[BUGFIX] Don't persist tenants without blocks in the ingester#1947 (@joe-elliott)
[BUGFIX] Return more consistent search results by combining partial traces # (@mapno)

grafana/tempo v2.0.0-rc.0 on GitHub