Breaking Changes

• [CHANGE] Breaking Change Remove support tolerate_failed_blocks. #2416 (@joe-elliott)
  Removed config option:

  query_frontend:
    tolerate_failed_blocks: <int>
  

• [CHANGE] Breaking Change Rename s3.insecure_skip_verify #2407 (@zalegrala)

storage:
  trace:
    s3:
      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify

• [CHANGE] Breaking Change Convert metrics generator from deployment to a statefulset in jsonnet. Refer to the PR for seamless migration instructions. #2533 #2467 (@zalegrala)

Changes

• [CHANGE] Make vParquet2 the default block format #2526 (@stoewer)
• [CHANGE] Change log level of two compactor messages from debug to info. #2443 (@DylanGuedes)
• [CHANGE] Remove tenant_header_key option from tempo-query config #2414 (@kousikmitra)
• [CHANGE] Prefix service graph extra dimensions labels with server_ and client_ if enable_client_server_prefix is enabled #2335 (@domasx2)
• [CHANGE] Ignore context canceled errors in the queriers #2440 (@joe-elliott)
• [CHANGE] Start flush queue worker after wal replay and block rediscovery #2456 (@ie-pham)
• [CHANGE] Update Go to 1.20.4 #2486 (@ie-pham)

Features

• [FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor #2368 #2418 #2424 #2442 #2480 #2481 #2501 #2579 #2582 (@mdisibio @zalegrala)
• [FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) #2625 #2660 (@mdisibio)
• [FEATURE] Add user-configurable overrides module #2543 (@electron0zero @kvrhdn)
• [FEATURE] Add support for q query param in /api/v2/search/<tag.name>/values to filter results based on a TraceQL query #2253 (@mapno)
  To make use of filtering, configure autocomplete_filtering_enabled.
• [FEATURE] Add support for by() and coalesce() to TraceQL. #2490
• [FEATURE] Add a GRPC streaming endpoint for traceql search #2366 (@joe-elliott)
• [FEATURE] Add select() to TraceQL #2494 (@joe-elliott)
• [FEATURE] Add traceDuration, rootName and rootServiceName intrinsics to TraceQL #2503 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Add support for query batching between frontend and queriers to improve throughput #2677 (@joe-elliott)
• [ENHANCEMENT] Add initial RBAC support for serverless backend queries, limited to Google CloudRun #2487 (@modulitos)
• [ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped #2538(@mghildiy)
• [ENHANCEMENT] Fill parent ID column and nested set columns #2487 (@stoewer)
• [ENHANCEMENT] Add metrics generator config option to allow customizable ring port #2399 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL regex #2484 (@mdisibio)
• [ENHANCEMENT] log client ip to help identify which client is no org id #2436 (@Komey)
• [ENHANCEMENT] Add spss parameter to /api/search/tags to configure the spans per span set in response#2308 (@stoewer)
• [ENHANCEMENT] Continue polling tenants on error with configurable threshold #2540 (@mdisibio)
• [ENHANCEMENT] Fully skip over parquet row groups with no matches in the column dictionaries #2676 (@mdisibio)
• [ENHANCEMENT] Add prefix configuration option for s3 #2362 (@kousikmitra)
• [ENHANCEMENT] Add support to filter using negated regex operator !~ #2410 (@kousikmitra)
• [ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2386 (@kousikmitra)
• [ENHANCEMENT] Add prefix configuration option to storage.trace.s3 #2362 (@kousikmitra)
• [ENHANCEMENT] Add support for concurrent_shards under trace_by_id #2416 (@joe-elliott)

  query_frontend:
    trace_by_id:
      concurrent_shards: 3
  

• [ENHANCEMENT] Enable cross cluster querying by adding two config options. #2598 (@joe-elliott)

  querier:
    secondary_ingester_ring: <string>
  metrics_generator:
    override_ring_key: <string>
  

• [ENHANCEMENT] Add scope parameter to /api/search/tags #2282 (@joe-elliott)
  Create new endpoint /api/v2/search/tags that returns all tags organized by scope.
• [ENHANCEMENT] Ability to toggle off latency or count metrics in metrics-generator #2070 (@AlexDHoffer)
• [ENHANCEMENT] Extend /flush to support flushing a single tenant #2260 (@kvrhdn)
• [ENHANCEMENT] Add override to limit number of blocks inspected in tag value search #2358 (@mapno)
• [ENHANCEMENT] New synchronous read mode for vParquet and vParquet2 #2165 #2535 (@mdisibio)
• [ENHANCEMENT] Add option to override metrics-generator ring port #2399 (@mdisibio)
• [ENHANCEMENT] Add support for IPv6 #1555 (@zalegrala)
• [ENHANCEMENT] Add span filtering to spanmetrics processor #2274 (@zalegrala)
• [ENHANCEMENT] Add ability to detect virtual nodes in the servicegraph processor #2365 (@mapno)
• [ENHANCEMENT] Introduce overrides.Interface to decouple implementation from usage #2482 (@kvrhdn)
• [ENHANCEMENT] Improve TraceQL throughput by asynchronously creating jobs #2530 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix Search SLO by routing tags to a new handler. #2468 (@electron0zero)
• [BUGFIX] tempodb integer divide by zero error #2167 (@kroksys)
• [BUGFIX] metrics-generator: ensure Prometheus will scale up shards when remote write is lagging behind #2463 (@kvrhdn)
• [BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. #2490
• [BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error #2534 (@mdisibio)
• [BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. #2585 (@WildCatFish)
• [BUGFIX] Delay logging config warnings until the logger has been initialized #2645 (@kvrhdn)
• [BUGFIX] Fix issue where metrics-generator was setting wrong labels for traces_target_info #2546 (@ie-pham)