Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. #3552 (@teyyubismayil)
• [BUGFIX] Correctly parse traceql queries with > 1024 character attribute names or static values. #3571 (@joe-elliott)
• [BUGFIX] Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants. #3612 (@mapno)
• [BUGFIX] Fix panic in query-frontend when combining results #3683 (@mapno)