This release contains 81 PRs from 27 authors, including new contributors Edgaras Giedrė, Julius Hinze. Thank you!

Grafana Mimir version 2.17.0-rc.1 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.1

Grafana Mimir

• [FEATURE] Distributor: Add experimental -distributor.otel-native-delta-ingestion option to allow primitive delta metrics ingestion via the OTLP endpoint. #11631
• [FEATURE] MQE: Add support for experimental sort_by_label and sort_by_label_desc PromQL functions. #11930
• [FEATURE] Ingester/Block-builder: Handle the created timestamp field for remote-write requests. #11977
• [FEATURE] Cost attribution: Labels specified in the limit configuration may specify an output label in order to override emitted label names. #12035
• [ENHANCEMENT] Ingester: Display user grace interval in the tenant list obtained through the /ingester/tenants endpoint. #11961
• [ENHANCEMENT] kafkatool: add consumer-group delete-offset command as a way to delete the committed offset for a consumer group. #11988
• [ENHANCEMENT] Block-builder-scheduler: Detect gaps in scheduled and completed jobs. #11867
• [ENHANCEMENT] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol has been updated. Created timestamps are now supported. This feature includes some limitations. If samples in a write request aren't ordered by time, the created timestamp might be dropped. Additionally, per-series metadata is automatically merged on the metric family level. Ingestion might fail if the client sends ProtoBuf fields out-of-order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0, depending on the detected remote-write protocol. #11977
• [ENHANCEMENT] Query-frontend: Added labels query optimizer that automatically removes redundant __name__!="" matchers from label names and label values queries, improving query performance. You can enable the optimizer per-tenant with the labels_query_optimizer_enabled runtime configuration flag. #12054 #12066 #12076 #12080
• [ENHANCEMENT] Query-frontend: Standardise non-regex patterns in query blocking upon loading of config. #12102
• [ENHANCEMENT] Ruler: Propagate GCS object mutation rate limit for rule group uploads. #12086
• [BUGFIX] Distributor: Validate the RW2 symbols field and reject invalid requests that don't have an empty string as the first symbol. #11953
• [BUGFIX] Distributor: Check max_inflight_push_requests_bytes before decompressing incoming requests. #11967
• [BUGFIX] Query-frontend: Allow limit parameter to be 0 in label queries to explicitly request unlimited results. #12054
• [BUGFIX] Distributor: Fix a possible panic in the OTLP push path while handling a gRPC status error. #12072
• [BUGFIX] Query-frontend: Evaluate experimental duration expressions before sharding, splitting, and caching. Otherwise, the result is not correct. #12038
• [BUGFIX] Block-builder-scheduler: Fix bugs in handling of partitions with no commit. #12130

Mixin

• [CHANGE] Remove support for the experimental read-write deployment mode. #11975
• [CHANGE] Alerts: Replace namespace with job label in golang_alerts. #11957
• [FEATURE] Add an alert if the block-builder-scheduler detects that it has skipped data. #12118

Jsonnet

• [CHANGE] Removed support for the experimental read-write deployment mode. #11974
• [ENHANCEMENT] Add assertion to ensure ingester ScaledObject has minimum and maximum replicas set to a value greater than 0. #11979
• [ENHANCEMENT] Add ingest_storage_migration_ignore_ingest_storage_errors and ingest_storage_migration_ingest_storage_max_wait_time configs to control error handling of the partition ingesters during ingest storage migrations. #12105
• [ENHANCEMENT] Add block-builder job processing duration timings and offset-skipped errors to the Block-builder dashboard. #12118

Documentation

• [ENHANCEMENT] Update the MimirIngestedDataTooFarInTheFuture runbook with a note about false positives and the endpoint to flush TSDB blocks by user. #11961

All changes in this release: mimir-2.17.0-rc.0...mimir-2.17.0-rc.1