This release contains 578 PRs from 62 authors, including new contributors Andrew Hall, Frank Kloeker, Gregor Zeitlinger, Ilia Lazebnik, Julien, Maruth Goyal, Matt Clegg, Mattia Barbon, Mohijeet, Oleg Vorobev, Oliver Herrmann, Phil Kates, Pranshu Raj, Rob B, Timon Engelke, Vadim Stepanov, dannyc-grafana, unkls ben. Thank you!

Grafana Mimir version 3.0.0-rc.0 release notes

Changelog

3.0.0-rc.0

Grafana Mimir

[CHANGE] Build: Include updated Mozilla CA bundle from Debian Testing. #12247
[CHANGE] Query-frontend: Add support for UTF-8 label and metric names in /api/v1/cardinality/{label_values|label_values|active_series} endpoints. #11848.
[CHANGE] Querier: Add support for UTF-8 label and metric names in label_join, label_replace and count_values PromQL functions. #11848.
[CHANGE] Remove support for Redis as a cache backend. #12163
[CHANGE] Memcached: Remove experimental -<prefix>.memcached.addresses-provider flag to use alternate DNS service discovery backends. The more reliable backend introduced in 2.16.0 (#10895) is now the default. As a result of this change, DNS-based cache service discovery no longer supports search domains. #12175 #12385
[CHANGE] Query-frontend: Remove the CLI flag -query-frontend.downstream-url and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191 #12517
[CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
[CHANGE] Query-frontend, querier: Replace query-frontend.prune-queries flag with querier.mimir-query-engine.enable-prune-toggles as pruning middleware has been moved into MQE. #12303 #12375
[CHANGE] Distributor: Remove deprecated global HA tracker timeout configuration flags. #12321
[CHANGE] Query-frontend: Use the Mimir Query Engine (MQE) by default. #12361
[CHANGE] Query-frontend: Remove the CLI flags -querier.frontend-address, -querier.max-outstanding-requests-per-tenant, and -query-frontend.querier-forget-delay and corresponding YAML configurations. This is part of a change that makes the query-scheduler a required component. This removes the ability to run the query-frontend with an embedded query-scheduler. Instead, you must run a dedicated query-scheduler component. #12200
[CHANGE] Ingester: Remove deprecated -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option. #12615
[CHANGE] Querier: Require non-zero values for -querier.streaming-chunks-per-ingester-buffer-size and -querier.streaming-chunks-per-store-gateway-buffer-size CLI flags and corresponding YAML configurations. This is part of a change that makes streaming required between queriers, ingesters, and store-gateways. Streaming has been the default since Mimir 2.14. #12790 #12818 #12897 #12929 #12973
[CHANGE] Remove support for the experimental read-write deployment mode. #12584
[CHANGE] Store-gateway: Update default value of -store-gateway.dynamic-replication.multiple to 5 to increase replication of recent blocks. #12433
[CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
[CHANGE] Querier, query-frontend: Add _total suffix to cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced, cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated and cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected metric names. #12636
[CHANGE] Distributor: Remove the experimental setting service_overload_status_code_on_rate_limit_enabled which used an HTTP 529 error (non-standard) instead of HTTP 429 for rate limiting. #13012
[CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
[CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
[CHANGE] Distributor: gRPC errors with the mimirpb.ERROR_CAUSE_INSTANCE_LIMIT cause are now mapped to codes.Unavailable and http.StatusServiceUnavailable instead of codes.Internal and http.StatusInternalServerError. #13003 #13032
[CHANGE] Admin: use relative links instead of absolute ones in the administration web UI. #13034
[CHANGE] Distributor: Use memberlist by default for the HA tracker. #12998
[CHANGE] Block-builder: Remove cortex_blockbuilder_process_partition_duration_seconds metric and related dashboard panels. #12631
[FEATURE] Ingester: Expose the number of active series ingested via OTLP as cortex_ingester_active_otlp_series. #12678
[FEATURE] Distributor, ruler: Add experimental -validation.name-validation-scheme flag to specify the validation scheme for metric and label names. #12215
[FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See remote evaluation mode for more details. This feature can be used to federate data from multiple Mimir instances. #11415 #11833
[FEATURE] Distributor: Add experimental -distributor.otel-translation-strategy flag to support configuring the metric and label name translation strategy in the OTLP endpoint. #12284 #12306 #12369
[FEATURE] Query-frontend: Add query-frontend.rewrite-propagate-matchers flag that enables a new MQE AST optimization pass that copies relevant label matchers across binary operations. #12304
[FEATURE] Query-frontend: Add query-frontend.rewrite-histogram-queries flag that enables a new MQE AST optimization pass that rewrites histogram queries for a more efficient order of execution. #12305
[FEATURE] Query-frontend: Support delayed name removal (Prometheus experimental feature) in MQE. #12509
[FEATURE] Usage-tracker: Introduce a new experimental service to enforce active series limits before Kafka ingestion. #12358 #12895 #12940 #12942 #12970
[FEATURE] Ingester: Add experimental -include-tenant-id-in-profile-labels flag to include tenant ID in pprof profiling labels for sampled traces. Currently only supported by the ingester. This can help debug performance issues for specific tenants. #12404
[FEATURE] Alertmanager: Add experimental -alertmanager.storage.state-read-timeout flag to configure the timeout for reading the Alertmanager state (notification log, silences) from object storage during the initial sync. #12425
[FEATURE] Ingester: Add experimental -blocks-storage.tsdb.index-lookup-planning.* flags to configure use of a cost-based index lookup planner. This should reduce the cost of queries in the ingester. #12197 #12199 #12245 #12248 #12457 #12530 #12407 #12460 #12550 #12597 #12603 #12608 #12658 #12696 #12731 #12755 #12738 #12752 #12807 #12830 #12896 #13039
[FEATURE] MQE: Add support for applying extra selectors to one side of a binary operation to reduce data fetched. #12577
[FEATURE] Query-frontend: Add a native histogram presenting the length of query expressions handled by the query-frontend #12571
[FEATURE] Query-frontend and querier: Add experimental support for performing query planning in query-frontends and distributing portions of the plan to queriers for execution. #12302 #12551 #12665 #12687 #12745 #12757 #12798 #12808 #12809 #12835 #12856 #12870 #12883 #12885 #12886 #12911 #12933 #12934 #12961 #13016 #13027
[FEATURE] Alertmanager: add Microsoft Teams V2 as a supported integration. #12680
[FEATURE] Distributor: Add experimental flag -validation.label-value-length-over-limit-strategy to configure how to handle label values over the length limit. #12627 #12844
[FEATURE] Ingester: Introduce metric cortex_ingester_owned_target_info_series for counting the number of owned target_info series by tenant. #12681
[FEATURE] MQE: Add support for step invariant expression handling in query planning and evaluation. #12931
[FEATURE] MQE: Add support for experimental ts_of_min_over_time, ts_of_max_over_time, ts_of_first_over_time and ts_of_last_over_time PromQL functions. #12819
[FEATURE] Ingester: Add experimental flags -ingest-storage.write-logs-fsync-before-kafka-commit-enabled and -ingest-storage.write-logs-fsync-before-kafka-commit-concurrency to fsync write logs before the offset is committed to Kafka. This is enabled by default. #12816
[FEATURE] MQE: Add support for experimental mad_over_time PromQL function. #12995
[FEATURE] Continuous test: Add experimental -tests.ingest-storage-record.enabled flag to verify ingest-storage record correctness by validating the V2 record format against live write requests. #12500
[ENHANCEMENT] Query-frontend: CLI flag -query-frontend.enabled-promql-experimental-functions and its associated YAML configuration is now stable. #12368
[ENHANCEMENT] Query-scheduler/query-frontend: Add native histogram definitions to cortex_query_{scheduler|frontend}_queue_duration_seconds. #12288
[ENHANCEMENT] Querier: Add native histogram definition to cortex_bucket_index_load_duration_seconds. #12094
[ENHANCEMENT] Query-frontend: Allow users to set the query-frontend.extra-propagated-headers flag to specify the extra headers allowed to pass through to the rest of the query path. #12174
[ENHANCEMENT] MQE: Add support for applying common subexpression elimination to range vector expressions in instant queries. #12236
[ENHANCEMENT] Ingester: Improve the performance of active series custom trackers matchers. #12184
[ENHANCEMENT] Ingester: Add postings cache sharing and invalidation. You can enable sharing and head cache invalidation via -blocks-storage.tsdb.shared-postings-for-matchers-cache and -blocks-storage.tsdb.head-postings-for-matchers-cache-invalidation respectively, and you can configure the number of metric versions per cache via -blocks-storage.tsdb.head-postings-for-matchers-cache-versions. #12333 #12932
[ENHANCEMENT] Overrides-exporter: The overrides-exporter can now export arbitrary fields from the limits configuration. Metric names are automatically discovered from YAML tags in the limits structure, eliminating the need to maintain hardcoded lists when adding new exportable metrics. #12244
[ENHANCEMENT] OTLP: Stick to OTLP vocabulary on invalid label value length error. #12273
[ENHANCEMENT] Elide SeriesChunksStreamReader.StartBuffering span on queries; show as events on parent span. #12257
[ENHANCEMENT] Ruler: Add -ruler.max-notification-batch-size CLI flag that can be used to configure the maximum Alertmanager notification batch size. #12469
[ENHANCEMENT] Ingester: Skip read path load shedding when an ingester is the only available replica. #12448
[ENHANCEMENT] Querier: Include more information about inflight queries in the activity tracker. A querier logs this information after it restarts following a crash. #12526
[ENHANCEMENT] Ruler: Add native histogram version of cortex_ruler_sync_rules_duration_seconds. #12628
[ENHANCEMENT] Block-builder: Implement concurrent consumption within a job when -ingest-storage.kafka.fetch-concurrency-max is given. #12222
[ENHANCEMENT] Query-frontend: Labels query optimizer is no longer experimental and is enabled by default. It can be disabled with -query-frontend.labels-query-optimizer-enabled=false CLI flag. #12606
[ENHANCEMENT] Distributor: Add value length to "label value too long" error. #12583
[ENHANCEMENT] Distributor: The metric cortex_distributor_uncompressed_request_body_size_bytes now differentiates by the handler serving the request. #12661
[ENHANCEMENT] Query-frontend, querier: Add support for experimental first_over_time PromQL function. #12662
[ENHANCEMENT] OTLP: native support for OpenTelemetry metric start time to Prometheus metric created timestamp conversion, instead of converting to QuietZeroNaNs introduced in #10238. The configuration parameter -distributor.otel-start-time-quiet-zero is therefore deprecated and will be removed. Now supports start time for exponential histograms. This is a major rewrite of the endpoint in upstream Prometheus and Mimir. #12652
[ENHANCEMENT] Distributor: Support zstd decompression of OTLP messages. #12229
[ENHANCEMENT] Distributor: Optimize Remote Write 1.0 to 2.0 translation by improving symbolization and reducing allocations. #12329
[ENHANCEMENT] Ingester: Improved the performance of active series custom trackers matchers. #12663
[ENHANCEMENT] Compactor: Log sizes of downloaded and uploaded blocks. #12656
[ENHANCEMENT] Block-builder-scheduler: The scheduler now handles multiple concurrent jobs within a partition if allowed by -block-builder-scheduler.max-jobs-per-partition. #12772
[ENHANCEMENT] Ingester: Add cortex_ingest_storage_reader_receive_and_consume_delay_seconds metric tracking the time between when a write request is received in the distributor and its content is ingested in ingesters, when the ingest storage is enabled. #12751
[ENHANCEMENT] Ruler: Add ruler_evaluation_consistency_max_delay per-tenant configuration option support, to specify the maximum tolerated ingestion delay for eventually consistent rule evaluations. This feature is used only when ingest storage is enabled. By default, no maximum delay is enforced. #12751
[ENHANCEMENT] Ingester: Export cortex_attributed_series_overflow_labels metric on the /usage-metrics metrics endpoint with the configured cost-attribution labels set to overflow value. #12846
[ENHANCEMENT] Usage stats: Report ingest-storage mode as part of usage statistics. #12753
[ENHANCEMENT] All: Add cluster validation flag -server.cluster-validation.additional-labels configuration support, to accept multiple cluster labels during cluster migrations. #12850
[ENHANCEMENT] Distributor: Add new optional config flag distributor.ha-tracker.failover-sample-timeout for HA tracker as an additional failover timeout check based on sample time instead of server time. #12331
[ENHANCEMENT] Distributor: Add reactive concurrency limiters to protect push operations from overload. #12923 #13003 #13033
[ENHANCEMENT] Ingester: Add experimental matcher set reduction to cost-based lookup planning. #12831
[ENHANCEMENT] Ruler: Add reason label to cortex_prometheus_rule_evaluation_failures_total metric to distinguish between "user" and "operator" errors. #12971
[ENHANCEMENT] Update Docker base images from alpine:3.22.1 to alpine:3.22.2. #12991
[ENHANCEMENT] Ruler: Add the ruler_max_rule_evaluation_results per-tenant configuration option to limit the maximum number of alerts an alerting rule or series a recording rule can produce for the group. By default, no limit is enforced. #12832
[ENHANCEMENT] Jsonnet: Changed the default KV store for the HA tracker from etcd to memberlist. Etcd and Consul are now deprecated for HA tracker usage but remain supported for backward compatibility. #13000
[ENHANCEMENT] Querier: prefer querying ingesters and store-gateways in a specific zone when -querier.prefer-availability-zone is configured. Added the following metrics tracking the data transfer between the querier and ingesters / store-gateways respectively: #13045
- cortex_ingester_client_transferred_bytes_total{ingester_zone="..."}
- cortex_storegateway_client_transferred_bytes_total{store_gateway_zone="..."}
[ENHANCEMENT] Compactor: Add experimental -compactor.first-level-compaction-skip-future-max-time flag to skip first-level compaction if any source block has a MaxTime more recent than the wait period threshold. #13040
[ENHANCEMENT] Block-builder-scheduler: Add gap monitoring for planned and completed jobs via cortex_blockbuilder_scheduler_job_gap_detected metric. #11867
[BUGFIX] Distributor: Calculate WriteResponseStats before validation and PushWrappers. This prevents clients using Remote-Write 2.0 from seeing a diff in written samples, histograms and exemplars. #12682
[BUGFIX] Compactor: Fix cortex_compactor_block_uploads_failed_total metric showing type="unknown". #12477
[BUGFIX] Querier: Samples with the same timestamp are merged deterministically. Previously, this could lead to flapping query results when an out-of-order sample is ingested that conflicts with a previously ingested in-order sample's value. #8673
[BUGFIX] Store-gateway: Fix potential goroutine leak by passing the scoped context in LabelValues. #12048
[BUGFIX] Distributor: Fix pooled memory reuse bug that can cause corrupt data to appear in the err-mimir-label-value-too-long error message. #12266
[BUGFIX] Querier: Fix timeout responding to query-frontend when response size is very close to -querier.frontend-client.grpc-max-send-msg-size. #12261
[BUGFIX] Block-builder-scheduler: Fix a caching bug in initial job probing causing excessive memory usage at startup. #12389
[BUGFIX] Ruler: Support labels at the rule group level. These were previously ignored even when set via the API. #12397
[BUGFIX] Distributor: Fix metric metadata of type Unknown being silently dropped from RW2 requests. #12461
[BUGFIX] Distributor: Preserve inconsistent metric metadata in Remote Write 1.0 to 2.0 conversion. Previously, when converting RW1.0 requests with multiple different metadata for the same series, only the first metadata was kept. Now all inconsistent metadata are preserved to match Prometheus behavior. This only affects experimental Remote Write 2.0. #12541 #12804
[BUGFIX] Ruler: Fix ruler remotequerier request body consumption on retries. #12514
[BUGFIX] Block-builder: Fix a bug where a consumption error can cause a job to stay assigned to a worker for the remainder of its lifetime. #12522
[BUGFIX] Querier: Fix possible panic when evaluating a nested subquery where the parent has no steps. #12524
[BUGFIX] Querier: Fix bug where the pruning toggles AST optimization pass doesn't work in the query planner. #12783
[BUGFIX] Ingester: Fix a bug where prepare-instance-ring-downscale endpoint would return an error while compacting and not read-only. #12548
[BUGFIX] Block-builder: Fix a bug where lease renewals would cease during graceful shutdown, leading to an elevated rate of job reassignments. #12643
[BUGFIX] OTLP: Return HTTP OK for partially rejected requests, e.g. due to OOO exemplars. #12579
[BUGFIX] Store-gateway: Fix a panic in BucketChunkReader when chunk loading encounter a broken chunk length. #12693 #12729
[BUGFIX] Ingester, Block-builder: silently ignore duplicate sample if it's due to zero sample from created timestamp. Created timestamp equal to the timestamp of the first sample of series is a common case if created timestamp comes from OTLP where start time equal to timestamp of the first sample simply means unknown start time. #12726
[BUGFIX] Distributor: Fix error when native histograms bucket limit is set then no NHCB passes validation. #12741
[BUGFIX] Ingester: Fix continous reload of active series counters when cost-attribution labels are above the max cardinality. #12822
[BUGFIX] Distributor: Report the correct size in the err-mimir-distributor-max-write-message-size error. #12799
[BUGFIX] Query-frontend: Fix issue where expressions containing unary negation could be sharded incorrectly in some cases. #12911
[BUGFIX] Query-frontend: Fix issue where shardable expressions containing aggregations with a shardable parameter (eg. sum(foo) in topk(scalar(sum(foo)), sum by (pod) (bar))) would not have the parameter sharded. #12958
[BUGFIX] Ingester: Fix max_inflight_push_requests metric and internal counter not decremented under pressure, possibly causing the rejection of all push requests. #12975
[BUGFIX] Store-gateway: Fix not being able to scale down via the POST /prepare-shutdown endpoint unless there are some active tenants with sharded blocks to the store-gateway replica. #12972
[BUGFIX] MQE: Fix invalid source label name in label_join error message, so it refers to the source label rather than the destination label. #12185
[BUGFIX] Continuous test: Fix false positive in metadata assertion when duplicate metadata is present in ingest-storage record correctness test. #12891

Mixin

[CHANGE] Enable ingest storage panels by default in all compiled mixins. #13023
[CHANGE] Alerts: Removed MimirFrontendQueriesStuck alert given this is not relevant when the query-scheduler is running and the query-scheduler is now a required component. #12810
[CHANGE] Alerts: Make MimirIngesterHasNotShippedBlocksSinceStart weaker to account for block-builder restarts. The change only affects the block-builder version of the alert. #12319
[ENHANCEMENT] Rollout progress dashboard: make panels higher to fit more components. #12429
[ENHANCEMENT] Add max_series limit to Writes Resources > Ingester > In-memory series panel. #12476
[ENHANCEMENT] Alerts: Add MimirHighGRPCConcurrentStreamsPerConnection alert. #11947
[ENHANCEMENT] Alerts: Add rollout_stuck_alert_ignore_deployments and rollout_stuck_alert_ignore_statefulsets configuration options to exclude particular Deployments or StatefulSets from the MimirRolloutStuck alert. #12951
[ENHANCEMENT] Alerts: Replace experimental BlockBuilderLagging alert with BlockBuilderSchedulerPendingJobs alert. The new alert triggers when the block-builder scheduler has pending jobs, indicating that block-builders are unable to keep up with the workload. #12593
[ENHANCEMENT] Rollout-operator: Vendor rollout-operator monitoring dashboard from rollout-operator repository. #12688
[BUGFIX] Block-builder dashboard: fix reference to detected gaps metric in errors panel. #12401

Jsonnet

[CHANGE] Removed etcd-operator from the Jsonnet configuration. Users can still use etcd as a KV store for rings, but need to deploy and manage etcd themselves rather than via the operator. #13049
[CHANGE] Distributor: Reduce calculated GOMAXPROCS to be closer to the requested number of CPUs. #12150
[CHANGE] Query-scheduler: The query-scheduler is now a required component that is always used by queriers and query-frontends. #12187
[CHANGE] Rollout-operator: Add watch permission to the rollout-operators's cluster role. #12360. See rollout-operator#262
[CHANGE] Updates to CPU and memory scaling metric. Use irate() when calculating the CPU metric and remove or vector(0) from a leg of the memory query. These changes prevent downscaling deployments when scraping fails. #12406
[CHANGE] Memcached: Remove configuration for enabling mTLS connections to Memcached servers. #12434
[CHANGE] Ingester: Disable shipping of blocks on the third zone (zone-c) when using ingest_storage_ingester_zones: 3 on ingest storage #12743 #12744
[CHANGE] Distributor: Increase server.grpc-max-concurrent-streams from 100 to 1000. #12742
[CHANGE] Ruler Query Frontend: Increase server.grpc-max-concurrent-streams from 100 to 300. #12742
[CHANGE] Rollout-operator: Vendor jsonnet from rollout-operator repository. #12688 #12962 #12996
[CHANGE] Removed per-component configuration options to set the pods toleration when multi-zone is enabled. Tolerations can still be configured globally using _config.multi_zone_schedule_toleration. The following configuration options have been removed: #13043
- _config.multi_zone_distributor_schedule_toleration
- _config.multi_zone_etcd_schedule_toleration
[FEATURE] Memcached: Allow minReadySeconds to be set via _config.cache_frontend_min_ready_seconds (etc.) to slow down Memcached rollouts. #12938
[FEATURE] Distributor: Allow setting GOMEMLIMIT equal to memory request, via _config.distributor_gomemlimit_enabled. If enabled, distributor horizontal auto-scaling memory trigger is also removed, since it doesn't make sense in combination with GOMEMLIMIT. #12963
[ENHANCEMENT] Add timeout validation for querier and query-frontend. Enhanced parseDuration to support milliseconds and combined formats (e.g., "4m30s"). #12766
[ENHANCEMENT] Allow the max number of OTEL events in a span to be configure via _config.otel_span_event_count_limit. #12865
[ENHANCEMENT] Memcached: added the following fields to customise the memcached's node affinity matchers: #12987
- $.memcached_frontend_node_affinity_matchers
- $.memcached_index_queries_node_affinity_matchers
- $.memcached_chunks_node_affinity_matchers
- $.memcached_metadata_node_affinity_matchers
[ENHANCEMENT] Rollout-operator: expose rollout_operator_enabled in $._config. #12419

Documentation

[CHANGE] Remove references to queriers having a Prometheus HTTP API. Instead, the query-frontend is now required for a Prometheus HTTP API. #12949
[CHANGE] Helm: Remove GEM (Grafana Enterprise Metrics) references from Helm chart documentation. #13019 #13020 #13021
[CHANGE] Update HA tracker documentation to use memberlist as the default KV store instead of consul/etcd. Consul and etcd are now marked as deprecated for the HA tracker as of Mimir 3.0. #13002
[ENHANCEMENT] Add migration guide for HA tracker from Consul or etcd to memberlist. #13011
[ENHANCEMENT] Improve the MimirIngesterReachingSeriesLimit runbook. #12356
[ENHANCEMENT] Improve the description of how to limit the number of buckets in native histograms. #12797
[ENHANCEMENT] Document native histograms with custom buckets. #12823
[BUGFIX] Add a missing attribute to the list of default promoted OTel resource attributes in the docs: deployment.environment. #12181

Tools

[ENHANCEMENT] Base mimirtool, metaconvert, copyblocks, and query-tee images on distroless/static-debian12. #13014
[ENHANCEMENT] kafkatool: add format=json to kafkatool dump print. #12737

Query-tee

[CHANGE] If you configure multiple secondary backends and enable comparisons, query-tee reports comparison results of the preferred backend against each of the secondaries. #13022

All changes in this release: mimir-2.17.1...mimir-3.0.0-rc.0

grafana/mimir mimir-3.0.0-rc.0 3.0.0-rc.0 on GitHub