This release contains 531 PRs from 55 authors, including new contributors Benjamin, Dominik Kepinski, Jonathan Donzallaz, Juraj Michálek, Kai.Ke, Ludovic Terrier, Luke, Maciej Lech, Matthew Penner, Michael Potter, Mihai Țimbota-Belin, Rasmus Werner Salling, Ying WANG, chencs, fayzal-g, kalle (jag), renovate[bot], sarthaktyagi-505, whoami. Thank you!

Grafana Mimir version 2.11.0-rc.0 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
• Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
• Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
• Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
• Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
• Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

• Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
• Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
• Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
• Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
  • For the blocks storage:
    • -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
  • For the query frontend:
    • -query-frontend.results-cache.memcached.read-buffer-size-bytes
    • -query-frontend.results-cache.memcached.write-buffer-size-bytes
  • For the ruler storage:
    • -ruler-storage.cache.memcached.read-buffer-size-bytes
    • -ruler-storage.cache.memcached.write-buffer-size-bytes
• Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
• Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
• Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
• Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

• The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

• The CLI flag -querier.iterators.
• The CLI flag -query.batch-iterators.
• The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

• The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

• cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

• The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
• The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
• The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
• The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

• Ingester: Respect context cancelation during query execution. PR 6085
• Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
• Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
• Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067

Changelog

2.11.0-rc.0

Grafana Mimir

• [CHANGE] The following deprecated configurations have been removed: #6673 #6779 #6808 #6814
  • -querier.iterators
  • -querier.batch-iterators
  • -blocks-storage.bucket-store.max-chunk-pool-bytes
  • -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
  • -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
  • -blocks-storage.bucket-store.bucket-index.enabled
• [CHANGE] Querier: Split worker GRPC config into separate client configs for the frontend and scheduler to allow TLS to be configured correctly when specifying the tls_server_name. The GRPC config specified under -querier.frontend-client.* will no longer apply to the scheduler client, and will need to be set explicitly under -querier.scheduler-client.*. #6445 #6573
• [CHANGE] Store-gateway: enable sparse index headers by default. Sparse index headers reduce the time to load an index header up to 90%. #6005
• [CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
• [CHANGE] General: enabled -log.buffered by default. The -log.buffered has been deprecated and will be removed in Mimir 2.13. #6131
• [CHANGE] Ingester: changed default -blocks-storage.tsdb.series-hash-cache-max-size-bytes setting from 1GB to 350MB. The new default cache size is enough to store the hashes for all series in a ingester, assuming up to 2M in-memory series per ingester and using the default 13h retention period for local TSDB blocks in the ingesters. #6130
• [CHANGE] Query-frontend: removed cortex_query_frontend_workers_enqueued_requests_total. Use cortex_query_frontend_enqueue_duration_seconds_count instead. #6121
• [CHANGE] Ingester / querier: enable ingester to querier chunks streaming by default and mark it as stable. #6174
• [CHANGE] Ingester / querier: enable ingester query request minimisation by default and mark it as stable. #6174
• [CHANGE] Ingester: changed the default value for the experimental configuration parameter -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage from 10 to 15. #6186
• [CHANGE] Ingester: /ingester/push HTTP endpoint has been removed. This endpoint was added for testing and troubleshooting, but was never documented or used for anything. #6299
• [CHANGE] Experimental setting -log.rate-limit-logs-per-second-burst renamed to -log.rate-limit-logs-burst-size. #6230
• [CHANGE] Distributor: instead of errors with HTTP status codes, Push() now returns errors with gRPC codes: #6377
  • http.StatusAccepted (202) code is replaced with codes.AlreadyExists.
  • http.BadRequest (400) code is replaced with codes.FailedPrecondition.
  • http.StatusTooManyRequests (429) and the non-standard 529 (The service is overloaded) codes are replaced with codes.ResourceExhausted.
• [CHANGE] Ingester: by setting the newly introduced experimental CLI flag -ingester.return-only-grpc-errors to true, ingester will return only gRPC errors. This feature changes the following status codes: #6443 #6680 #6723
  • http.StatusBadRequest (400) is replaced with codes.FailedPrecondition on the write path.
  • http.StatusServiceUnavailable (503) is replaced with codes.Internal on the write path, and with codes.ResourceExhausted on the read path.
  • codes.Unknown is replaced with codes.Internal on both write and read path.
• [CHANGE] Upgrade Node.js to v20. #6540
• [CHANGE] Querier: cortex_querier_blocks_consistency_checks_failed_total is now incremented when a block couldn't be queried from any attempted store-gateway as opposed to incremented after each attempt. Also cortex_querier_blocks_consistency_checks_total is incremented once per query as opposed to once per attempt (with 3 attempts). #6590
• [CHANGE] Ingester: Modify utilization based read path limiter to base memory usage on Go heap size. #6584
• [FEATURE] Distributor: added option -distributor.retry-after-header.enabled to include the Retry-After header in recoverable error responses. #6608
• [FEATURE] Query-frontend: add experimental support for query blocking. Queries are blocked on a per-tenant basis and is configured via the limit blocked_queries. #5609
• [FEATURE] Vault: Added support for new Vault authentication methods: AppRole, Kubernetes, UserPass and Token. #6143
• [FEATURE] Add experimental endpoint /api/v1/cardinality/active_series to return the set of active series for a given selector. #6536 #6619 #6651 #6667
• [FEATURE] Added -<prefix>.s3.part-size flag to configure the S3 minimum file size in bytes used for multipart uploads. #6592
• [FEATURE] Add the experimental -<prefix>.s3.send-content-md5 flag (defaults to false) to configure S3 Put Object requests to send a Content-MD5 header. Setting this flag is not recommended unless your object storage does not support checksums. #6622
• [FEATURE] Distributor: add an experimental flag -distributor.reusable-ingester-push-worker that can be used to pre-allocate a pool of workers to be used to send push requests to the ingesters. #6660
• [FEATURE] Distributor: Support enabling of automatically generated name suffixes for metrics ingested via OTLP, through the flag -distributor.otel-metric-suffixes-enabled. #6542
• [ENHANCEMENT] Query-frontend: don't treat cancel as an error. #4648
• [ENHANCEMENT] Ingester: exported summary cortex_ingester_inflight_push_requests_summary tracking total number of inflight requests in percentile buckets. #5845
• [ENHANCEMENT] Query-scheduler: add cortex_query_scheduler_enqueue_duration_seconds metric that records the time taken to enqueue or reject a query request. #5879
• [ENHANCEMENT] Query-frontend: add cortex_query_frontend_enqueue_duration_seconds metric that records the time taken to enqueue or reject a query request. When query-scheduler is in use, the metric has the scheduler_address label to differentiate the enqueue duration by query-scheduler backend. #5879 #6087 #6120
• [ENHANCEMENT] Store-gateway: add metric cortex_bucket_store_blocks_loaded_by_duration for counting the loaded number of blocks based on their duration. #6074 #6129
• [ENHANCEMENT] Expose /sync/mutex/wait/total:seconds Go runtime metric as go_sync_mutex_wait_total_seconds_total from all components. #5879
• [ENHANCEMENT] Query-scheduler: improve latency with many concurrent queriers. #5880
• [ENHANCEMENT] Ruler: add new per-tenant cortex_ruler_queries_zero_fetched_series_total metric to track rules that fetched no series. #5925
• [ENHANCEMENT] Implement support for limit, limit_per_metric and metric parameters for <Prometheus HTTP prefix>/api/v1/metadata endpoint. #5890
• [ENHANCEMENT] Distributor: add experimental support for storing metadata when ingesting metrics via OTLP. This makes metrics description and type available when ingesting metrics via OTLP. Enable with -distributor.enable-otlp-metadata-storage=true. #5693 #6035 #6254
• [ENHANCEMENT] Ingester: added support for sampling errors, which can be enabled by setting -ingester.error-sample-rate. This way each error will be logged once in the configured number of times. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. #5584 #6014
• [ENHANCEMENT] Ruler: Fetch secrets used to configure TLS on the Alertmanager client from Vault when -vault.enabled is true. #5239
• [ENHANCEMENT] Query-frontend: added query-sharding support for group by aggregation queries. #6024
• [ENHANCEMENT] Fetch secrets used to configure server-side TLS from Vault when -vault.enabled is true. #6052.
• [ENHANCEMENT] Packaging: add logrotate config file. #6142
• [ENHANCEMENT] Ingester: add the experimental configuration options -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes to enforce a limit in bytes on the PostingsForMatchers() cache used by ingesters (the cache limit is per TSDB head and block basis, not a global one). The experimental configuration options -blocks-storage.tsdb.head-postings-for-matchers-cache-size and -blocks-storage.tsdb.block-postings-for-matchers-cache-size have been deprecated. #6151
• [ENHANCEMENT] Ingester: use the PostingsForMatchers() in-memory cache for label values queries with matchers too. #6151
• [ENHANCEMENT] Ingester / store-gateway: optimized regex matchers. #6168 #6250
• [ENHANCEMENT] Distributor: Include ingester IDs in circuit breaker related metrics and logs. #6206
• [ENHANCEMENT] Querier: improve errors and logging when streaming chunks from ingesters and store-gateways. #6194 #6309
• [ENHANCEMENT] Querier: Add cortex_querier_federation_exemplar_tenants_queried and cortex_querier_federation_tenants_queried metrics to track the number of tenants queried by multi-tenant queries. #6374 #6409
• [ENHANCEMENT] All: added an experimental -server.grpc.num-workers flag that configures the number of long-living workers used to process gRPC requests. This could decrease the CPU usage by reducing the number of stack allocations. #6311
• [ENHANCEMENT] All: improved IPv6 support by using the proper host:port formatting. #6311
• [ENHANCEMENT] Querier: always return error encountered during chunks streaming, rather than the stream has already been exhausted. #6345 #6433
• [ENHANCEMENT] Query-frontend: add instance_enable_ipv6 to support IPv6. #6111
• [ENHANCEMENT] Store-gateway: return same detailed error messages as queriers when chunks or series limits are reached. #6347
• [ENHANCEMENT] Querier: reduce memory consumed for queries that hit store-gateways. #6348
• [ENHANCEMENT] Ruler: include corresponding trace ID with log messages associated with rule evaluation. #6379 #6520
• [ENHANCEMENT] Querier: clarify log messages and span events emitted while querying ingesters, and include both ingester name and address when relevant. #6381
• [ENHANCEMENT] Memcached: introduce new experimental configuration parameters -<prefix>.memcached.write-buffer-size-bytes -<prefix>.memcached.read-buffer-size-bytes to customise the memcached client write and read buffer size (the buffer is allocated for each memcached connection). #6468
• [ENHANCEMENT] Ingester, Distributor: added experimental support for rejecting push requests received via gRPC before reading them into memory, if ingester or distributor is unable to accept the request. This is activated by using -ingester.limit-inflight-requests-using-grpc-method-limiter for ingester, and -distributor.limit-inflight-requests-using-grpc-method-limiter for distributor. #5976 #6300
• [ENHANCEMENT] Add capability in store-gateways to accept number of tokens through config. -store-gateway.sharding-ring.num-tokens, default-value=512 #4863
• [ENHANCEMENT] Query-frontend: return warnings generated during query evaluation. #6391
• [ENHANCEMENT] Server: Add the option -server.http-read-header-timeout to enable specifying a timeout for reading HTTP request headers. It defaults to 0, in which case reading of headers can take up to -server.http-read-timeout, leaving no time for reading body, if there's any. #6517
• [ENHANCEMENT] Add connection-string option, -<prefix>.azure.connection-string, for Azure Blob Storage. #6487
• [ENHANCEMENT] Ingester: Add -ingester.instance-limits.max-inflight-push-requests-bytes. This limit protects the ingester against requests that together may cause an OOM. #6492
• [ENHANCEMENT] Ingester: add new per-tenant cortex_ingester_local_limits metric to expose the calculated local per-tenant limits seen at each ingester. Exports the local per-tenant series limit with label {limit="max_global_series_per_user"} #6403
• [ENHANCEMENT] Query-frontend: added "queue_time_seconds" field to "query stats" log. This is total time that query and subqueries spent in the queue, before queriers picked it up. #6537
• [ENHANCEMENT] Server: Add -server.report-grpc-codes-in-instrumentation-label-enabled CLI flag to specify whether gRPC status codes should be used in status_code label of cortex_request_duration_seconds metric. It defaults to false, meaning that successful and erroneous gRPC status codes are represented with success and error respectively. #6562
• [ENHANCEMENT] Server: Add -ingester.client.report-grpc-codes-in-instrumentation-label-enabled CLI flag to specify whether gRPC status codes should be used in status_code label of cortex_ingester_client_request_duration_seconds metric. It defaults to false, meaning that successful and erroneous gRPC status codes are represented with 2xx and error respectively. #6562
• [ENHANCEMENT] Server: Add -server.http-log-closed-connections-without-response-enabled option to log details about connections to HTTP server that were closed before any data was sent back. This can happen if client doesn't manage to send complete HTTP headers before timeout. #6612
• [ENHANCEMENT] Query-frontend: include length of query, time since the earliest and latest points of a query, time since the earliest and latest points of a query, cached/uncached bytes in "query stats" logs. Time parameters (start/end/time) are always formatted as RFC3339 now. #6473 #6477 #6709 #6710
• [ENHANCEMENT] Distributor: added support for reducing the resolution of native histogram samples upon ingestion if the sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default and can be turned off by setting -validation.reduce-native-histogram-over-max-buckets to false. #6535
• [ENHANCEMENT] Query-frontend: optionally wait for the frontend to complete startup if requests are received while the frontend is still starting. Disabled by default, set -query-frontend.not-running-timeout to a non-zero value to enable. #6621
• [ENHANCEMENT] Distributor: Include source IPs in OTLP push handler logs. #6652
• [ENHANCEMENT] Query-frontend: return clearer error message when a query request is received while shutting down. #6675
• [ENHANCEMENT] Querier: return clearer error message when a query request is cancelled by the caller. #6697
• [BUGFIX] Distributor: return server overload error in the event of exceeding the ingestion rate limit. #6549
• [BUGFIX] Ring: Ensure network addresses used for component hash rings are formatted correctly when using IPv6. #6068
• [BUGFIX] Query-scheduler: don't retain connections from queriers that have shut down, leading to gradually increasing enqueue latency over time. #6100 #6145
• [BUGFIX] Ingester: prevent query logic from continuing to execute after queries are canceled. #6085
• [BUGFIX] Ensure correct nesting of children of the querier.Select tracing span. #6085
• [BUGFIX] Packaging: fix preremove script preventing upgrades on RHEL based OS. #6067
• [BUGFIX] Querier: return actual error rather than attempted to read series at index XXX from stream, but the stream has already been exhausted (or even no error at all) when streaming chunks from ingesters or store-gateways is enabled and an error occurs while streaming chunks. #6346
• [BUGFIX] Querier: reduce log volume when querying ingesters with zone-awareness enabled and one or more instances in a single zone unavailable. #6381
• [BUGFIX] Querier: don't try to query further ingesters if ingester query request minimization is enabled and a query limit is reached as a result of the responses from the initial set of ingesters. #6402
• [BUGFIX] Ingester: Don't cache context cancellation error when querying. #6446
• [BUGFIX] Ingester: don't ignore errors encountered while iterating through chunks or samples in response to a query request. #6469
• [BUGFIX] All: fix issue where traces for some inter-component gRPC calls would incorrectly show the call as failing due to cancellation. #6470
• [BUGFIX] Querier: correctly mark streaming requests to ingesters or store-gateways as successful, not cancelled, in metrics and traces. #6471 #6505
• [BUGFIX] Querier: fix issue where queries fail with "context canceled" error when an ingester or store-gateway fails healthcheck while the query is in progress. #6550
• [BUGFIX] Tracing: When creating an OpenTelemetry tracing span, add it to the context for later retrieval. #6614
• [BUGFIX] Querier: always report query results to query-frontends, even when cancelled, to ensure query-frontends don't wait for results that will otherwise never arrive. #6703
• [BUGFIX] Querier: attempt to query ingesters in PENDING state, to reduce the likelihood that scaling up the number of ingesters in multiple zones simultaneously causes a read outage. #6726 #6727
• [BUGFIX] Querier: don't cancel inflight queries from a query-scheduler if the stream between the querier and query-scheduler is broken. #6728
• [BUGFIX] Store-gateway: Fix double-counting of some duration metrics. #6616
• [BUGFIX] Fixed possible series matcher corruption leading to wrong series being included in query results. #6884

Mixin

• [CHANGE] Dashboards: enabled reporting gRPC codes as status_code label in Mimir dashboards. In case of gRPC calls, the successful status_code label on cortex_request_duration_seconds and gRPC client request duration metrics has changed from 'success' and '2xx' to 'OK'. #6561
• [CHANGE] Alerts: remove MimirGossipMembersMismatch alert and replace it with MimirGossipMembersTooHigh and MimirGossipMembersTooLow alerts that should have a higher signal-to-noise ratio. #6508
• [ENHANCEMENT] Dashboards: Optionally show rejected requests on Mimir Writes dashboard. Useful when used together with "early request rejection" in ingester and distributor. #6132 #6556
• [ENHANCEMENT] Alerts: added a critical alert for CompactorSkippedBlocksWithOutOfOrderChunks when multiple blocks are affected. #6410
• [ENHANCEMENT] Dashboards: Added the min-replicas for autoscaling dashboards. #6528
• [BUGFIX] Alerts: fixed issue where GossipMembersMismatch warning message referred to per-instance labels that were not produced by the alert query. #6146
• [BUGFIX] Dashboards: Fix autoscaling dashboard panels for KEDA > 2.9. Requires scraping the KEDA operator for metrics since they moved. #6528
• [BUGFIX] Alerts: Fix autoscaling alerts for KEDA > 2.9. Requires scraping the KEDA operator for metrics since they moved. #6528

Jsonnet

• [CHANGE] Ingester: reduce -server.grpc-max-concurrent-streams to 500. #5666
• [CHANGE] Changed default _config.cluster_domain from cluster.local to cluster.local. to reduce the number of DNS lookups made by Mimir. #6389
• [CHANGE] Query-frontend: changed default _config.autoscaling_query_frontend_cpu_target_utilization from 1 to 0.75. #6395
• [CHANGE] Distributor: Increase HPA scale down period such that distributors are slower to scale down after autoscaling up. #6589
• [FEATURE] Store-gateway: Allow automated zone-by-zone downscaling, that can be enabled via the store_gateway_automated_downscale_enabled flag. It is disabled by default. #6149
• [FEATURE] Ingester: Allow to configure TSDB Head early compaction using the following _config parameters: #6181
  • ingester_tsdb_head_early_compaction_enabled (disabled by default)
  • ingester_tsdb_head_early_compaction_reduction_percentage
  • ingester_tsdb_head_early_compaction_min_in_memory_series
• [ENHANCEMENT] Double the amount of rule groups for each user tier. #5897
• [ENHANCEMENT] Set maxUnavailable to 0 for distributor, overrides-exporter, querier, query-frontend, query-scheduler ruler-querier, ruler-query-frontend, ruler-query-scheduler and consul deployments, to ensure they don't become completely unavailable during a rollout. #5924
• [ENHANCEMENT] Update rollout-operator to v0.9.0. #6022 #6110 #6558 #6681
• [ENHANCEMENT] Update memcached to memcached:1.6.22-alpine. #6585
• [ENHANCEMENT] Store-gateway: replaced the following deprecated CLI flags: #6319
  • -blocks-storage.bucket-store.index-header-lazy-loading-enabled replaced with -blocks-storage.bucket-store.index-header.lazy-loading-enabled
  • -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout replaced with -blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout
• [ENHANCEMENT] Store-gateway: Allow selective enablement of store-gateway automated scaling on a per-zone basis. #6302
• [BUGFIX] Autoscaling: KEDA > 2.9 removed the ability to set metricName in the trigger metadata. To help discern which metric is used by the HPA, we set the trigger name to what was the metricName. This is available as the scaler label on keda_* metrics. #6528

Mimirtool

• [ENHANCEMENT] Analyze Grafana: Improve support for variables in range. #6657
• [BUGFIX] Fix out of bounds error on export with large timespans and/or series count. #5700
• [BUGFIX] Fix the issue where --read-timeout was applied to the entire mimirtool analyze grafana invocation rather than to individual Grafana API calls. #5915
• [BUGFIX] Fix incorrect remote-read path joining for mimirtool remote-read commands on Windows. #6011
• [BUGFIX] Fix template files full path being sent in mimirtool alertmanager load command. #6138
• [BUGFIX] Analyze rule-file: .metricsUsed field wasn't populated. #6953

Mimir Continuous Test

Query-tee

Documentation

• [ENHANCEMENT] Document the concept of native histograms and how to send them to Mimir, migration path. #5956 #6488 #6539
• [ENHANCEMENT] Document native histograms query and visualization. #6231

Tools

• [CHANGE] tsdb-index: Rename tool to tsdb-series. #6317
• [FEATURE] tsdb-labels: Add tool to print label names and values of a TSDB block. #6317
• [ENHANCEMENT] trafficdump: Trafficdump can now parse OTEL requests. Entire request is dumped to output, there's no filtering of fields or matching of series done. #6108

All changes in this release: mimir-2.10.4...mimir-2.11.0-rc.0