This release contains 168 contributions from 29 contributors. We also have 16 new contributors. Thank you all for the contributions!
Some notable changes release are:
- Experimental OTLP ingestion
- Experimental minimize spread token generator strategy on Ingester
- Advanced query scheduling with Query Priority
- ListRules API high availability by rule group replication and backup
- Various improvements on Store Gateway Index Cache
mem-ballast-size-bytes
flag has been marked as deprecated and not functional anymore-querier.ingester-streaming
flag has been marked as deprecated and ingester streaming is always enabled nowquerier.iterators
andquerier.batch-iterators
flags have been marked as deprecated and batch iterator is always enabled in Querier now
Cortex
- [CHANGE] Azure Storage: Upgraded objstore dependency and support Azure Workload Identity Authentication. Added
connection_string
to support authenticating via SAS token. Markedmsi_resource
config as deprecating. #5645 - [CHANGE] Store Gateway: Add a new fastcache based inmemory index cache. #5619
- [CHANGE] Index Cache: Multi level cache backfilling operation becomes async. Added
-blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency
and-blocks-storage.bucket-store.index-cache.multilevel.max-async-buffer-size
configs and metriccortex_store_multilevel_index_cache_backfill_dropped_items_total
for number of dropped items. #5661 - [CHANGE] Ingester: Disable uploading compacted blocks and overlapping compaction in ingester. #5735
- [CHANGE] Distributor: Count the number of rate-limited samples in
distributor_samples_in_total
. #5714 - [CHANGE] Ruler: Remove
cortex_ruler_write_requests_total
,cortex_ruler_write_requests_failed_total
,cortex_ruler_queries_total
,cortex_ruler_queries_failed_total
, andcortex_ruler_query_seconds_total
metrics for the tenant when the ruler deletes the manager for the tenant. #5772 - [CHANGE] Main: Mark
mem-ballast-size-bytes
flag as deprecated. #5816 - [CHANGE] Querier: Mark
-querier.ingester-streaming
flag as deprecated. Now query ingester streaming is always enabled. #5817 - [CHANGE] Compactor/Bucket Store: Added
-blocks-storage.bucket-store.block-discovery-strategy
to configure different block listing strategy. Reverted the current recursive block listing mechanism and use the strategyConcurrent
as in 1.15. #5828 - [CHANGE] Compactor: Don't halt compactor when overlapped source blocks detected. #5854
- [CHANGE] S3 Bucket Client: Expose
-blocks-storage.s3.send-content-md5
flag and set default checksum algorithm to MD5. #5870 - [CHANGE] Querier: Mark
querier.iterators
andquerier.batch-iterators
flags as deprecated. Now querier always use batch iterators. #5868 - [FEATURE] OTLP ingestion experimental. #5813
- [FEATURE] Ingester: Add per-tenant new metric
cortex_ingester_tsdb_data_replay_duration_seconds
. #5477 - [FEATURE] Query Frontend/Scheduler: Add query priority support. #5605
- [FEATURE] Tracing: Add
kuberesolver
to resolve endpoints address withkubernetes://
prefix as Kubernetes service. #5731 - [FEATURE] Tracing: Add
tracing.otel.round-robin
flag to useround_robin
gRPC client side LB policy for sending OTLP traces. #5731 - [FEATURE] Ruler: Add
ruler.concurrent-evals-enabled
flag to enable concurrent evaluation within a single rule group for independent rules. Maximum concurrency can be configured viaruler.max-concurrent-evals
. #5766 - [FEATURE] Distributor Queryable: Experimental: Add config
zone_results_quorum_metadata
. When querying ingesters using metadata APIs such as label names and values, only results from quorum number of zones will be included and merged. #5779 - [FEATURE] Storage Cache Clients: Add config
set_async_circuit_breaker_config
to utilize the circuit breaker pattern for dynamically thresholding asynchronous set operations. Implemented in both memcached and redis cache clients. #5789 - [FEATURE] Ruler: Add experimental
experimental.ruler.api-deduplicate-rules
flag to remove duplicate rule groups from the Prometheus compatible rules API endpoint. Add experimentalruler.ring.replication-factor
andruler.ring.zone-awareness-enabled
flags to configure rule group replication, but only the first ruler in the replicaset evaluates the rule group, the rest will just hold a copy as backup. Add experimentalexperimental.ruler.api-enable-rules-backup
flag to configure rulers to send the rule group backups stored in the replicaset to handle events when a ruler is down during an API request to list rules. #5782 - [ENHANCEMENT] Store Gateway: Added
-store-gateway.enabled-tenants
and-store-gateway.disabled-tenants
to explicitly enable or disable store-gateway for specific tenants. #5638 - [ENHANCEMENT] Compactor: Add new compactor metric
cortex_compactor_start_duration_seconds
. #5683 - [ENHANCEMENT] Index Cache: Multi level cache adds config
max_backfill_items
to cap max items to backfill per async operation. #5686 - [ENHANCEMENT] Query Frontend: Log number of split queries in
query stats
log. #5703 - [ENHANCEMENT] Logging: Added new options for logging HTTP request headers:
-server.log-request-headers
enables logging HTTP request headers,-server.log-request-headers-exclude-list
allows users to specify headers which should not be logged. #5744 - [ENHANCEMENT] Query Frontend/Scheduler: Time check in query priority now considers overall data select time window (including range selectors, modifiers and lookback delta). #5758
- [ENHANCEMENT] Querier: Added
querier.store-gateway-query-stats-enabled
to enable or disable store gateway query stats log. #5749 - [ENHANCEMENT] AlertManager: Retrying AlertManager Delete Silence on error #5794
- [ENHANCEMENT] Ingester: Add new ingester metric
cortex_ingester_max_inflight_query_requests
. #5798 - [ENHANCEMENT] Query: Added
query_storage_wall_time
to Query Frontend and Ruler query stats log for wall time spent on fetching data from storage. Query evaluation is not included. #5799 - [ENHANCEMENT] Query: Added additional max query length check at Query Frontend and Ruler. Added
-querier.ignore-max-query-length
flag to disable max query length check at Querier. #5808 - [ENHANCEMENT] Querier: Add context error check when converting Metrics to SeriesSet for GetSeries on distributorQuerier. #5827
- [ENHANCEMENT] Ruler: Improve GetRules response time by refactoring mutexes and introducing a temporary rules cache in
ruler/manager.go
. #5805 - [ENHANCEMENT] Querier: Add context error check when merging slices from ingesters for GetLabel operations. #5837
- [ENHANCEMENT] Ring: Add experimental
-ingester.tokens-generator-strategy=minimize-spread
flag to enable the new minimize spread token generator strategy. #5855 - [ENHANCEMENT] Query Frontend: Ensure error response returned by Query Frontend follows Prometheus API error response format. #5811
- [ENHANCEMENT] Ring Status Page: Add
Ownership Diff From Expected
column in the ring table to indicate the extent to which the ownership of a specific ingester differs from the expected ownership. #5889 - [BUGFIX] Distributor: Do not use label with empty values for sharding #5717
- [BUGFIX] Query Frontend: queries with negative offset should check whether it is cacheable or not. #5719
- [BUGFIX] Redis Cache: pass
cache_size
config correctly. #5734 - [BUGFIX] Distributor: Shuffle-Sharding with IngestionTenantShardSize == 0, default sharding strategy should be used #5189
- [BUGFIX] Cortex: Fix GRPC stream clients not honoring overrides for call options. #5797
- [BUGFIX] Ring DDB: Fix lifecycle for ring counting unhealthy pods as healthy. #5838
- [BUGFIX] Ring DDB: Fix region assignment. #5842
New Contributors
- @testwill made their first contribution in #5644
- @dsabsay made their first contribution in #5684
- @pawarpranav83 made their first contribution in #5719
- @Kramer0x0 made their first contribution in #5743
- @tesla59 made their first contribution in #5746
- @blorby made their first contribution in #5767
- @CharlieTLe made their first contribution in #5784
- @lekaf974 made their first contribution in #5793
- @euniceek made their first contribution in #5794
- @mustafain117 made their first contribution in #5823
- @availhang made their first contribution in #5826
- @erlan-z made their first contribution in #5827
- @yj-yoo made their first contribution in #5775
- @kindknow made their first contribution in #5856
- @momantech made their first contribution in #5863
- @till made their first contribution in #5874
Full Changelog: v1.16.1...v1.17.0