Changelog
Cortex
- [CHANGE] Query Frontend: deprecate
-querier.compress-http-responses
in favour of-api.response-compression-enabled
. #3544 - [CHANGE] Querier: deprecated
-store.max-look-back-period
. You should use-querier.max-query-lookback
instead. #3452 - [CHANGE] Blocks storage: increased
-blocks-storage.bucket-store.chunks-cache.attributes-ttl
default from24h
to168h
(1 week). #3528 - [CHANGE] Blocks storage: the config option
-blocks-storage.bucket-store.index-cache.postings-compression-enabled
has been deprecated and postings compression is always enabled. #3538 - [CHANGE] Ruler: gRPC message size default limits on the Ruler-client side have changed: #3523
- limit for outgoing gRPC messages has changed from 2147483647 to 16777216 bytes
- limit for incoming gRPC messages has changed from 4194304 to 104857600 bytes
- [FEATURE] Distributor/Ingester: Provide ability to not overflow writes in the presence of a leaving or unhealthy ingester. This allows for more efficient ingester rolling restarts. #3305
- [FEATURE] Query-frontend: introduced query statistics logged in the query-frontend when enabled via
-frontend.query-stats-enabled=true
. When enabled, the metriccortex_query_seconds_total
is tracked, counting the sum of the wall time spent across all queriers while running queries (on a per-tenant basis). The metricscortex_request_duration_seconds
andcortex_query_seconds_total
are different: the first one tracks the request duration (eg. HTTP request from the client), while the latter tracks the sum of the wall time on all queriers involved executing the query. #3539 - [ENHANCEMENT] API: Add GZIP HTTP compression to the API responses. Compression can be enabled via
-api.response-compression-enabled
. #3536 - [ENHANCEMENT] Added zone-awareness support on queries. When zone-awareness is enabled, queries will still succeed if all ingesters in a single zone will fail. #3414
- [ENHANCEMENT] Blocks storage ingester: exported more TSDB-related metrics. #3412
cortex_ingester_tsdb_wal_corruptions_total
cortex_ingester_tsdb_head_truncations_failed_total
cortex_ingester_tsdb_head_truncations_total
cortex_ingester_tsdb_head_gc_duration_seconds
- [ENHANCEMENT] Enforced keepalive on all gRPC clients used for inter-service communication. #3431
- [ENHANCEMENT] Added
cortex_alertmanager_config_hash
metric to expose hash of Alertmanager Config loaded per user. #3388 - [ENHANCEMENT] Query-Frontend / Query-Scheduler: New component called "Query-Scheduler" has been introduced. Query-Scheduler is simply a queue of requests, moved outside of Query-Frontend. This allows Query-Frontend to be scaled separately from number of queues. To make Query-Frontend and Querier use Query-Scheduler, they need to be started with
-frontend.scheduler-address
and-querier.scheduler-address
options respectively. #3374 #3471 - [ENHANCEMENT] Query-frontend / Querier / Ruler: added
-querier.max-query-lookback
to limit how long back data (series and metadata) can be queried. This setting can be overridden on a per-tenant basis and is enforced in the query-frontend, querier and ruler. #3452 #3458 - [ENHANCEMENT] Querier: added
-querier.query-store-for-labels-enabled
to query store for label names, label values and series APIs. Only works with blocks storage engine. #3461 #3520 - [ENHANCEMENT] Ingester: exposed
-blocks-storage.tsdb.wal-segment-size-bytes
config option to customise the TSDB WAL segment max size. #3476 - [ENHANCEMENT] Compactor: concurrently run blocks cleaner for multiple tenants. Concurrency can be configured via
-compactor.cleanup-concurrency
. #3483 - [ENHANCEMENT] Compactor: shuffle tenants before running compaction. #3483
- [ENHANCEMENT] Compactor: wait for a stable ring at startup, when sharding is enabled. #3484
- [ENHANCEMENT] Store-gateway: added
-blocks-storage.bucket-store.index-header-lazy-loading-enabled
to enable index-header lazy loading (experimental). When enabled, index-headers will be mmap-ed only once required by a query and will be automatically released after-blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout
time of inactivity. #3498 - [ENHANCEMENT] Alertmanager: added metrics
cortex_alertmanager_notification_requests_total
andcortex_alertmanager_notification_requests_failed_total
. #3518 - [ENHANCEMENT] Ingester: added
-blocks-storage.tsdb.head-chunks-write-buffer-size-bytes
to fine-tune the TSDB head chunks write buffer size when running Cortex blocks storage. #3518 - [ENHANCEMENT] /metrics now supports OpenMetrics output. HTTP and gRPC servers metrics can now include exemplars. #3524
- [ENHANCEMENT] Expose gRPC keepalive policy options by gRPC server. #3524
- [ENHANCEMENT] Blocks storage: enabled caching of
meta.json
attributes, configurable via-blocks-storage.bucket-store.metadata-cache.metafile-attributes-ttl
. #3528 - [ENHANCEMENT] Compactor: added a config validation check to fail fast if the compactor has been configured invalid block range periods (each period is expected to be a multiple of the previous one). #3534
- [ENHANCEMENT] Blocks storage: concurrently fetch deletion marks from object storage. #3538
- [ENHANCEMENT] Blocks storage ingester: ingester can now close idle TSDB and delete local data. #3491 #3552
- [ENHANCEMENT] Blocks storage: add option to use V2 signatures for S3 authentication. #3540
- [ENHANCEMENT] Exported process metrics to monitor the number of memory map areas allocated. #3537
-
process_memory_map_areas
-
process_memory_map_areas_limit
-
- [ENHANCEMENT] Ruler: Expose gRPC client options. #3523
- [ENHANCEMENT] Compactor: added metrics to track on-going compaction. #3535
cortex_compactor_tenants_discovered
cortex_compactor_tenants_skipped
cortex_compactor_tenants_processing_succeeded
cortex_compactor_tenants_processing_failed
- [ENHANCEMENT] Added new experimental API endpoints:
POST /purger/delete_tenant
andGET /purger/delete_tenant_status
for deleting all tenant data. Only works with blocks storage. Compactor removes blocks that belong to user marked for deletion. #3549 #3558 - [ENHANCEMENT] Chunks storage: add option to use V2 signatures for S3 authentication. #3560
- [BUGFIX] Blocks storage ingester: fixed some cases leading to a TSDB WAL corruption after a partial write to disk. #3423
- [BUGFIX] Blocks storage: Fix the race between ingestion and
/flush
call resulting in overlapping blocks. #3422 - [BUGFIX] Querier: fixed
-querier.max-query-into-future
which wasn't correctly enforced on range queries. #3452 - [BUGFIX] Fixed float64 precision stability when aggregating metrics before exposing them. This could have lead to false counters resets when querying some metrics exposed by Cortex. #3506
- [BUGFIX] Querier: the meta.json sync concurrency done when running Cortex with the blocks storage is now controlled by
-blocks-storage.bucket-store.meta-sync-concurrency
instead of the incorrect-blocks-storage.bucket-store.block-sync-concurrency
(default values are the same). #3531 - [BUGFIX] Querier: fixed initialization order of querier module when using blocks storage. It now (again) waits until blocks have been synchronized. #3551