VictoriaMetrics/VictoriaMetrics v1.60.0 on GitHub

Changes

FEATURE: add ability to limit the number of unique time series, which can be added to storage per hour and per day. This can help dealing with high cardinality and high churn rate issues. See these docs.
FEATURE: vmagent: add ability to limit the number of unique time series, which can be sent to remote storage systems per hour and per day. This can help dealing with high cardinality and high churn rate issues. See these docs.
FEATURE: vmalert: add ability to run alerting and recording rules for multiple tenants. See this issue and these docs.
FEATURE: vminsert: add support for data ingestion via other vminsert nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of vminsert nodes to another level of vminsert nodes. See these docs and this comment for details.
FEATURE: vmagent: reload bearer_token_file, credentials_file and password_file contents every second. This allows dynamically changing the contents of these files during target scraping and service discovery without the need to restart vmagent. See this issue.
FEATURE: vmalert: add a flag to control behaviour on startup for state restore errors. Such errors were returned and logged before as well. Now user can specify whether to just log these errors (-remoteRead.ignoreRestoreErrors=true) or to stop the process (-remoteRead.ignoreRestoreErrors=false). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. See this issue.
FEATURE: vmalert: add ability to pass round_digits query arg to datasource via -datasource.roundDigits command-line flag. This can be used for limiting the number of decimal digits after the point in recording rule results. See this issue.
FEATURE: return X-Server-Hostname header in http responses of all the VictoriaMetrics components. This should simplify tracing the origin server behind a load balancer or behind auth proxy during troubleshooting.
FEATURE: vmselect: allow to use 2x more memory for query processing at vmselect nodes in VictoriaMetrics cluster. This should allow processing heavy queries without the need to increase RAM size at vmselect nodes.
FEATURE: add ability to filter /api/v1/status/tsdb output with arbitrary time series selectors passed via match[] query args. See these docs and this issue for details.
FEATURE: automatically detect memory and cpu limits for VictoriaMetrics components running under cgroup v2 environments such as HashiCorp Nomad. See this issue.
FEATURE: vmauth: allow -auth.config reloading via /-/reload http endpoint. See this issue.
FEATURE: add timezone_offset(tz) function. It returns offset in seconds for the given timezone tz relative to UTC. This can be useful when combining with datetime-related functions. For example, day_of_week(time()+timezone_offset("America/Los_Angeles")) would return weekdays for America/Los_Angeles time zone. Special Local time zone can be used for returning an offset for the time zone set on the host where VictoriaMetrics runs. See this issue and MetricsQL docs for more details.
FEATURE: vmagent: add support for OAuth2 authorization for scrape targets and service discovery in the same way as Prometheus does. See these docs.
FEATURE: vmagent: add support for OAuth2 authorization when writing data to -remoteWrite.url. See -remoteWrite.oauth2.* config params in /path/to/vmagent -help output.
FEATURE: vmalert: add ability to set extra_filter_labels at alerting and recording group configs. See these docs.
FEATURE: vmstorage: reduce memory usage by up to 30% when ingesting big number of active time series.
BUGFIX: vmagent: do not retry scraping targets, which don't support HTTP. This should reduce CPU load and network usage at vmagent and at scrape target. See this issue.
BUGFIX: vmagent: fix possible race when refreshing role: endpoints and role: endpointslices scrape targets in kubernetes_sd_config. Prevoiusly pod objects could be updated after the related endpoints object update. This could lead to missing scrape targets. See this issue.
BUGFIX: vmagent: properly spread scrape targets among vmagent replicas if -promscrape.cluster.replicationFactor exceeds 1. See this pull request.
BUGFIX: vmagent: limit scrape_timeout by scrape_interval. This guarantees that only a single sample is lost during the configured scrape_interval when scrape target responds slowly. See this comment for details.
BUGFIX: properly remove stale parts outside the configured retention if -retentionPeriod is smaller than one month. Previously stale parts could remain active for up to a month after they go outside the retention.
BUGFIX: stop the process on panic errors, since such errors may leave the process in inconsistent state. Previously panics could be recovered, which could result in unexpected hard-to-debug further behavior of running process.
BUGFIX: vminsert, vmagent: make sure data ingestion connections are closed before completing graceful shutdown. Previously the connection may remain open, which could result in trailing samples loss.
BUGFIX: vmauth, vmalert: properly re-use HTTP keep-alive connections to backends and datasources. Previously only 2 keep-alive connections per backend could be re-used. Other connections were closed after the first request. See this issue for details.
BUGFIX: vmalert: fix false positive error result contains metrics with the same labelset after applying rule labels, which could be triggered when recording rules generate unique metrics. See this issue.
BUGFIX: vmctl: properly import InfluxDB rows if they have a field and a tag with identical names. See this issue.
BUGFIX: properly reload configs if SIGHUP signal arrives during service initialization. Previously such SIGHUP signal could be ingonred and configs weren't reloaded.
BUGFIX: vmalert: properly import default rules from OpenShift. See this issue.
BUGFIX: reduce the probability of the removal queue is full panic when highly loaded VictoriaMetrics stores data on NFS. See this issue.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

The corresponding docker images are available here.
Cluster version is available here.