Changes
-
FEATURE: vmalert: add web UI with the list of alerting groups, alerts and alert statuses. See this pull request.
-
FEATURE: vmalert: add
-rule.maxResolveDuration
command-line flag, which could be used for limiting the auto-resolve duration for the alerting rule. By default it is limited to 3x evaluation interval. This could be too high for big evaluation intervals. See this issue. -
FEATURE: vmalert: add support for Bearer token authorization for
-datasource.url
,-remoteRead.url
and-remoteWrite.url
. See this issue. -
FEATURE: vmagent: send stale markers for disappeared metrics like Prometheus does. Previously stale markers were sent only when the scrape target disappears or when it becomes temporarily unavailable. See these docs for details.
-
FEATURE: vmagent: add ability to set
series_limit
option for a particular scrape target via__series_limit__
label. This allows setting the limit on the number of time series on a per-target basis. See these docs for details. -
FEATURE: vmagent: add ability to set
stream_parse
option for a particular scrape target via__stream_parse__
label. This allows managing the stream parsing mode on a per-target basis. See these docs for details. -
FEATURE: vmagent: add ability to set
scrape_interval
andscrape_timeout
options for a particular target via__scrape_interval__
and__scrape_timeout__
labels in the same way as Prometheus 2.30 does. See this pull request. -
FEATURE: vmagent: generate
scrape_timeout_seconds
metric per each scrape target, so the target saturation could be calculated withscrape_duration_seconds / scrape_timeout_seconds
. See the corresponding pull request from Prometheus 2.30. -
FEATURE: vmagent: reduce CPU usage when calculating the number of newly added series per scrape (this number is sent to remote storage in
scrape_series_added
metric). -
FEATURE: vmagent: reduce CPU usage when applying
series_limit
to scrape targets with constant set of metrics. See more information aboutseries_limit
here. -
FEATURE: vminsert: disable rerouting by default when a few of
vmstorage
nodes start accepting data at lower speed than the rest ofvmstorage
nodes. This should improve VictoriaMetrics cluster stability during rolling restarts and during spikes in time series churn rate. The rerouting can be enabled by passing-disableRerouting=false
command-line flag tovminsert
. -
FEATURE: vmauth: do not put invalid auth tokens into log by default due to security reasons. The logging can be returned back by passing
-logInvalidAuthTokens
command-line flag tovmauth
. Requests with invalid auth tokens are counted atvmagent_http_request_errors_total{reason="invalid_auth_token"}
metric exposed byvmauth
at/metrics
page. -
FEATURE: add new relabeling actions:
keep_metrics
anddrop_metrics
. This simplifies metrics filtering by metric names. See these docs for more details. -
FEATURE: allow splitting long
regex
in relabeling filters into an array of shorter regexps, which can be put into multiple lines for better readability and maintainability. See these docs for more details. -
FEATURE: optimize performance for queries with regexp filters on metric name like
{__name__=~"metric1|...|metricN"}
. See this pull request from @faceair. -
FEATURE: vmui: use Prometheus-compatible query args, so
vmui
could be accessed from graph editor in Grafana. See this pull request. Thanks to @Loori-R. -
FEATURE: vmselect: automatically add missing port to
-storageNode
hostnames. For example,-storageNode=vmstorage1,vmstorage2
is automatically translated to-storageNode=vmstorage1:8401,vmstorage2:8401
. This simplifies manual setup of VictoriaMetrics cluster. -
FEATURE: vminsert: automatically add missing port to
-storageNode
hostnames. For example,-storageNode=vmstorage1,vmstorage2
is automatically translated to-storageNode=vmstorage1:8400,vmstorage2:8400
. This simplifies manual setup of VictoriaMetrics cluster. -
FEATURE: add mad(q) function to MetricsQL. It calculates Median absolute deviation for groups of points with identical timestamps across multiple time series.
-
FEATURE: add outliers_mad(tolerance, q) function to MetricsQL. It returns time series with peaks outside the Median absolute deviation multiplied by
tolerance
. -
FEATURE: add
histogram_quantiles("phiLabel", phi1, ..., phiN, buckets)
function to MetricsQL. It calculates the givenphi*
-quantiles over the givenbuckets
and returns time series per each quantile with the corresponding{phiLabel="phi*"}
label. -
FEATURE: add
quantiles_over_time("phiLabel", phi1, ..., phiN, series_selector[d])
function to MetricsQL. It calculates the givenphi*
-quantiles over raw samples selected byseries_selector
on the given lookbehind windowd
. It returns time series per each quantile with the corresponding{phiLabel="phi*"}
label. -
FEATURE: enterprise: do not ask for
-eula
flag if-version
flag is passed to enteprise app. See this issue. -
BUGFIX: properly handle queries with multiple filters matching empty labels such as
metric{label1=~"foo|",label2="bar|"}
. This filter must match the following series:metric
,metric{label1="foo"}
,metric{label2="bar"}
andmetric{label1="foo",label2="bar"}
. Previously it was matching onlymetric{label1="foo",label2="bar"}
. See this issue. -
BUGFIX: vmselect: reset connection timeouts after each request to
vmstorage
. This should prevent fromcannot read data in 0.000 seconds: unexpected EOF
warning in logs. See this issue. Thanks to @mxlxm . -
BUGFIX: keep metric name for time series returned from rollup_candlestick function, since the returned series don't change the meaning of the original series. See this issue.
-
BUGFIX: use Prometheus-compatible label value formatting for count_values function. Previously big values could be improperly formatted, which could break query results, which rely on label value such as
... on(label) count_values("label", ...)
. -
BUGFIX: vmagent: properly use
https
scheme for wildcard TLS certificates forrole: ingress
targets in Kubernetes service discovery. See this issue. -
BUGFIX: vmagent: support host networking mode for
docker_sd_config
. See this issue. -
BUGFIX: fix non-repeatable results from
quantile_over_time()
function when the number of input samples exceeds 1000. See this issue. -
BUGFIX: vmagent: fix EC2 zone discovery when
filters
are specified in ec2_sc_config. See this issue.
See full changelog.
How to run VictoriaMetrics
Unpack the victoria-metrics-*.tar.gz
archive and read these docs.
vmutils-*.tag.gz
archive contains the following tools:
vmutils-*-enterprise.tar.gz
archive contains the following additional enterprise tools:
The corresponding docker images are available here.
Cluster version is available here.