Changes
v1.79.0
Released at 14-07-2022
Update note 1: this release introduces backwards-incompatible changes to vm_partial_results_total
metric by changing its labels to be consistent with vm_requests_total
metric. If you use alerting rules or Grafana dashboards, which rely on this metric, then they must be updated. The official dashboards for VictoriaMetrics don't use this metric.
Update note 2: vmalert adds /vmalert/
prefix to web urls according to this issue. This may affect vmalert
instances with non-empty -http.pathPrefix
command-line flag. After the update, configuring this flag is no longer needed. Here's why.
Update note 3: this release introduces backwards-incompatible changes to communication protocol between vmselect
and vmstorage
nodes in cluster version of VictoriaMetrics because of added ability to query vmselect
data from other vmselect
nodes - see these docs, so read requests to vmselect
will fail until the upgrade is complete. These errors will stop after all the vmselect
and vmstorage
nodes are updated to the new release. It is safe to downgrade to previous releases at any time.
Update note 4: this release removes support of deprecated in 1.70.0 param extra_filter_labels
from vmalert's groups definition. This deprecated param was replaced with params.
Update note 5: this release changes naming for published linux binaries at releases. Now names for binaries for all the supported platforms match the following template - $(APP_NAME)-$(GOOS)-$(GOARCH)-$(VERSION).tar.gz
. For example, victoria-metrics-linux-amd64-v1.79.0.tar.gz
. Previously linux binaries didn't have $(GOOS)
part, e.g. they had the name victoria-metrics-amd64-v1.79.0.tar.gz
. Please update automation scripts for upgrading VictoriaMetrics releases according to this change.
- FEATURE: vmagent: add azure_sd_configs service discovery mechanism. It allows discovering Virtual Machines at Azure Cloud. See this issue.
- FEATURE: vmalert: deprecate alert's status link
/api/v1/<groupID>/<alertID>/status
in favour of/api/v1/alert?group_id=<group_id>&alert_id=<alert_id>
. The old alert's status link is still supported, but will be removed in future releases. See this issue. - FEATURE: cluster version of VictoriaMetrics: add support for querying lower-level
vmselect
nodes from upper-levelvmselect
nodes. This makes possible to build multi-level cluster setups for global querying view and HA purposes without the need to use Promxy. See these docs and this issue. - FEATURE: add
-search.setLookbackToStep
command-line flag, which enables InfluxDB-like gap filling during querying. See these docs for details. - FEATURE: vmui: add an UI for query tracing. It can be enabled by clicking
trace query
checkbox and re-running the query. See this feature request. - FEATURE: vmagent: add
-remoteWrite.headers
command-line option for specifying optional HTTP headers to send to the configured-remoteWrite.url
. For example,-remoteWrite.headers='Foo:Bar^^Baz:x'
would sendFoo: Bar
andBaz: x
HTTP headers with every request to-remoteWrite.url
. See this feature request. - FEATURE: vmagent: push per-target
scrape_samples_limit
metric to the cofigured-remoteWrite.url
ifsample_limit
option is set for this target in scrape_config. See this feature request. - FEATURE: vmagent: attach node-level labels to kubernetes_sd_config targets if
attach_metadata: {"node": true}
is set forrole: endpoints
androle: endpointslice
. This is a feature backport from Prometheus 2.37 - see this pull request. - FEATURE: vmagent: add ability to specify additional HTTP headers to send to scrape targets via
headers
section inscrape_configs
. This can be used when the scrape target requires custom authorization and authentication like in this stackoverflow question. For example, the following config instructs sendingMy-Auth: top-secret
andTenantID: FooBar
headers with each request tohttp://host123:8080/metrics
:
scrape_configs:
- job_name: foo
headers:
- "My-Auth: top-secret"
- "TenantID: FooBar"
static_configs:
- targets: ["host123:8080"]
-
FEATURE: add ability to pass
limit
query arg toapi/v1/series
endpoint. This can be used if only a sample of up tolimit
series must be returned from the endpoint. See this feature request and these docs. -
FEATURE: query tracing: show timestamps in query traces in human-readable format (aka
RFC3339
in UTC timezone) instead of milliseconds since Unix epoch. For example,2022-06-27T10:32:54.506Z
instead of1656325974506
. This improves traces' readability. -
FEATURE: improve performance of /api/v1/series requests, which return big number of time series.
-
FEATURE: VictoriaMetrics cluster: improve query performance when replication is enabled.
-
FEATURE: MetricsQL: properly handle partial counter resets in remove_resets function. Now
remove_resets(sum(m))
should returns the expected increasing line when some time series matchingm
disappear on the selected time range. Previously such a query would return horizontal line after the disappeared series. -
FEATURE: expose
vm_next_retention_seconds
metric athttp://victoriametrics:8428/metrics
, which shows the number of seconds left until the nextindexdb
rotation. Thanks to @guidao for the pull request. -
FEATURE: expose additional histogram metrics at
http://victoriametrics:8428/metrics
, which may help understanding query workload:vm_rows_read_per_query
- the number of raw samples read per query.vm_rows_scanned_per_query
- the number of raw samples scanned per query. This number can exceedvm_rows_read_per_query
ifstep
query arg passed to /api/v1/query_range is smaller than the lookbehind window set in square brackets of rollup function. For example, ifincrease(some_metric[1h])
is executed with thestep=5m
, then the same raw samples on a hour time range are scanned1h/5m=12
times. See this article for details.vm_rows_read_per_series
- the number of raw samples read per queried series.vm_series_read_per_query
- the number of series read per query.
-
FEATURE: publish binaries for FreeBSD and OpenBSD at releases page.
-
FEATURE: vmui: allow selecting the needed columns at table view. This functionaly may help when the selected time series contain many different labels. See this feature request and this pull request.
-
BUGFIX: consistently name binaries at releases page in the form
$(APP_NAME)-$(GOOS)-$(GOARCH)-$(VERSION).tar.gz
. For example,victoria-metrics-linux-amd64-v1.79.0.tar.gz
. Previously the$(GOOS)
part was missing in binaries for Linux. -
BUGFIX: vmalert: allow using
__name__
label (aka metric name) in alerting annotations. For example:
{{ $labels.__name__ }}: Too high connection number for "{{ $labels.instance }}
- BUGFIX: limit max memory occupied by the cache, which stores parsed regular expressions. Previously too long regular expressions passed in MetricsQL queries could result in big amounts of used memory (e.g. multiple of gigabytes). Now the max cache size for parsed regexps is limited to a a few megabytes.
- BUGFIX: MetricsQL: properly handle partial counter resets when calculating rate, irate and increase functions. Previously these functions could return zero values after partial counter resets until the counter increases to the last value before partial counter reset. See this issue.
- BUGFIX: MetricsQL: properly calculate histogram_quantile over Prometheus buckets with unexpected values. See this issue.
- BUGFIX: MetricsQL: properly evaluate timezone_offset function over time range covering time zone offset switches. See this issue.
- BUGFIX: vmagent: properly add service-level labels (
__meta_kubernetes_service_*
) to discovered targets forrole: endpointslice
in kubernetes_sd_config. Previously these labels were missing. See this issue. - BUGFIX: vmagent: make sure that stale markers are generated with the actual timestamp when unsuccessful scrape occurs. This should prevent from possible time series overlap on scrape target restart in dynmaic envirnoments such as Kubernetes.
- BUGFIX: vmagent: properly reload changed
-promscrape.config
file when-promscrape.configCheckInterval
option is set. The changed config file wasn't reloaded in this case since v1.69.0. See this pull request. Thanks to @ttyv for the fix. - BUGFIX: vmagent: properly set
Host
header during target scraping whenproxy_url
is set to http proxy. Previously theHost
header was set to the proxy hostname instead of the target hostname. See this issue. - BUGFIX: VictoriaMetrics cluster: assume that the response is complete if
-search.denyPartialResponse
is enabled and up to-replicationFactor - 1
vmstorage
nodes are unavailable. See this issue. - BUGFIX: vmselect: update
vm_partial_results_total
metric labels to be consistent withvm_requests_total
labels. - BUGFIX: accept tags without values when reading data in DataDog format. Thanks to @PerGon for the pull request.
- BUGFIX: vmui: properly pass the end of the selected time range to
time
query arg to /api/v1/query when displaying the requested data in JSON and table views. Previously thetime
query arg wasn't set, so/api/v1/query
was always returning query results for the current time regardless of the selected time range. See this issue. - BUGFIX: vmui: allow clicking on the suggestion from autocomplete list. See this issue.
- BUGFIX: vmui: apply the selected time range in date picker only after clicking the
Apply
button. See this issue.
See full changelog.
How to run VictoriaMetrics
Unpack the victoria-metrics-*.tar.gz
archive and read these docs.
vmutils-*.tag.gz
archive contains the following tools:
vmutils-*-enterprise.tar.gz
archive contains the following additional enterprise tools:
The corresponding docker images are available here.
Cluster version is available here.