VictoriaMetrics/VictoriaMetrics v1.98.0 on GitHub

Released at 2024-02-14

SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7,
plus the changelog for Go1.22.0.
FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at -httpListenAddr (aka mTLS). See these docs and this feature request.
FEATURE: all VictoriaMetrics components: add support for accepting http requests over multiple distinct TCP addresses. This can be done by starting VictoriaMetrics component with multiple -httpListenAddr command-line flags. For example, ./victoria-metrics -httpListenAddr=some-host:12345 -httpListenAddr=localhost:8428 starts VictoriaMetrics, which accepts incoming http requests at both some-host:12345 and localhost:8428. See this feature request.
FEATURE: all VictoriaMetrics components: add support for empty command-line flag values in short array notation. For example, -remoteWrite.sendTimeout=',20s,' specifies three -remoteWrite.sendTimeout values - the first one and the last one have default values (30s in this case), while the second one is set to 20s.
FEATURE: all VictoriaMetrics components: do not close connections to -httpListenAddr every 2 minutes. This behavior didn't help spreading load among multiple backend servers behind load-balancing TCP proxy. Instead, it could lead to hard-to-debug issues like this one. If you still need periodically closing client connections because of some reason, then pass the desired timeout to -http.connTimeout command-line flag.
FEATURE: vmauth: add support for mTLS-based request routing to different backends depending on the subject of the TLS certificate provided by the client. See these docs and this feature request.
FEATURE: vmagent and single-node VictoriaMetrics: add support for data ingestion via DataDog lambda extension aka /api/beta/sketches endpoint. See these docs and this feature request. Thanks to @AndrewChubatiuk for the pull request.
FEATURE: vmagent: add -remoteWrite.tlsHandshakeTimeout command-line flag for tuning the timeout needed for establishing TLS connections to -remoteWrite.url. Bigger tls handshake timeouts should reduce the probability of http: TLS handshake error from ...: EOF errors at the remote storage side under high load. See this issue.
FEATURE: VictoriaMetrics cluster: add -disableReroutingOnUnavailable command-line flag to vminsert, which can be used for reducing resource usage spikes at vmstorage nodes during rolling restart. See these docs. Thanks to @Muxa1L for the pull request.
FEATURE: add -search.resetRollupResultCacheOnStartup command-line flag for resetting query cache on startup. See this feature request.
FEATURE: MetricsQL: propagate label filters across label_set and alias functions. For example, label_set(q1, "a", "b") + q2{c="d"} is automatically transformed to label_set(q1{c="d"}, "a", "b") + q2{a="b",c="d"} now. This should improve performance for such queries. See this issue.
FEATURE: MetricsQL: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions. See this feature request.
FEATURE: dashboards/vmagent: add Targets scraped/s stat panel showing the number of targets scraped by the vmagent per-second.
FEATURE: dashboards/all: add new panel CPU spent on GC. It should help identifying cases when too much CPU is spent on garbage collection, and advice users on how this can be addressed.
FEATURE: vmalert: support filtering for /api/v1/rules API. See the pull request by @victoramsantos.
FEATURE: vmbackup: support client-side TLS configuration for creating and deleting snapshots via -snapshot.tls* cmd-line flags. See this feature request. Thanks to @khushijain21 for the pull request.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: all VictoriaMetrics components: consistently return 200 http status code from /-/reload endpoint. Previously single-node VictoriaMetrics was returning 204 http status code. See this feature request.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
BUGFIX: vmui: fix the graph dragging for Firefox and Safari. See this issue.
BUGFIX: vmui: fix handling invalid timezone. See this issue.
BUGFIX: vmui: fix the bug where the select does not open. See this issue.
BUGFIX: vmui: clear entered text in select after selecting a value. See this issue.
BUGFIX: vmui: improve the operation of the context for autocomplete. See this, this and this issues.
BUGFIX: dashboards: update Storage full ETA panels for Single-node and Cluster dashboards to prevent them from showing negative or blank results caused by increase of deduplicated samples. Deduplicated samples were part of the expression to provide a better estimate for disk usage, but due to sporadic nature of deduplication in VictoriaMetrics it rather produced skewed results. See this pull request.

How to run VictoriaMetrics

Unpack the victoria-metrics-*.tar.gz archive and read these docs.

vmutils-*.tag.gz archive contains the following tools:

vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:

The corresponding docker images are available here.
Cluster version is available here.