VictoriaMetrics/operator v0.75.0

on GitHub

Update note 1: vmagent, vmalert, vmalertmanager, vmsingle: the config-reloader sidecar now gets an extra --target-dir= argument per watched directory, which changes the pod spec. All VMAlertmanager pods, and every VMAgent, VMAlert or VMSingle pod whose reloader watches a directory (spec.configMaps, relabeling or stream aggregation configs, rule ConfigMaps), will be rolled out once during this upgrade.

Update note 2: vmcluster: a serviceSpec with useAsDefault: true that makes the default Service of vmstorage non-headless, that is a spec.type other than ClusterIP, or an explicit spec.clusterIP/spec.clusterIPs other than None, is now rejected at spec.vmstorage.serviceSpec. Reconciliation of such a VMCluster fails until the serviceSpec keeps the Service headless, or until useAsDefault is removed, which turns it into an additional Service. See #2491.

Update note 3: vmcluster, vmalertmanager: spec.vmselect.serviceSpec of VMCluster and spec.serviceSpec of VMAlertmanager with useAsDefault: true and an explicit spec.type, but no spec.clusterIP, now produce a Service with a virtual IP instead of a headless one. That Service is recreated during the upgrade and gets a new clusterIP. For a VMCluster with spec.requestsLoadBalancer.enabled, the recreated Service is the vmauth proxy of vmselect, while its internal Service stays headless. Add spec.clusterIP: None, or remove spec.type, to keep the previous behaviour. See #2487.

Update note 4: vmrule, vmuser, vmalertmanagerconfig, vmanomalyconfig, vmservicescrape, vmpodscrape, vmnodescrape, vmprobe, vmstaticscrape, vmscrapeconfig: status.updateStatus of these config-only objects no longer reports whether the applications selecting them accepted their content. It is now operational unless the operator cannot parse the object's own spec, in which case it is set to failed. The ignored value is no longer set. The state reported by each application is available at status.conditions.

• Dependency: vmoperator: Updated default versions for VM apps to v1.152.0.

• Dependency: vmoperator: Updated default versions for VT apps to v0.11.1.

• Dependency: vmoperator: Updated default version for VMAnomaly to v1.30.7.

• Dependency: vmoperator: Updated default version for Alertmanager to v0.34.1.

• FEATURE: vmanomaly: support bounded spec.reader.workers, query-level business policies, and settings.native_threads_per_worker introduced in vmanomaly v1.30.2. See #2496.

• FEATURE: vmanomaly: add typed spec.writer.batchMaxSeries, spec.writer.batchMaxBytes, and spec.writer.metricPrefixCacheMaxEntries fields for the bounded writer controls introduced in vmanomaly v1.30.3. See #2542.

• FEATURE: vmoperator: add k8s_version label with the Kubernetes server version to the vm_app_version metric. See #2530.

• FEATURE: vmoperator: add operator_controller_errors_total{controller,namespace,name,reason} metric, which counts reconcile errors of every controller by reason: get_object, parse_object, cancel_context, conflict, declined and other. It deprecates operator_controller_object_get_errors_total, operator_controller_object_parsing_errors_total, operator_controller_reconcile_conflict_errors_total and operator_controller_reconcile_errors_total, which are still exported, but will be removed in a future release. See #2536.

• FEATURE: vtagent: add VTAgent CRD for vtagent. It can accept OTLP trace spans over HTTP and gRPC and replicates them to one or more VictoriaTraces instances. The new VTAgent CRD must be installed in order to use it. See #2510.

• FEATURE: vtagent, vtsingle, vtcluster: add grpcSpec field (spec.insert.grpcSpec for VTCluster) to accept OTLP trace spans over gRPC in addition to HTTP. Its listenPort is exposed at the component Service and can be secured with the optional tlsConfig. See #2510.

• FEATURE: vlagent, vlsingle, vlcluster: add minVersion and cipherSuites fields to tlsConfig of syslog TCP listeners. See #2510.

• FEATURE: vlsingle, vtsingle: add removePvcAfterDelete field, which makes Kubernetes garbage-collect the PersistentVolumeClaim together with the deleted VLSingle or VTSingle object. It is disabled by default, so the PersistentVolumeClaim with the stored data is now preserved after the object deletion. Previously, it was always removed together with the object. See #2545.

• FEATURE: vmcluster, vmalertmanager: allow a serviceSpec with useAsDefault: true to replace the headless default Service of vmselect and VMAlertmanager with a regular one by defining spec.type. Previously, spec.type: ClusterIP still inherited clusterIP: None from the default Service, so the only way to get a virtual IP was to pin spec.clusterIP by hand. See #2487.

• BUGFIX: vmoperator: set status.updateStatus for config-only objects - VMRule, VMUser, VMAlertmanagerConfig, VMAnomalyConfig, VMServiceScrape, VMPodScrape, VMNodeScrape, VMProbe, VMStaticScrape and VMScrapeConfig - even when no application selects them. Previously, such objects were left without any status, which failed health checks of GitOps tools such as ArgoCD. The status is now operational, or failed if the operator cannot parse the object's spec. See #2649.

• BUGFIX: vmagent, vmanomaly: set the default value 0 to spec.shardCount in the CRD schema. Previously, the field was absent unless sharding was configured, so every read of the object's /scale subresource failed with the spec replicas field ".spec.shardCount" does not exist, which broke spec.vpa, spec.hpa and any external autoscaler pointed at the object. The VMAgent and VMAnomaly CRDs must be updated to get this fix. See #2473.

• BUGFIX: vmoperator: set default values for each possible level label of operator_log_messages_total metric. See #2477.

• BUGFIX: vmoperator: add the default preStop sleep hook only on Kubernetes >= 1.30. Previously it was added on 1.29 as well, where the PodLifecycleSleepAction feature gate is alpha and off by default, the affected components could fail to reconcile. This bug was introduced in v0.71.0. See #2522.

• BUGFIX: vmagent: remove the generated HorizontalPodAutoscaler object once it no longer applies, both when spec.hpa is unset and when spec.daemonSetMode is enabled with spec.hpa still set. Previously, the object was named after its scale target, which is the VMAgent object itself, while the cleanup looked for the vmagent-<name> name it never had. The object is now named vmagent-<name>, consistently with the other objects generated by the operator, and the object with the old name is removed on the next reconcile. This bug was introduced in v0.71.0. See #2518.

• BUGFIX: vmagent, vmanomaly: name the generated VerticalPodAutoscaler object vmagent-<name> and vmanomaly-<name> instead of the VMAgent and VMAnomaly object name, consistently with the other objects generated by the operator. The object with the old name is removed on the next reconcile. See #2518.

• BUGFIX: vmauth, vmuser: identify VMUser objects with spec.jwt as <namespace>/jwt:<name>. It allows to configure multiple VMUser objects with identical spec.jwt sections at the same namespace. Previously, such objects shared the same id as soon as the operator created their vmuser-<name> secrets, so only one of them was kept at the generated vmauth config and the rest were dropped with a duplicate id error at status.currentSyncError. Objects with spec.disableSecretCreation were not affected. See #2532.

• BUGFIX: vmagent, vmsingle: remove the service discovery Role and RoleBinding once the object stops using the operator managed ServiceAccount, and remove them from the namespaces watched by the operator once the object is deleted. Previously, setting spec.serviceAccountName left the Role and RoleBinding at the object's namespace behind, and with WATCH_NAMESPACE set the objects created at the other watched namespaces were never removed. This bug was introduced in v0.73.0, when the operator started creating these objects at the object's namespace in cluster-wide mode and at every watched namespace in namespaced mode. See #2655.

• BUGFIX: vmsingle: fix VMSingle pods crash-looping when spec.relabelConfig or spec.inlineRelabelConfig is set. This bug was introduced in v0.69.0. See #2552.

• BUGFIX: vmagent, vmsingle, vmalert: config-reloader now triggers an application reload after its initial directory sync, and retries a failed reload request with backoff instead of dropping it. Previously, the application container could start and read a directory before the reloader filled it, with no reload to follow. This bug was introduced in v0.72.0. See #2527.

• BUGFIX: vmoperator: generate the operator's ClusterRole from the +kubebuilder:rbac markers in the code, instead of maintaining config/rbac/role.yaml by hand. Previously, the markers were silently ignored by controller-gen and the hand-written ClusterRole was missing the patch verb for pods, so the operator failed to trigger the rules reload on VMAlert pods after a VMRule change, logging only a 403 Forbidden error while the updated rules were applied with a delay. The regenerated ClusterRole also drops permissions the operator does not need, such as escalate and bind on ClusterRoles. The operator's ClusterRole must be updated to get this fix. See #2543.

• BUGFIX: vmcluster: reject a serviceSpec with useAsDefault: true that makes the default Service of vmstorage non-headless, that is a spec.type other than ClusterIP, or an explicit spec.clusterIP/spec.clusterIPs other than None. Previously, such a serviceSpec was applied as is, and vminsert and vmselect silently lost the per-pod DNS records of vmstorage that only a headless Service provides. See #2487.

• BUGFIX: vmauth: accept spec.unauthorizedUserAccessSpec with only access_log set, without url_map, url_prefix or targetRefs. Previously such a spec was rejected both by the admission webhook and by the reconcile with at least one of url_map, url_prefix or targetRefs must be defined, so there was no way to log the requests that vmauth answers with 401. It requires vmauth v1.147.0 or newer. See #2551.

• BUGFIX: vmoperator: expose the reason why a PersistentVolumeClaim resize was declined under the resource's status.reason, e.g. when it is rejected by a non-expandable StorageClass, and still apply the pending metadata changes. Previously, the reason was only logged and the reconcile could be reported as fully successful. See #2512.

• BUGFIX: vmanomaly: fix propagation of spec.extraEnvsFrom to anomaly pods, previously it was omitted. See #2567.

• BUGFIX: vmalertmanager: fix propagation of spec.extraEnvsFrom to vmalertmanager pod, previously it was omitted. See #2582.

• BUGFIX: vmauth: fix unmarshalling of spec.httpRoute.extraRules with multiple rules. Previously, a rule could inherit fields from the previous one, e.g. a rule without filters silently got filters from the rule above it. This bug was introduced in v0.66.0, when spec.httpRoute was added. See #2605.

• BUGFIX: vmoperator: remove the invalid items key under dnsConfig in the VMAgent, VMAlertmanager, VMAnomaly, VMCluster, VLAgent, VLCluster, VTAgent and VTCluster CRDs. Previously, while Kubernetes itself ignored this key, it could fail validation in strict JSON Schema consumers such as KubeVela. See #2618.

• BUGFIX: vmoperator: apply storage.volumeClaimTemplate.spec.volumeAttributesClassName changes to already created PersistentVolumeClaims. Previously, the field was silently ignored after the claim was created. See #2598.

• BUGFIX: VMRule CRD: drop the leftover default value "" from the record and alert fields. This default caused Kubernetes to add an empty alert: "" or record: "" to every rule, so GitOps tools such as ArgoCD always detected a diff and kept VMRule objects OutOfSync. This default value was introduced in v0.73.0, which required every rule to have both record and alert fields set. v0.73.1 stopped identifying rules this way, since a group may contain multiple rules with the same name. The VMRule CRD must be updated to get this fix. See #2622.

• BUGFIX: vmagent, vmsingle, vmalert: fix config-reloader sidecar crashing with --target-dir count must match watched directory count error. Previously, the bug could be triggered by VMAlert with spec.configMaps defined and at least one VMRule selected, or by VMAgent and VMSingle whose scrape config was large enough while also using spec.configMaps, relabeling configs or stream aggregation configs. This bug was introduced in v0.72.0. See #2583.