v0.7.0 adds mTLS certificate-based ACL authentication, hostname-based cluster discovery, version-gated config and a configurable cluster-node-timeout, along with fixes to failure detection and roll recovery. Thank you to everyone who contributed to this release!
Upgrade notes
Kubernetes 1.32 or newer is required
The CRDs now use the CEL format library to validate serverName. A 1.31 API server rejects both CRDs, so the minimum supported version moves from 1.31 to 1.32 (#466, closes #465).
Apply the CRDs before upgrading the operator
The v0.7.0 operator writes new fields (clusterDomain, tls.serverName, tls.clientAuth) to each ValkeyNode. Against the v0.6.0 CRDs those fields are pruned, the operator sees a diff on every reconcile, and the cluster stays in Reconciling/UpdatingNodes without finishing the upgrade roll.
Apply the v0.7.0 CRDs first:
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.7.0/config/crd/bases/valkey.io_valkeyclusters.yaml
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.7.0/config/crd/bases/valkey.io_valkeynodes.yamlHelm does not upgrade CRDs shipped in a chart's crds/ directory, so this step applies to Helm installs too. --force-conflicts is needed there, because Helm owns the CRD fields it installed and a plain server-side apply is rejected.
The operator needs delete on pods
The node controller now deletes a pod that its StatefulSet can no longer replace (#410). The pods rule in the ClusterRole gains delete. The kustomize manifests and the v0.7.0 valkey-operator Helm chart include it. If you maintain your own RBAC, add the verb before upgrading.
Upgrading triggers a rolling restart
The pod template changes for every ValkeyCluster, so all pods are rolled one node at a time, replicas first, with a proactive failover before each primary. What changes the template:
cluster-node-timeout 2000is no longer part of the operator's base config (#450), which changes the config hash annotation.- Cluster-owned StatefulSets move to the cluster's headless Service as
spec.serviceName(#378).serviceNameis immutable, so the operator orphan-deletes each StatefulSet and recreates it during that node's roll. The pod is adopted and then replaced by the roll. - On TLS clusters, the health probes no longer present the server certificate as a client certificate unless
clientAuth.modeisRequired(#242), and the metrics exporter getsREDIS_EXPORTER_TLS_SERVER_NAME(#409).
cluster-node-timeout now defaults to 15 seconds
The operator used to hardcode cluster-node-timeout 2000. It now leaves the directive unset unless you set it in spec.config, so clusters move to the Valkey default of 15000 ms after the upgrade roll (#450, closes #434).
Failure detection and automatic failover take longer as a result. A primary that crashes and restarts within 15 seconds rejoins as the primary rather than being failed over. Without persistence it reloads its last RDB snapshot, and its replicas resync from it, so writes since that snapshot are lost. To keep the previous behaviour, set:
spec:
config:
cluster-node-timeout: "2000"Changing spec.config.cluster-node-timeout rolls the cluster.
New TLSEndpointWarning condition on TLS clusters
A TLS cluster that announces pod IPs (the default) now reports TLSEndpointWarning=True with reason TLSWithIPAnnounce. Clients that follow CLUSTER SLOTS to a pod IP usually fail certificate name checks. The condition does not affect Ready. Set networking.discovery.preferredEndpointType: Hostname to clear it (see below).
Features
mTLS certificate-based ACL authentication (#242)
spec.networking.tls.clientAuth controls client certificates:
mode:Optional(default, renderstls-auth-clients optionalas before),Required(tls-auth-clients yes) orDisabled(tls-auth-clients no).certificateUser:CNorURImaps the client certificate to the ACL user of that name (tls-auth-clients-user), so the client authenticates withoutAUTH.CNrequires Valkey 9.0 andURIrequires Valkey 9.1. The default,Disabled, leaves the directive out.
With mode: Required, the operator, probes, exporter and replication links present the node's server certificate. See docs/mtls.md.
Hostname-based discovery (#378, closes #365)
spec.networking.discovery.preferredEndpointType: Hostname makes each node announce <pod>.valkey-<name>.<namespace>.svc.<clusterDomain>, so TLS clients can verify the certificate after following a MOVED or CLUSTER SLOTS redirect. It requires workloadType: StatefulSet. The default stays IP. spec.networking.clusterDomain sets the cluster DNS domain (default cluster.local). The announced name has no trailing dot, so it is valid for SNI (#429, closes #425).
TLS server name for operator connections (#409, closes #405)
spec.networking.tls.serverName sets the name the operator verifies when it dials a node by pod IP. It defaults to valkey-<name>.<namespace>.svc.<clusterDomain>, which was previously hardcoded. ValkeyNodes created before this release fall back to the same default until the cluster updates them (#468, closes #464).
Configurable cluster-node-timeout (#450, closes #434)
cluster-node-timeout can now be set in spec.config. See the upgrade note above for the default change.
Version-gated config (#307, closes #287)
The operator reads the Valkey version from the spec.image tag and drops spec.config directives the image does not support, reporting ConfigurationWarning with reason UnsupportedConfigDirective. tls-auto-reload-interval requires Valkey 9.1.0 or newer. On images whose version cannot be read from the tag (latest, digest-only, custom tags) gated directives are dropped.
Faster ValkeyNode.status.role (#320, closes #261)
status.role is now updated within a few seconds of a failover or restart, instead of on the 60 second resync. A 5 second poller watches live topology and triggers a node reconcile when the role changes. A restarted replica no longer reports primary while it resyncs, and a pod that is not ready reports an empty role.
pprof endpoint (#435)
--pprof-bind-address serves Go's pprof endpoints for profiling the operator. It is off by default. The endpoint is unauthenticated and exposes heap contents, so bind it to localhost. See the developer guide.
Fixes
A pod stuck on a superseded revision is replaced (#410, closes #408)
Under OrderedReady, the StatefulSet controller does not replace a pod that never becomes Ready, so correcting a bad spec had no effect until someone deleted the pod. The node controller now deletes a pod that is not Ready and is on a revision the StatefulSet has superseded, and emits SupersededPodDeleted. It leaves the pod alone while it is loading a dataset or syncing from its primary.
Failure flags are read from the node's peers (#442, closes #441)
The ready-shard count and failover target selection checked the node's own CLUSTER NODES line for fail/pfail, which never carries them. They now ask the peers, so a node its peers report as fail or fail? no longer counts toward a ready shard and is not chosen as a failover target.
A shard with an unidentifiable primary no longer blocks the whole cluster (#452, closes #451)
When the operator could not identify one shard's primary, it stopped reconciling every shard, including the steps that would have repaired that shard. It now skips only that shard's roll and keeps Ready withheld until the primary is identified.
serviceName migration race (#432, closes #424)
A StatefulSet recreate that raced its own delete could leave the pod without a StatefulSet. The operator now re-reads uncached and completes the create.
Lower memory use (#436)
Operator connections to Valkey use smaller buffers and command rings. Before this change, each short-lived control connection allocated about 1.3 MiB, and with the default 128Mi memory limit the operator could be OOM-killed on clusters of around 20 shards or more.
What's Changed
- feat: add networking.tls.serverName for operator TLS verify by @spotu-dev in #409
- chore(deps): bump docker/metadata-action from 6.1.0 to 6.2.0 by @dependabot[bot] in #419
- chore(deps): bump docker/setup-buildx-action from 4.1.0 to 4.3.0 by @dependabot[bot] in #418
- feat: discovery preferredEndpointType and headless serviceName by @daanvinken in #378
- chore(deps): bump docker/login-action from 4.2.0 to 4.6.0 by @dependabot[bot] in #420
- (feat) version gating initial support by @sandeepkunusoth in #307
- fix: announced Hostname FQDN has no trailing dot by @daanvinken in #429
- fix: recover live-template STS Create after serviceName migrate race (#424) by @daanvinken in #432
- test(e2e): Hostname migrate keeps Ready pod UID by @daanvinken in #383
- feat: optional pprof endpoint for profiling the operator by @bjosv in #435
- perf: tune valkey-go connections for control-plane use by @bjosv in #436
- enhancement: reliable, event-driven ValkeyNode.Status.Role by @jdheyburn in #320
- (feat) support Mutual TLS (mTLS) certificate-based ACL authentication by @sandeepkunusoth in #242
- refactor: parse CLUSTER NODES into a struct instead of seven ad-hoc field walks by @bjosv in #433
- fix: replace a pod its StatefulSet can no longer replace by @melancholictheory in #410
- feat: make cluster-node-timeout user configurable by @deepakpunjabi in #450
- test(e2e): add AST-based scan to verify _operator ACL covers all operator commands by @tkarger in #392
- fix: ask the peers, not the node, whether a cluster member is failing by @melancholictheory in #442
- fix: skip only the shard whose primary is unidentifiable by @bjosv in #452
- docs: raise the minimum Kubernetes version to 1.32 by @jdheyburn in #466
- fix: fall back to the cluster server name for node TLS by @jdheyburn in #468
- fix: roll one node at a time when upgrading from v0.6.0 by @jdheyburn in #467
New Contributors
- @spotu-dev made their first contribution in #409
Full Changelog: v0.6.0...v0.7.0