v0.8.0 lets you set a ServiceAccount and custom labels and annotations on Valkey pods, and keeps one long-lived operator connection per node. It also fixes clusters on IPv6-only pods, cluster names too long for their StatefulSets, and the exporter's COMMANDLOG metrics. Thank you to everyone who contributed to this release!
Upgrade notes
Apply both CRDs before upgrading the operator
The operator copies the new podMetadata and serviceAccountName fields from each ValkeyCluster to its ValkeyNodes. With the v0.7.1 ValkeyNode CRD, the API server prunes both fields from the ValkeyNodes. The cluster still reports Ready and no condition or event says anything is wrong, but the labels and annotations never reach the pods, and the pods keep the namespace's default ServiceAccount.
Apply both v0.8.0 CRDs first:
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.8.0/config/crd/bases/valkey.io_valkeyclusters.yaml
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.8.0/config/crd/bases/valkey.io_valkeynodes.yamlHelm does not upgrade CRDs shipped in a chart's crds/ directory, so this step applies to Helm installs too. --force-conflicts is needed there, because Helm owns the CRD fields it installed and a plain server-side apply is rejected.
The operator needs get on ServiceAccounts
When spec.serviceAccountName is set, the operator checks that the ServiceAccount exists (#415). The ClusterRole gains a serviceaccounts rule with the get verb. The kustomize manifests and the v0.8.0 valkey-operator Helm chart include it. If you maintain your own RBAC, add the rule before upgrading. Without it, a cluster that sets serviceAccountName reports ConfigurationWarning with reason ServiceAccountLookupFailed.
Upgrading does not roll pods
The pod template only changes for a cluster that sets one of the new fields, so upgrading leaves existing pods running. The new _exporter ACL from #481 is loaded into each running pod without a restart, and each cluster emits an InternalSecretsUpdated event when the new ACL is applied.
A cluster whose name is now too long cannot be updated
v0.8.0 limits a ValkeyCluster's name to what its StatefulSets can carry (#496). With workloadType: StatefulSet and single-digit shard and node indexes, that is 41 characters, and each extra digit in the highest index takes one character away. With workloadType: Deployment the limit is 56 characters. The workload type docs give the full rule.
The API server applies the rule to existing clusters too. It rejects every update to a cluster created before v0.8.0 whose name breaks the rule, including the operator's status writes, until the cluster is scaled down to a size the rule accepts. Deleting it still works. If the name is too long even with single-digit indexes, delete the cluster and recreate it under a shorter name.
To list StatefulSet clusters over 41 characters:
kubectl get valkeycluster -A -o json | jq -r '.items[] | select((.spec.workloadType // "StatefulSet") == "StatefulSet" and (.metadata.name | length) > 41) | "\(.metadata.namespace)/\(.metadata.name)"'A held roll now says which shard it is waiting on
While the roll of a shard's primary is held, the Ready and Progressing conditions keep reason UpdatingNodes, and the message now names the shard, the node and the cause (#483, closes #267), for example:
Updating ValkeyNodes: the roll of shard 2 primary valkey-c-2-0 is waiting, the shard has no synced replica to fail over to
A healthy roll passes through this state too, while the replica it rolled first does its initial sync. See status conditions for an alert on a roll that runs too long.
Features
ServiceAccount for Valkey pods (#415, closes #414)
spec.serviceAccountName sets the ServiceAccount each ValkeyNode pod runs as. The ServiceAccount must already exist in the cluster's namespace: the operator does not create it or its RBAC. If it is missing, pod admission rejects the pods, and the cluster reports ConfigurationWarning with reason ServiceAccountNotFound. Changing the field rolls every pod. See Service account.
Custom pod labels and annotations (#523, closes #522)
spec.podMetadata.labels and spec.podMetadata.annotations are added to every Valkey pod, for example to select the pods from a PodMonitor, a NetworkPolicy or a Service. They stay out of the workload selector, so you can change them at any time. A change rolls every pod. Keys under valkey.io/ and the app.kubernetes.io/ name, instance, component, part-of and managed-by labels are rejected, and keys and values are checked with the rules Kubernetes applies to pod metadata. See Pod metadata.
One standing connection per node (#469, closes #266)
The operator used to dial, authenticate and close a new Valkey client for every check: on kind, about 18 connections per node per minute. The cluster reconcile, the node reconcile and the role poller now share one long-lived _operator client per pod. A client is replaced when the operator password, the TLS Secret, the server name or the client certificate mode changes, and closed after 2 minutes unused.
Fixes
Clusters form on IPv6-only pods (#541, closes #540)
The operator built node addresses as ip:port, so an IPv6 pod IP failed to dial with too many colons in address. On IPv6-only pods no shard formed, and every cluster stayed at ClusterFormed=False with reason MissingShards. IPv6 addresses are now dialled as [ip]:6379.
A cluster name too long for its StatefulSets is rejected (#496, closes #449)
Kubernetes stamps each StatefulSet pod with a controller-revision-hash label made from the StatefulSet name, and label values are limited to 63 characters. A cluster with a long name was accepted, but its StatefulSets never got their pods. The name limit is now checked at admission, including on a scale-out that adds a digit to an index. See the upgrade note above for existing clusters.
Exporter COMMANDLOG metrics (#481, closes #480)
The _exporter user was denied COMMAND INFO and COMMANDLOG LEN, so every scrape logged a NOPERM on the server and the redis_commandlog_* metrics were never exported. The user now has +command|info and +commandlog|len.
A superseded pod is probed at its current IP (#538, closes #461)
Before replacing a superseded pod that is not Ready, the operator checks with INFO that it is not loading a dataset. That check dialled the IP stored in the ValkeyNode status, which is not refreshed while the replacement is deferred. If the pod had moved to another IP, every check failed and the pod was never replaced. The check now dials the pod's current IP.
Known issues
Without persistence, a crashed primary can restart empty (#533)
If a primary's valkey-server is killed and restarts within cluster-node-timeout, it rejoins as the primary with the same node ID and no data, and its replicas resync from it, so the shard loses all its keys. The fix is in review in #537.
What's Changed
- (feat) allow custom serviceAccountName on valkey cluster pods by @sandeepkunusoth in #415
- fixed lint issues with go 1.26 by @sandeepkunusoth in #479
- fix: let the exporter user probe and read COMMANDLOG by @dimitris-christou-skillonnet in #481
- chore(deps): bump docker/build-push-action from 7.3.0 to 7.4.0 by @dependabot[bot] in #498
- chore(deps): bump docker/setup-buildx-action from 4.3.0 to 4.4.1 by @dependabot[bot] in #497
- fix: say in the status when a primary's roll is waiting for a synced replica by @melancholictheory in #483
- refactor(valkey): pass a dial func to GetClusterState by @jdheyburn in #455
- feat: add ClientProvider and dial the cluster scrape through it by @jdheyburn in #456
- feat: dial ValkeyNode clients through ClientProvider.ForNode by @jdheyburn in #457
- chore: update kubebuilder scaffold v4.15.0 -> v4.16.0 by @bjosv in #447
- feat: add a pool of long-lived Valkey clients by @jdheyburn in #470
- ci: require the current year in new files' copyright header by @jdheyburn in #505
- feat: take Valkey clients from the pool by @jdheyburn in #469
- fix: bracket ipv6 pod addresses so clusters form on ipv6-only pods by @fadymak in #541
- test(e2e): add recovery tests for replica scale-down by @bjosv in #453
- fix: reject a ValkeyCluster name its StatefulSets cannot carry by @melancholictheory in #496
- feat: add spec.podMetadata for custom pod labels and annotations by @spron-in in #523
- test: cover ValkeyClusterSpec persistence CEL rules in envtest by @highpon in #475
- fix: probe a superseded pod at its current IP before replacing it by @melancholictheory in #538
- fix: ignore the deprecated serviceAccount field when comparing pod templates by @jdheyburn in #546
New Contributors
- @dimitris-christou-skillonnet made their first contribution in #481
- @fadymak made their first contribution in #541
- @spron-in made their first contribution in #523
- @highpon made their first contribution in #475
Full Changelog: v0.7.1...v0.8.0