github valkey-io/valkey-operator v0.9.0
on GitHub

one hour ago

v0.9.0 moves ACL users to spec.acl.users, turns the Valkey default user off unless a cluster declares it, and adds replicasPerShard, a separate client CA for mTLS and a Service per node. It also keeps a crashed primary's data when the shard has a replica, and keeps reconciling when a user's password Secret is missing. Thank you to everyone who contributed to this release!

Upgrade notes

Apply both CRDs before upgrading the operator

The operator records the ACL revision each Valkey node has loaded in the ValkeyNode status. With the v0.8.0 ValkeyNode CRD, the API server prunes that field, and the operator runs ACL LOAD on every Valkey node about every 30 seconds. The old CRDs also drop the new fields: networking.perNodeServices never reaches the ValkeyNodes, and a patch that replaces replicas with replicasPerShard leaves the cluster with 0 replicas.

Apply both v0.9.0 CRDs first: 

kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.9.0/config/crd/bases/valkey.io_valkeyclusters.yaml
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/valkey-io/valkey-operator/v0.9.0/config/crd/bases/valkey.io_valkeynodes.yaml

Helm does not upgrade CRDs shipped in a chart's crds/ directory, so this step applies to Helm installs too. --force-conflicts is needed there, because Helm owns the CRD fields it installed and a plain server-side apply is rejected.

If the CRDs are applied after the operator, restart the operator Deployment afterwards. An operator that was running when the CRDs changed does not create per-node Services until it restarts.

The operator requires two image flags

The operator no longer has default images built in (#551). It takes them from --default-valkey-image and --default-exporter-image, and exits at startup if either is missing: 

--default-valkey-image must not be empty

The kustomize manifests and the v0.9.0 valkey-operator Helm chart pass both flags, set to the images v0.8.0 had built in. If you maintain your own operator Deployment, add them before changing the image. To keep the previous defaults: 

--default-valkey-image=valkey/valkey:9.0.0
--default-exporter-image=oliver006/redis_exporter:v1.88.0

Changing a flag later rolls every cluster that does not set spec.image or spec.exporter.image. See Default images.

Upgrading rolls every cluster once

The pod template changes, so the operator replaces every pod, one node at a time, replicas before primaries, with a failover before each primary. The changes:

  • The server command runs through a wrapper that handles a crashed primary (#537).
  • The startup probe allows 80 failures, up from 20, on pods that use that wrapper (#537).
  • A server-init init container and an init-conf volume are added (#527).
  • allowPrivilegeEscalation: false is set on the server and exporter containers (#548).
  • initialDelaySeconds is removed from the liveness and readiness probes (#534).

A custom image that needs privilege escalation must now set allowPrivilegeEscalation: true on its container in spec.containers. server-init is a reserved container name: the API server rejects a containers entry that uses it.

The ClusterRole and the minimum Kubernetes version are unchanged.

The default user is turned off

Before v0.9.0, a cluster that declared no default user kept Valkey's open one (on nopass +@all). The operator now writes user default off ~* &* +@all on every such cluster (#620). After the upgrade:

  • A client that connects without credentials gets NOAUTH. That includes valkey-cli run inside a Valkey pod.
  • Connections that were logged in as default are closed with CLIENT KILL USER default (#627). That includes clients that have connected but not yet authenticated, which reconnect.
  • Each ValkeyNode gains a DefaultUserSessionsOpen condition once default has been seen enabled on it (#633).

To keep an open default, declare it on each cluster before upgrading. On v0.8.0 that is spec.users: 

spec:
  users:
    - name: default
      nopass: true
      permissions: "~* &* +@all"

See The default user, including the two Valkey bugs that affect a default declared with fewer permissions.

spec.users and spec.replicas are deprecated

Both fields still work and are removed in v1beta1. Their replacements are spec.acl.users (#558) and spec.replicasPerShard (#590). A cluster that sets spec.users, or a non-zero spec.replicas, reports a ConfigurationWarning condition with reason DeprecatedField. The API server rejects a spec that sets a deprecated field together with its replacement.

Migrate after the upgrade, once the new CRDs are applied and the old operator pod is gone. A v0.8.0 operator reads 0 replicas from a cluster that only sets replicasPerShard, and deletes its replicas.

Users move like this: 

# Before
spec:
  users:
    - name: alice
      passwordSecret:
        name: my-users-secret
        keys: [alicepw]
      permissions: "+@read ~app:*"

# After
spec:
  acl:
    users:
      - name: alice
        password:
          secret:
            name: my-users-secret
            keys: [alicepw]
        permissions: "+@read ~app:*"

enabled, nopass and resetpass become state and password.type. The generated ACL is the same, so moving users changes nothing on the running nodes. See Migrating from spec.users and Migrating from spec.replicas. The second gives a patch that swaps the fields in one update without creating or deleting a pod.

The operator connects as _operator only

The operator used to fall back to the default user when a node rejected the _operator password. That fallback is gone (#542). A node that rejects the password reports ACLApplied=False with reason AuthenticationFailed. Its cluster reports Degraded with reason OperatorAuthenticationFailed and makes no topology changes until every node accepts the password again. See ACLApplied for the recovery steps.

Validation changes

  • A ValkeyCluster name that contains a dot is rejected on create (#631, closes #629). Such a cluster was accepted before and then stuck at Failed/ServiceError, because its headless Service cannot carry the name.
  • Validation errors now name the field to fix and use the type Forbidden (#619, closes #618). The messages, and what is accepted, are unchanged.

Features

ACL users under spec.acl.users (#558, closes #486)

spec.acl.users replaces spec.users. A user's state is state: Enabled or Disabled, and its password is password.type: Secret, Any or Disabled, in place of the enabled, nopass and resetpass booleans. A password Secret is named under password.secret, and still defaults to <cluster>-users with the username as the key. See Users.

The default user is off unless declared (#620, closes #593; #627, closes #624; #633)

A cluster that does not declare default no longer accepts unauthenticated clients. Under mTLS with certificateUser set, a client whose certificate names no ACL user is logged in as default, so it now gets NOAUTH too. When an ACL change turns default off on a running node, the operator closes the connections logged in as it. Each node's DefaultUserSessionsOpen condition is True while default is on, and stays True after it is turned off until those connections are closed. See DefaultUserSessionsOpen.

spec.replicasPerShard (#590, closes #559)

replicasPerShard is the new name for replicas: the number of replicas in each shard, in addition to its primary. It defaults to 0. See Shards and replicas.

A separate CA for client certificates (#508, #565, #573, #594)

networking.tls.clientAuth.ca lists Secrets and ConfigMaps whose roots are trusted for client certificates, in addition to the server Secret's ca.crt. Each source holds PEM certificates or a SPIFFE trust bundle, so clients can present SPIRE X.509-SVIDs while the server certificate comes from another CA. The operator merges the roots into a Secret named valkey-<cluster>-tls-trust, and nodes reload it without a restart when a source changes. Adding or removing clientAuth.ca rolls the nodes once.

The same bundle is used on the cluster bus, which has no authentication beyond TLS. Limit port 16379 to the cluster's own pods with a NetworkPolicy. See Trusting a separate client CA.

A ClusterIP Service per node (#490, part of #367)

networking.perNodeServices: {} creates one ClusterIP Service per Valkey node, named valkey-<cluster>-<shard>-<node> and selecting that node's pod on port 6379. It gives a client that dials one node directly a stable address. CLUSTER SLOTS is unchanged, so a client that follows redirects still uses the announced pod address. See Per-node Services.

Configurable default images (#551, closes #417)

The images used when a cluster sets no spec.image or spec.exporter.image now come from the operator flags --default-valkey-image and --default-exporter-image, so they can point at a private registry or a newer Valkey without editing each cluster.

Fixes

A crashed primary no longer wipes its shard (#537, closes #533)

Without persistence, a primary whose valkey-server was killed could restart inside cluster-node-timeout with its node ID and no data. Its replicas then resynced from it and the shard lost every key. The restarted server now waits for a reachable replica to take over, up to six cluster-node-timeout intervals, then clears its data and joins as a replica of the new primary. With no reachable replica it keeps its node ID and local files as before. See Failure handling.

A missing password Secret no longer blocks the cluster (#502, closes #499)

If a user's password Secret was missing or lacked a listed key, the operator stopped reconciling the whole cluster, including scaling and rolls. A cluster that already has an ACL file now keeps it, reports Degraded with reason UsersACLUnresolved naming the user and the Secret, and reconciles everything else. A new cluster still waits for every user's Secret before it starts.

Lowering spec.shards during a scale-out keeps slot-owning primaries (#493, closes #491)

Lowering spec.shards while a scale-out was still moving slots could delete a primary that owned slots, losing its keys. The operator now drains such a shard first, and emits a ScaleInBlocked warning event instead of deleting a primary that owns slots.

A shard left empty by a reverted scale-in gets its slots back (#550, closes #445)

A scale-in reverted right after its drain left a primary with no slots and its replica still attached. The cluster then stayed at Ready=False with reason UpdatingNodes, with no event. The operator now rebalances slots onto that shard, and the Ready message names the shard being filled.

Every orphaned shard is promoted (#477, closes #462)

When several primaries were lost together and the cluster had no failover quorum, the operator promoted a replica in the first shard only, and the others stayed without a primary. It now takes over one lost primary per reconcile until quorum is back. With quorum it leaves recovery to Valkey's own election, except for a replica that has cluster-replica-no-failover set.

A failing ACL or config apply no longer rewrites the node status on every reconcile (#474, closes #460)

A network error in a live ACL or config apply put the connection's local port in the condition message. The port differs on every attempt, so the operator rewrote the ValkeyNode status on each reconcile, about 67 writes per second on one node in the report. Messages that differ only in that address no longer cause a write.

Stale status writes no longer erase ValkeyNode conditions (#636, closes #635)

A ValkeyNode status write built on a cached read could replace the whole condition list, removing a condition written just before or bringing back one just removed. Status writes now carry the resource version and are retried on a conflict.

readyShards stays current during a stuck roll (#431)

While a node roll was waiting, status.readyShards and status.shards kept their previous values. They are now refreshed from the same cluster scrape on that path.

An admission policy that defaults allowPrivilegeEscalation no longer causes a roll loop (#548)

On a cluster whose admission policy sets allowPrivilegeEscalation: false, the admitted pod template never matched the one the operator wanted. The operator kept updating the template and failing over primaries with no change to the ValkeyCluster. The operator now sets the same default itself.

Probes start as soon as the startup probe passes (#534, closes #302)

The liveness and readiness probes waited another 5 seconds after the startup probe had succeeded. That delay is removed.

Known issues

A primary crash during a scale-in can lose acked writes (#638)

While a scale-in drains a shard, the operator deletes that shard's replica as soon as its primary stops answering. The restarted primary finishes the drain from what it has on disk, which without AOF is at most its last RDB snapshot. The cluster then reports Ready with the writes missing. With persistence and appendonly: "yes" in config the primary replays its AOF and no acked write was lost in testing. A fix is planned for v0.10.0.

A scale-in can delete an excess primary whose pod is being rescheduled (#597)

If the primary pod of a shard being removed is recreated during the drain and is slow to get an IP, for example because it stays Pending, the operator deletes its ValkeyNode, and its PVC under reclaimPolicy: Delete. The slots it had not drained stay unassigned and the cluster stays in Reconciling. Avoid lowering spec.shards during node maintenance. A fix is planned for v0.10.0.

Lowering replicas can delete a primary without a failover (#459)

Lowering replicasPerShard removes the nodes with the highest index in each shard. If one of them is the shard's primary, it is deleted without the operator failing it over first. The terminating primary normally hands its slots to a replica on SIGTERM. If that hand-off does not finish inside the termination grace period, the shard has no primary until a replica wins an election. Before lowering the count, check with kubectl get valkeynode that no node about to be removed is a primary.

Recreating the system passwords Secret locks the operator out (#506)

If the internal-<cluster>-system-passwords Secret is deleted, the operator generates new passwords that the running nodes do not accept. Since #542 the cluster reports Degraded with reason OperatorAuthenticationFailed instead of appearing healthy, and stays there until the previous passwords are written back to the Secret. The steps are under ACLApplied. Do not delete that Secret on a running cluster.

What's Changed

New Contributors

Full Changelog: v0.8.0...v0.9.0

Check out latest releases or
releases around valkey-io/valkey-operator v0.9.0

Don't miss a new valkey-operator release

NewReleases is sending notifications on new releases.

Get notifications