Altinity/clickhouse-operator release-0.27.4

on GitHub

Security

• BACKWARD INCOMPATIBLE. Removed the k8s_secret_ and k8s_secret_env_ user-settings syntax. A CHI that still uses the syntax is now rejected with RemovedSecretRefSyntax. Migrate to valueFrom/secretKeyRef syntax unstead (see docs/security_hardening.md):

  users:
    user1/password:
      valueFrom:
        secretKeyRef:
          name: clickhouse-secret
          key: pwduser1

  by @sunsingerus

• Bumped the Go toolchain 1.26.5 → 1.26.8 to address stdlib CVEs in the operator and metrics-exporter images (GO-2026-6090, GO-2026-6089, GO-2026-6218, GO-2026-5972, GO-2026-5026). No API or behavior changes. The FIPS module (GOFIPS140=v1.0.0) is unchanged. by @sunsingerus

• Bumped the OpenTelemetry SDK v1.43.0 → v1.45.0 (otel, otel/metric, otel/sdk, otel/sdk/metric, otel/trace). Clears CVE-2026-41178, an uncapped baggage header parse in otel/baggage and otel/propagation, both of which the operator links. Also clears CVE-2026-81870; it is filed against otel/sdk, so module-keyed scanners flagged us, but it logs trace exporter endpoints and the operator links no trace SDK or trace exporter. by @sunsingerus

Behavior Changes

• A pod that will not terminate is force-deleted so StatefulSet Recreate finishes in one pass. A wedged shutdown previously aborted Recreate with the host left at Replicas=0. The operator now deletes that pod after a full wait timeout, emits HostPodForceDeleted, and completes Delete+Create in the same pass. A terminating pod is no longer treated as a healthy shard peer. by @alex-zaitsev in #2080. Fixes #2078

• ZooKeeper endpoint changes no longer restart ClickHouse. configurationRestartPolicy has advertised zookeeper/*: "no" since 0.25.6; the policy is now honored, so endpoint edits reload via conf.d with no restarts. This also removes the one-time CHI→CHK restart noted in 0.27.2. by @identw in #2074

• Keeper rolling updates no longer take an ensemble below Raft quorum. A Ready Keeper replica is deferred ([RaftQuorumUnsafe]) when disrupting it would lose majority; not-Ready replicas are recovered first. Affects ClickHouseKeeperInstallation rolls. by @alex-zaitsev in #2070. Fixes #2069

• Helm CRD-install hook uses registry.k8s.io/kubectl:v1.36.3 instead of bitnami/kubectl:latest. Air-gapped and mirrored installs must add that image before upgrade, or set crdHook.image / crdHook.enabled: false. Helm installs only. by @fhoekstra in #2065

• A failed schema migration no longer reports the CHI as Completed. HostCreateTables errors now abort the pass instead of being recorded as success. A CHI that already had an un-creatable object (for example a Dictionary with an unreachable source) will turn Aborted on first reconcile after upgrade. by @Tyagiquamar in #2077. Fixes #2021

• Operator informers and caches no longer hold every object in the Kubernetes cluster. They are narrowed at the API server to clickhouse.altinity.com/app=chop, and clickhouse-keeper.altinity.com/app=chop respectively, so memory scales with the number of managed installations rather than cluster size. by @sunsingerus. Approach proposed by @gregakinman in #2058

• ActionPlan is rendered once on status update. A status update invoked String() many times per retry, and each call reflection-dumped every diffed object, so on a large CHI that dominated operator CPU. by @erikasb in #2067

• A failed Keeper start no longer disables Keeper reconciliation until operator restart. Start-up now contacts the API server (required by the narrowed cache), so a transient error can fail the first attempt. The operator retries indefinitely without exiting; ClickHouse reconciliation is not blocked. by @sunsingerus

• OTEL_GO_X_CARDINALITY_LIMIT no longer caps operator metrics. The SDK bump brings a default of 2000 data points per instrument per scrape, which silently folds everything past it into one otel.metric.overflow series; the operator now pins the limit to unlimited, so series keep scaling with the installations watched. Anyone who set the variable to protect Prometheus must limit at the scrape instead. by @sunsingerus

Fixed

• Host deletion is no longer reported as completed when the operator cannot tell whether the host still exists. Only NotFound means the host is gone; other API errors now raise DeleteFailed. Present since 0.10.0. Residual PVC-orphan on finalizer removal is tracked in #2056. by @somanchi004-code in #2057

• Operator no longer keeps probing a ZooKeeper ensemble it can no longer reach. A disconnected ZK session leaked the client and re-dialed stale addresses until operator restart. Every terminal state now closes the connection. Root-path ensure is cancellable, skipped on empty recovery passes, and raises CreateFailed after retries (still non-fatal). by @alex-zaitsev in #2073

• Oversized CRD ConfigMap broke GitOps installs of the Helm chart. The CRD-install hook packed three CRDs into one ConfigMap past the 256Ki annotation ceiling, so ArgoCD client-side apply and helm template | kubectl apply failed. Each CRD now gets its own ConfigMap. helm install / helm upgrade were unaffected. by @sunsingerus

• Stale ClickHouse HTTP connection pools are reset after connection failures. by @norrs in #2068

• A failed read is no longer mistaken for a missing object. Only NotFound means absent; any other error — RBAC still propagating, an overloaded API server — escalated to recovery, deleting a StatefulSet and its pods once a later read succeeded, or reporting phantom data loss on a Keeper PersistentVolumeClaim. Such reads now raise a host reconcile error and retry on the next pass. Affects ClickHouse and Keeper alike. by @sunsingerus

• Lost-volume recovery no longer ignores shutdown. The poll that waits up to an hour for a PersistentVolumeClaim to disappear did not observe the operator context, so a stop issued during it could be held up until the poll ran out. by @sunsingerus

New Contributors

• @fhoekstra made their first contribution in #2065
• @somanchi004-code made their first contribution in #2057
• @norrs made their first contribution in #2068
• @erikasb made their first contribution in #2067
• @identw made their first contribution in #2074
• @Tyagiquamar made their first contribution in #2077

Full Changelog: release-0.27.3...release-0.27.4