github enix/kube-image-keeper v3.0.0-beta.1

pre-release3 hours ago

kuik 3.0.0-beta.1 is the first beta of kuik v3, a rewrite of kuik from a clean tree. Every feature planned for 3.0 is in, except credentials from a cloud provider, but it is not stable yet: keep v2.3 for production. Features and breaking changes count from v2.3.1; fixes count from 3.0.0-alpha.6.

✨ Highlights

  • New resources: kuik v3 routes, copies and monitors pod images with 3 new cluster-scoped resources: ImageAlternative, ImageMirror and ImageMonitor.
  • Shared mirrors: an ImageMirror can share its destination registry with other clusters: each cluster only creates, checks and deletes its own tags.
  • 3 processes: kuik runs as a webhook, a reconciler and a secret syncer, each with its own Deployment and 2 replicas by default.
  • Paced registry reads: every background read of a registry is paced per host, to spare rate limits and quotas.
  • No upgrade from v2 (yet): install v3 on a cluster without v2.

⚠️ Breaking changes

  • From v2: v3 reads none of the v2 resources (ImageSetMirror, ClusterImageSetMirror, ReplicatedImageSet, ClusterReplicatedImageSet, ClusterImageSetAvailability) nor the v2 chart values. Install v3 on a cluster without v2; a migration guide is planned after 3.0. (#670)
  • From 3.0.0-alpha.4 or earlier: the chart requires Kubernetes 1.30 or later. (#752)
  • From 3.0.0-alpha.4 or earlier: with serviceAccount.create: false, set secretSyncer.serviceAccount.name to an existing ServiceAccount other than default, or the chart refuses to render. (#752)
  • From 3.0.0-alpha.5 or earlier: config.clusterID is removed, and a release that still sets it fails to render. To keep the tags your mirrors copied, create the ConfigMap kuik-cluster-id with the old value before upgrading, then remove config.clusterID (with --reuse-values, add --set config.clusterID=null), see Upgrading from a configured clusterID in the configuration docs. (#764, closes #727)
  • From 3.0.0-alpha.2 or earlier: the processes log JSON instead of console text, with no chart value to switch back. (#703)
  • From 3.0.0-alpha.2 or earlier: the root replicas value defaults to 2. (#701)

🚧 Known limits

Install

  • Cloud credentials: credentials from a cloud provider (auth.provider: aws, gcp, azure) are accepted but not used yet: kuik moves on to the next credential. Planned for 3.1.
  • One install per cluster: 2 secret syncers would refuse each other's writes.
  • Reconciler first: the webhook admits pods unchanged until the reconciler has created kuik-cluster-id, so never run with reconciler.replicas: 0.
  • Cluster ID: deleting the ConfigMap kuik-cluster-id, or its namespace, generates a new cluster ID and orphans every tag already copied. Never copy it to another cluster that shares a destination: both would delete each other's tags. See the Cluster ID section of the configuration docs.
  • Backups: a restore tool such as Velero is refused the pull Secrets kuik manages; the syncer recreates them, and the restore ends partially failed.

Mirroring

  • Tag deletion: a destination that never accepts tag deletion holds a deleted ImageMirror for good: set cleanup.enabled: false before deleting it, see Deleting an ImageMirror.
  • Harbor: Harbor refuses tag deletion through the registry API (any DELETE by tag answers 400), so cleanup deletes nothing on a Harbor destination, and a deleted ImageMirror with cleanup.enabled: true stays held by its finalizer. Set cleanup.enabled: false on a Harbor destination and use Harbor's tag retention rules; support for tag deletion is requested in goharbor/harbor#24213.

Monitoring

  • Drift digests: drift compares the digest the container runtime reports with the registry's: a runtime that reports a platform or config digest shows a tag as drifted, and a preloaded image is never checked for drift, see the ImageMonitor reference.
  • Retention after a restart: an image whose last pod goes away while no reconciler runs (restart, leader change) is not retained by an ImageMonitor.

Registries

  • Node runtime: the tls key and insecure: true configure kuik only: configure the nodes' container runtime for the same hosts, or pods fail to pull.

📋 Changelog

Features

25 new features

Routing

  • An ImageAlternative lists repositories that hold the same images, and the webhook places the first candidate that answers, as a fallback (rewritePolicy: OnFailure, the default) or ahead of the original (Always). (#682)
  • Images match on whole path segments, with repository for one repository or repositoryGroup for every repository under a path, and digest-pinned images are routed too. (#680, #682)
  • An entry marked unavailable: true declares a source gone, so an image from it is tried last. (#682)
  • The webhook records each decision on the pod in the kuik.enix.io/rewrites, kuik.enix.io/conceded-rewrites and kuik.enix.io/no-alternatives annotations. (#682)
  • A candidate the background loops report failing is tried last (webhook.availabilityCheck.demoteKnownFailures, on by default). (#751, #753)
  • The status of an ImageAlternative reports the pods it rewrote, its active fallbacks and the images no candidate could serve. (#696)

Pull secrets

  • A candidate's auth.secretRef is copied as a kuik-inject-<kind>-<name> pull Secret into the namespaces that need it, and added to the pod's imagePullSecrets. (#682, #701)

Mirroring

  • An ImageMirror copies the images of the pods it selects to a destination registry, digest unchanged, and routes to the copy unless rewritePolicy is None. (#682, #710)
  • Several clusters can share one destination: each copy carries a _<clusterID> tag suffix, and each cluster only creates, checks and deletes its own tags. (#710)
  • excludeImages keeps images out of a mirror with globs such as ghcr.io/foo-bar/** or **:latest. (#682, #710)
  • driftPolicy (Ignore, Warn, Sync) decides what happens when an upstream tag moves after its copy. (#710)
  • cleanup deletes the destination tags no pod uses any more once retention elapsed (168h by default). (#710)
  • A deleted ImageMirror waits while pods still run its copies, then deletes its tags. (#710)

Monitoring

  • An ImageMonitor checks that the images of the pods it selects are still available upstream, and lists the failing ones in status.unavailableImages. (#717, #732)
  • An image no pod uses any more stays monitored for unusedImageRetention (168h by default), so a CronJob's image stays checked between runs. (#717)
  • With driftDetection (on by default), it reports the tags whose upstream digest moved under the pods that run them, and the tags pods run under several digests. (#739)
  • With monitorAlternatives: true, it also checks the alternatives ImageAlternative resources offer for those images. (#745)

Registries

  • Background checks and copies are paced per host on fixed windows, set by registries in the global config. (#697)
  • Credentials are tried in the kubelet's order: the resource's auth, the pod's pull secrets, the most specific fallbackAuth entry, then anonymous. (#680)
  • The tls key of the global config trusts a private CA (caSecretRef) or a self-signed certificate (insecureSkipVerify) per registry host, and insecure: true on an entry reaches a registry over plain HTTP. (#682, #710, #767, closes #750)

Configuration

  • The global config file is validated whole and reloaded without a restart; a reload that does not validate keeps the previous config. (#680)
  • kuik generates its cluster ID and keeps it in the ConfigMap kuik-cluster-id. (#764, closes #727)

Observability

  • Kubernetes events on pods and resources, kuik_* Prometheus metrics for routing, copies, checks, anomalies and pacing, JSON logs, and conditions where only Ready is True when all is well. (#696, #703, #710, #732)

Chart

  • The webhook, the reconciler and the secret syncer run as 3 Deployments, with 2 replicas each by default and a standby that takes over the lease. (#672, #701, closes #548)
  • A ValidatingAdmissionPolicy refuses any Secret write by the secret syncer that is not a pull Secret it owns. (#752)

Fixes

8 fixes

Routing

  • A new webhook replica reports ready only once its routing index is built, so it never admits a pod unrouted. (#778)

Pull secrets

  • The pull Secret of an OnFailure resource is provided only in the namespaces its namespaceSelector selects. (#777)

Mirroring

  • A copy reads the original image first, before the alternatives of an ImageAlternative with rewritePolicy: Always. (#779)
  • A source that answers its check but fails the copy (a 429 on a blob, for instance) gives way to the next source instead of being retried at every window. (#779)
  • A tag missing from the destination is copied again at the next copy window of its host, ahead of the initial copies. (#779)
  • Adding or removing the cleanup finalizer no longer rewrites the spec, so cleanup.retention stays as written and GitOps tools see no drift. (#785)

Registries

  • A CA bundle with a truncated certificate block is rejected, even when a complete block follows it. (#787)

Observability

  • kuik_registry_requests_total is exported by the webhook and the reconciler, the processes that read registries. (#786)

Don't miss a new kube-image-keeper release

NewReleases is sending notifications on new releases.