[gardener]

⚠️ Breaking Changes

• [DEVELOPER] The monitoring scrape configurations (particularly the tls_config and authorization section) of extension controllers must be adapted such that they match the example in https://github.com/gardener/gardener/blob/master/docs/extensions/logging-and-monitoring.md#extensions-monitoring-integration. (gardener/gardener#5008, @rfranzke)

✨ New Features

• [USER] It's now possible to override the grace periods for the infrastructure cleanup wait step in the shoot deletion by specifying the shoot.gardener.cloud/cleanup-infrastructure-resources-grace-period-seconds annotation on the Shoot (default behaviour: "300"). Please be aware that overriding this value might lead to orphaned infrastructure artifacts. (gardener/gardener#5044, @rfranzke)
• [USER] If the WorkerPoolKubernetesVersion feature gate is enabled then it is possible to specify the Kubernetes version per worker pool for Shoots by setting .spec.provider.workers[].kubernetes.version. Please consult this document for more information. (gardener/gardener#4971, @rfranzke)
• [OPERATOR] It is now possible to disable the dependency-watchdogs in the seed cluster by configuring the .spec.settings.dependencyWatchdog section. Please consult the documentation for more information. (gardener/gardener#5075, @rfranzke)
• [OPERATOR] A new shoot status label value unknown that corresponds to Unknown conditions has been introduced. (gardener/gardener#5041, @stoyanr)
• [OPERATOR] Updating the shoot status label has been moved to GCM. (gardener/gardener#5022, @stoyanr)
• [OPERATOR] The copying of seed conditions to shoots registered as seeds has been moved to GCM. (gardener/gardener#5013, @stoyanr)
• [OPERATOR] Node resource capacity (cpu, gpu, memory) are now filled in the extensionsv1alpha1.Worker resource under spec.pools[].nodeTemplate.capacity from the CloudProfile for the corresponding machine type. These resources will be carried to the MachineClass and will later be used by cluster-autoscaler for scale-from-zero. (gardener/gardener#4980, @himanshu-kun)
• [OPERATOR] The WorkerPoolKubernetesVersion feature gate must only be enabled when all provider extensions explicitly announce in their release notes that they support specific Kubernetes versions per worker pool. Otherwise, worker nodes of shoot clusters might be misconfigured or rolled out unexpectedly. (gardener/gardener#4971, @rfranzke)
• [DEVELOPER] It is now possible to provide an ObjectSelector when registering an extension webhook by configuring github.com/gardener/gardener/extensions/pkg/webhook.Webhook. (gardener/gardener#5043, @rfranzke)

🐛 Bug Fixes

• [USER] The node bootstrapping mechanism has been enhanced to retry failed hyperkube preload operations. (gardener/gardener#5019, @timebertt)
• [OPERATOR] The validation whether referenced extension types are actually registered in the system is now only performed when a resource is newly created or when its spec section has changed. (gardener/gardener#5049, @rfranzke)
• [OPERATOR] A bug has been fixed which could prevent proper deletion of ControllerInstallations when a Seed was marked for deletion. (gardener/gardener#5047, @rfranzke)
• [DEVELOPER] A bug has been fixed which caused issues when hack/generate-seed-crds.sh was called with an empty <file-name-prefix>. (gardener/gardener#5053, @timuthy)
• [DEVELOPER] Parts of make-generate are no longer executed in parallel. (gardener/gardener#5020, @BeckerMax)

🏃 Others

• [OPERATOR] Removed log output of readiness/liveness probes from vpn-seed-server log. (gardener/gardener#5087, @ScheererJ)
• [OPERATOR] Fixed several panels and corresponding metric ingestion in the istio mesh dashboard. (gardener/gardener#5077, @ScheererJ)
• [OPERATOR] It's now possible to configure the SerializeImagePulls fields for the kubelet configuration (defaults: true) in the Shoot API via .spec.{provider.workers[]}.kubernetes.kubelet.SerializeImagePulls. (gardener/gardener#5074, @shafeeqes)
• [OPERATOR] Provide access logs for workload requests going through apiserver-proxy. (gardener/gardener#5065, @ScheererJ)
• [OPERATOR] Access logging in api-server-proxy now works. (gardener/gardener#5060, @ScheererJ)
• [OPERATOR] Enable metrics collection of and introduce dashboard for envoy proxy side car used in reversed vpn (gardener/gardener#5055, @ScheererJ)
• [OPERATOR] Expose a metric on frequent container restarts in the seed (gardener/gardener#5048, @istvanballok)
• [OPERATOR] Expose a metric for inconsistent persistent volume sizes (gardener/gardener#5040, @istvanballok)
• [OPERATOR] ⚠️ Due to the github.com/gardener/etcd-druid update, etcd pods of shoot clusters will be restarted during their next reconciliation (e.g. within next maintenance time window, manual reconciliation, spec updates). (gardener/gardener#5037, @abdasgupta)
• [OPERATOR] Enable access log of envoy proxy side car of reversed vpn server for better visibility. (gardener/gardener#5035, @ScheererJ)
• [OPERATOR] The condition handling was improved in Gardener which sometimes resulted in conditions having outdated reasons or messages. (gardener/gardener#5021, @timuthy)
• [OPERATOR] During the restoration phase of control plane migration ETCD backups will be copied from the backup bucket of the source seed to the backup bucket of the destination seed. (gardener/gardener#4894, @plkokanov)
• [OPERATOR] The Gardenlet wait timeout for infrastructure reconciliation has been increased from 30 seconds to 3 minutes. This should reduce unnecessary reconciliations and improve the UX when updating the infrastructure of failed Shoots to a valid configuration. (gardener/gardener#4881, @danielfoehrKn)
• [OPERATOR] The gardenlet shoot controller will now set the owner check configuration parameters in the etcd-main Etcd resource. This will cause etcd-backup-restore to disable the cluster if the owner domain name no longer resolves to the specified owner ID. The creation and checking of owner DNS record can be disabled via the spec.settings.ownerChecks seed setting. (gardener/gardener#4813, @stoyanr)

📰 Noteworthy

• [OPERATOR] kube-rbac-proxy does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5010, @rfranzke)
• [OPERATOR] vpa-{admission-controller,recommender,updater} do no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5009, @rfranzke)
• [OPERATOR] kube-state-metrics does no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. prometheus has such a token as well, but for backwards-compatibility it also still has access to its client certificate (this will be dropped in the future). (gardener/gardener#5008, @rfranzke)
• [OPERATOR] kube-controller-manager and cluster-autoscaler do no longer use a client certificate but an auto-rotated ServiceAccount token which is only valid for 12h. (gardener/gardener#5007, @rfranzke)
• [OPERATOR] The gardener-resource-manager's TokenInvalidator and the ProjectedTokenMount webhooks are now enabled for the seed and shoot clusters. (gardener/gardener#5002, @rfranzke)
• [DEVELOPER] The ReversedVPN feature gate is now activated by default for local development. (gardener/gardener#5045, @rfranzke)
• [DEVELOPER] In order to make provider extensions ready for properly handling Shoot clusters with overridden Kubernetes versions per worker pool, you need to revendor the extensions library. (gardener/gardener#4971, @rfranzke)

[etcd-backup-restore]

🏃 Others

• [USER] Compaction job will now throw warning instead of error if revisions are already compacted (gardener/etcd-backup-restore#358, @shreyas-s-rao)
• [OPERATOR] The backup-restore sidecar will now check if the owner domain name resolves to the specified owner ID and if not, take a final full snapshot and disable the cluster. (gardener/etcd-backup-restore#383, @stoyanr)
• [OPERATOR] Added '--etcd-snapshot-timeout' and '--etcd-defrag-timeout' CLI flags in 'server' and 'compact' subcommands (gardener/etcd-backup-restore#361, @ishan16696)
• [OPERATOR] Updated number of chunks while uploading to never exceed the cloud provider limits. (gardener/etcd-backup-restore#334, @abdasgupta)

📰 Noteworthy

• [OPERATOR] Added health package to allow backup-restore to renew member leases to indicate member health and snapshot leases to indicate snapshots being taken successfully (gardener/etcd-backup-restore#382, @aaronfern)
  • Added flags to the server and compact subcommand. --enable-snapshot-lease-renewal to enable snapshot lease renewal, --enable-member-lease-renewal to enable member lease updates, full-snapshot-lease-name to specify the full snapshot lease name, and delta-snapshot-lease-name to specify the delta snapshot lease name
  • Pod name and namespace has to be passed via environment variables (POD_NAME and POD_NAMESPACE) when running the server subcommand when --enable-member-lease-renewal flag is set to true
  • Pod namespace needs to be passed via an environment variable (POD_NAMESPACE) when running the server subcommand when --enable-snapshot-lease-renewal flag is set to true
  • Pod namespace needs to be passed via an environment variable (POD_NAMESPACE) when running the compact subcommand when --enable-snapshot-lease-renewal flag is set to true
• [OPERATOR] Added compaction command support to ETCD Backup Restore. Users can now use compaction sub command with etcdbrctl tool to compact basesnapshot and all it's subsequent deltasnapshot to one single compacted snapshot. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] Snapshots are not stored under separate prefix (Backup-XXX) in backup storage. Instead, all of the snapshots are stored under v2 backup version prefix in a flat structure. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] Support for v1 backup version is still there but only for restoration and snapshot garbage collection. So backup storages that had v1 backup version and where snapshots were stored under separate prefix (Backup-XXX), can still be used with compaction sub command (as well as for restoration). However, the new snapshots will be stored under v2 backup version prefix. (gardener/etcd-backup-restore#301, @abdasgupta)
• [OPERATOR] The support for the v1 backup structure with separate prefix (Backup-XXX) will be dropped in a subsequent release. (gardener/etcd-backup-restore#301, @abdasgupta)

[etcd-custom-image]

📰 Noteworthy

• [OPERATOR] Update etcd version from v3.4.13-bootstrap to v3.4.13-bootstrap-1. ⚠️ This will cause a restart of the etcd. (gardener/etcd-custom-image@10c534ecbfe7)

[etcd-druid]

⚠️ Breaking Changes

• [OPERATOR] CronJob is no more used to schedule compaction job at regular interval. Instead, we are using Job. (gardener/etcd-druid#235, @abdasgupta)
  • Therefore, BackupCompactionSchedule field is removed from ETCD backup spec, as it was only necessary for scheduling CronJob.
• [DEVELOPER] The sub-module github.com/gardener/etcd-druid/api has been removed. Please use github.com/gardener/etcd-druid instead if your module(s) depend on etcd-druid. (gardener/etcd-druid#244, @timuthy)
• [DEVELOPER] Switch to github.com/gardener/etcd-druid-api if you vendor only the API of etcd-druid. (gardener/etcd-druid#169, @amshuman-kr)

✨ New Features

• [USER] druid will now also reconcile a serviceaccount, a role, and a rolebinding as part of it's etcd reconcile flow and associate it with the etcd pod (gardener/etcd-druid#233, @aaronfern)
• [OPERATOR] A new flag --enable-backup-compation has been introduced which globally enables automatic compaction of backups. (gardener/etcd-druid#258, @timuthy)
• [OPERATOR] A new controller named lease controller has been introduced. Lease controller will be responsible for creating compaction job based on the delta event lease. (gardener/etcd-druid#235, @abdasgupta)
  • For this, two new Leases are introduced: One to hold the value of the latest full snapshot revision and one for the last delta revision.
• [OPERATOR] Added support for etcd druid to regularly schedule backup compactions via configurable etcd spec spec.backup.compactionSchedule (gardener/etcd-druid#197, @aaronfern)
• [OPERATOR] Various condition and etcd member checks have been added to Etcd-Druid. The results of those checks will be reflected in the etcd.status sub-resource. (gardener/etcd-druid#188, @timuthy)
  • Conditions:
  • Ready check: Checks if resource has enough Ready members in status.members to fulfill the quorum.
  • AllMembersReady check: Checks if all members in status.members are Ready.
  • Members:
  • Ready check: Treats the LastUpdateTime as a heartbeat and checks if it is within the expected time range (configurable via --etcd-member-threshold).
• [OPERATOR] A re-sync mechanism has been added for the Custodian controller. The new flag --custodian-sync-period (default 30s) controls the duration after which the Custodian controller re-enqueues etcd resources for reconciliation. This can be considered as a health check interval. (gardener/etcd-druid#188, @timuthy)
• [OPERATOR] It is now possible to configure the count of custodian controller workers by --custodian-workers. (gardener/etcd-druid#180, @timuthy)

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that led to multiple update conflicts when the etcd resource was reconciled. (gardener/etcd-druid#264, @timuthy)
• [OPERATOR] The operation annotation is now removed before reconciling the Etcd resource. (gardener/etcd-druid#205, @shreyas-s-rao)
• [OPERATOR] If operation annotation is to be honoured, the reconciliation predicates now match only if either operation annotation is present, or last operation didn't succeed or if the resource is undergoing deletion. No other change (if not accompanied by any of these conditions) will trigger reconciliation. (gardener/etcd-druid#202, @amshuman-kr)

📖 Documentation

• [OPERATOR] The multi-node etcd proposal has been updated and now considers ephemeral volumes not being an integral part of the multi-node feature set. (gardener/etcd-druid#256, @timuthy)

🏃 Others

• [OPERATOR] It is now possible to specify owner check parameters in the Etcd resource. (gardener/etcd-druid#239, @stoyanr)
• [OPERATOR] Remove finalizers from Etcd resources after waiting for statefulset. (gardener/etcd-druid#222, @amshuman-kr)
• [OPERATOR] Added a new CLI flag --enable-compaction-tempfs to etcd druid to enable tempfs in the compaction job volumeMount (defaults to false) (gardener/etcd-druid#220, @aaronfern)
• [OPERATOR] Two new fields added to etcd spec, etcdSnapshotTimeout and etcdDefragTimeout which configure the snapshotter timeout and defragmentation timeout respectively of etcd-backup-restore (gardener/etcd-druid#216, @aaronfern)
• [OPERATOR] Druid now fetches Lease resources in order to derive the readiness state of an etcd cluster member. This serves as a preparation for the etcd multi-node feature. (gardener/etcd-druid#214, @timuthy)
• [OPERATOR] Updated number of chunks while uploading to never exceed the cloud provider limits. (gardener/etcd-druid#182, @amshuman-kr)
• [OPERATOR] The reconciliation loop count of the custodian controller has been improved. Now, reconciliation only happens at relevant create/update/delete events. (gardener/etcd-druid#180, @timuthy)
• [OPERATOR] Removed synchronisation before updating ETCD status. (gardener/etcd-druid#174, @abdasgupta)
• [DEVELOPER] A new Make target check-generate has been added to check if generated code and the vendor dir are up-to-date. (gardener/etcd-druid#177, @timuthy)

📰 Noteworthy

• [OPERATOR] A new module github.com/gardener/etcd-druid/api can be used to get the API definitions. (gardener/etcd-druid#169, @amshuman-kr)

• Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.37.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.37.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.37.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.37.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.37.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.37.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.37.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.37.0