[gardener]

⚠️ Breaking Changes

• [DEPENDENCY] The WorkerDelegate must implement method GetMachineControllerManagerCloudCredentials returning map with cloud credential keys and values just like they are used by the machine-controller-manager. (#3224, @vpnachev)
• [DEPENDENCY] The deprecated functions in the terraformer library (SetVariablesEnvironment and GenerateVariablesEnvironment) have been removed. (#3223, @timebertt)
• [DEPENDENCY] The Terraformer functions have been changed to allow passing proper contexts. Please adapt your usage accordingly. (#3223, @timebertt)
• [DEPENDENCY] The terraformer library was switched to logr instead of logrus in order to have more consistent and readable logging in the infrastructure controllers of provider extensions. Please adapt your usage accordingly. (#3223, @timebertt)

✨ New Features

• [USER] The shoot reconciler sets the conditions to Progressing after it finished a successful reconciliation, and the care controller starts to re-evaluate the health status after this happened. This helps end-users to better understand whether their cluster is indeed healthy after a reconciliation. Earlier, it could take up to 30s / 1m (based on the configured care controller sync period) until the actual status is reflected. (#3251, @rfranzke)
• [OPERATOR] The shoot controller inside the gardenlet has been adapted to cater with large Gardener landscapes: (#3242, @rfranzke)
  • 1. When the gardenlet has already reconciled a shoot cluster during its maintenance time window then it doesn't reconcile it again. Instead, it computes a random duration for the next time window and requeues the shoot. Already reconciled shoots are those whose last reconciliation was less then 24h ago.
  • 1. When the gardenlet is (re)started then it does no longer reconcile all shoots immediately whose maintenance time windows are met. Instead, it computes a random time for the current time window and requeues the shoot ("jittering", i.e., spreading the load). This will have the effect that not all shoots are getting reconciled at the same time right after startup.

🐛 Bug Fixes

• [USER] The KUBERNETES_SERVICE_HOST environment variable injected when APIServerSNI is enabled no longer includes a trailing dot (being a Fully Qualified Domain Name) due to several homebrew kubernetes clients not properly handling it and sending wrong server name when initiating a TLS conneciton. (#3235, @mvladev)
• [OPERATOR] apieserver-proxy now uses system-node-critical priority class. Memory limit is also increased to avoid OOM killer. (#3282, @mvladev)
• [OPERATOR] A bug has been fixed that caused the vpa-admission-controller to not being able to update its status (inside Lease object) when its enabled for shoot clusters. (#3265, @rfranzke)
• [OPERATOR] Fix an error during bootstrapping of fresh Seeds (#3262, @BeckerMax)
• [OPERATOR] A worker controller is now ensuring that all machine class secrets have up-to-date cloud credentials. (#3224, @vpnachev)
• [OPERATOR] A bug has been fixed which can lead to Seeds not getting ready when an image vector overwrite for the etcd-druid is configured. (#3212, @rfranzke)

🏃 Others

• [USER] Support scale to/from zero for MCM OOT providers - AWS, Azure. (#3276, @prashanth26)
• [USER] The severity of the user exposed logs is unified and recognizable by the Grafana. (#3270, @vlvasilev)
• [OPERATOR] The target cache of gardener-resource-manager instances running in the Shoot control plane is disabled now. (#3268, @timebertt)
• [OPERATOR] Gardener has improved infrastructure processing procedures in oder to avoid unnecessary reconciliation cycles. (#3255, @timuthy)
• [OPERATOR] Add Loki multitenancy integration test. (#3253, @vlvasilev)
• [OPERATOR] Istio is updated to 1.18.0. (#3250, @mvladev)
• [OPERATOR] Parse the time zone of a log when reading it from the node /var/log/containers directory. (#3219, @vlvasilev)
• [OPERATOR] When fluent-bit containers runs the tail plugin starts to read a file from the head(like it was prior fluent-bit 1.6). (#3219, @vlvasilev)
• [OPERATOR] Make the readiness and liveness probe fail after 30 seconds and the liveness probes starts after 90 seconds. (#3219, @vlvasilev)
• [OPERATOR] get, list and watch for Pods are removed from the fluent-bit RBAC as no longer needed. (#3219, @vlvasilev)
• [OPERATOR] Upgrade Prometheus to v2.22.2. Sometimes Prometheus would have the error mmap: invalid argument. Prometheus v2.22.1+ provides a fix for this issue. (#3213, @wyb1)
• [OPERATOR] metrics-server, node-problem-detector and vpn-shoot now have dnsPolicy: Default set to them to remove dependency to coredns. (#3211, @mvladev)
• [OPERATOR] Sort logs to fix out of order issue (#3188, @Kristian-ZH)
• [OPERATOR] The output plugin exposes custom metrics (#3188, @Kristian-ZH)
• [OPERATOR] Modified fluent-bit dashboard to include the new metrics (#3188, @Kristian-ZH)
• [OPERATOR] Fluent-bit tail plugin DB synchronization is set to FULL to avoid log duplication when fluent-bit pod is restarted. (#3091, @vlvasilev)
• [OPERATOR] Loki chunk_target_size option is set to 1536000 bytes as recommended by Grafana (#3091, @vlvasilev)
• [DEVELOPER] Integration test for the logging is added simulating seed with 100 shoots (#2996, @vlvasilev)

📰 Noteworthy

• [USER] The Shoot garbage collector now also deletes failed Pods with the reason OutOf* in the Seed namespace and the kube-system namespace of the Shoot. (#3248, @timebertt)
• [USER] The system components that were previous specifying label garden.sapcloud.io/role: (optional-addon|monitoring|system-component) are now adapted to specify gardener.cloud/role: (optional-addon|monitoring|system-component). (#3220, @ialidzhikov)
• [OPERATOR] Forbid control plane migration between Seeds with different cloud providers. (#3254, @plkokanov)
• [OPERATOR] The gardenlet enqueues shooted seeds immediately (without configured jitter) when the shooted seed's spec was changed or when the config in the use-as-seed annotation was changed. This enabled a faster rollout of the gardenlet. (#3249, @rfranzke)
• [OPERATOR] gardenlet is now restarted if APIServerSNI is enabled on the Seed cluster. (#3226, @mvladev)
• [OPERATOR] The Shoot namespace in the Seed no longer specifies label garden.sapcloud.io/role: shoot. (#3220, @ialidzhikov)
• [OPERATOR] Upgraded etcd version from v3.3.17 to v3.4.13 and moved from quay.io/coreos/etcd to Gardener-specific custom etcd image eu.gcr.io/gardener-project/gardener/etcd. ⚠️ This will cause an etcd restart. (#3205, @gardener-robot-ci-2)

[autoscaler]

🏃 Others

• [USER] Support scale to/from zero for MCM OOT providers - AWS, Azure. (gardener/autoscaler#65, @prashanth26)
• [OPERATOR] Ignore gardener-specific and csi-specific labels while comparing nodegroups (gardener/autoscaler#62, @hardikdr)

[etcd-backup-restore]

🏃 Others

• [OPERATOR] Validator now double checks latest revision by starting an embedded etcd if DB-based revision check fails. This can potentially avoid unnecessary data restoration when etcd terminates abnormally. (gardener/etcd-backup-restore#275, @ishan16696)
• [OPERATOR] Fix missing alternate full snapshots for some unhibernating shoots. (gardener/etcd-backup-restore#272, @shreyas-s-rao)
• [OPERATOR] Added support for OpenShift Container Storage (OCS) S3 storage type. (gardener/etcd-backup-restore#261, @stoyanr)
• [OPERATOR] Fixed the issue with consecutive restoration if backup-restore sidecar doesn't restart in between. (gardener/etcd-backup-restore#259, @amshuman-kr)
• [OPERATOR] Clarify manual backup restore process (gardener/etcd-backup-restore#224, @jfortin-sap)
• [OPERATOR] Fix snapshot metric initialization. (gardener/etcd-backup-restore#223, @shreyas-s-rao)
• [DEVELOPER] Add TestMachinery integration. (gardener/etcd-backup-restore#249, @shreyas-s-rao)

📰 Noteworthy

• [USER] Add support for Dell EMC ECS object store with S3 protocol (gardener/etcd-backup-restore#254, @lcavajani)
• [OPERATOR] Added CLI-flags (max-call-send-message-size, max-request-bytes and max-txn-ops) to enable restoration for delta snapshots with large amount of data (large number of events or events with large data). (gardener/etcd-backup-restore#282, @abdasgupta)
• [DEVELOPER] Revendor etcd library to v3.4.13 and change import paths to go.etcd.io/etcd for etcd and go.etcd.io/bbolt for bbolt. (gardener/etcd-backup-restore#269, @shreyas-s-rao)
• [DEVELOPER] Integration tests can be executed on any given Kubernetes cluster using the make integration-test-cluster target against the cluster pointed to by the environment variable INTEGRATION_TEST_KUBECONFIG. (gardener/etcd-backup-restore#225, @shreyas-s-rao)

[etcd-druid]

🏃 Others

• [OPERATOR] Added support for OpenShift Container Storage (OCS) S3 storage type. (gardener/etcd-druid#98, @stoyanr)

[gardener-resource-manager]

✨ New Features

• [OPERATOR] gardener-resource-manager now logs its own version on startup or when executed with --version. (gardener/gardener-resource-manager#96, @timebertt)
• [DEVELOPER] Docker images built by make docker-images are now tagged and build with the commit hash appended to the version. (gardener/gardener-resource-manager#96, @timebertt)
• [DEVELOPER] The cache of the kubernetes client for the target cluster can now be disabled via the --target-disable-cache flag. (gardener/gardener-resource-manager#95, @timebertt)

🏃 Others

• [OPERATOR] gardener-resource-manager now uses a DynamicRESTMapper, which will reduce the amount of explicit discovery calls and faster reconciliation loops and some cases. (gardener/gardener-resource-manager#95, @timebertt)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.14.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.14.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.14.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.14.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.14.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.14.0