[gardener]

⚠️ Breaking Changes

• [OPERATOR] The Logging feature gate is removed, now logging is enabled/disabled/configured via the gardenlet configuration. (gardener/gardener#5337, @acumino)
• [OPERATOR] A new field Enabled is introduced in Logging field of the gardenlet configuration to enable/disable logging. By default it is set to false. (gardener/gardener#5337, @acumino)
• [DEPENDENCY] The default leader election of extensions has been changed from configmapsleases to leases. Please make sure, that you had at least gardener@v1.17.0 in your go.mod before upgrading to this version so that it has successfully acquired leadership with the hybrid resource lock (configmapsleases) at least once. (gardener/gardener#5456, @acumino)
• [DEPENDENCY] The controllercmd.LogErrAndExit and controller.*EventLogger helper functions have been dropped in favor of proper error handling and structured logging, as their usage was not aligned with our logging guideline. (gardener/gardener#5442, @timebertt)

✨ New Features

• [OPERATOR] The kube-apiserver's Prometheus metrics have been extended with some metrics that describe the costs of handling LIST requests. They are as follows. (gardener/gardener#5445, @acumino)
  • apiserver_cache_list_total: Counter of LIST requests served from watch cache, broken down by resource_prefix and index_name
  • apiserver_cache_list_fetched_objects_total: Counter of objects read from watch cache in the course of serving a LIST request, broken down by resource_prefix and index_name
  • apiserver_cache_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from watch cache, broken down by resource_prefix
  • apiserver_cache_list_returned_objects_total: Counter of objects returned for a LIST request from watch cache, broken down by resource_prefix
  • apiserver_storage_list_total: Counter of LIST requests served from etcd, broken down by resource
  • apiserver_storage_list_fetched_objects_total: Counter of objects read from etcd in the course of serving a LIST request, broken down by resource
  • apiserver_storage_list_evaluated_objects_total: Counter of objects tested in the course of serving a LIST request from etcd, broken down by resource
  • apiserver_storage_list_returned_objects_total: Counter of objects returned for a LIST request from etcd, broken down by resource
• [OPERATOR] Gardener API Server now supports configuration for enabling service account token volume projection. It is exposed through the .Values.global.apiserver.serviceAccountTokenVolumeProjection section in the respective chart's values. (gardener/gardener#5431, @dimityrmirchev)
• [OPERATOR] It is now possible to configure a user instead of a serviceaccount subject in the clusterrolebinding for the Gardener API Server when using virtual garden setup by setting .Values.global.virtualGarden.apiserver.user.name. (gardener/gardener#5431, @dimityrmirchev)
• [OPERATOR] Gardener Scheduler now supports configuration for enabling service account token volume projection. It is exposed through the .Values.global.scheduler.serviceAccountTokenVolumeProjection section in the respective chart's values. (gardener/gardener#5430, @dimityrmirchev)
• [OPERATOR] It is now possible to configure a user instead of a serviceaccount subject in the clusterrolebinding for the Gardener Scheduler when using virtual garden setup by setting .Values.global.virtualGarden.scheduler.user.name. (gardener/gardener#5430, @dimityrmirchev)
• [OPERATOR] Gardener Controller Manager now supports configuration for enabling service account token volume projection. It is exposed through the .Values.global.controller.serviceAccountTokenVolumeProjection section in the respective chart's values. (gardener/gardener#5429, @dimityrmirchev)
• [OPERATOR] It is now possible to configure a user instead of a serviceaccount subject in the clusterrolebinding for the Gardener Controller Manager when using virtual garden setup by setting .Values.global.virtualGarden.controller.user.name. (gardener/gardener#5429, @dimityrmirchev)
• [OPERATOR] The unused static ServiceAccount tokens for the controllers part of kube-controller-manager in the kube-system namespace of shoot clusters are now invalidated. Note that the tokens for the {node,route,service} controllers will only be invalidated for Kubernetes 1.21+ clusters since the cloud-controller-managers of prior versions still rely on them. (gardener/gardener#5422, @rfranzke)
• [OPERATOR] Gardener Admission Controller now supports configuration for enabling service account token volume projection. It is exposed through the .Values.global.admission.serviceAccountTokenVolumeProjection section in the respective chart's values. (gardener/gardener#5386, @dimityrmirchev)
• [OPERATOR] It is now possible to configure a user instead of a serviceaccount subject in the clusterrolebinding for the Gardener Admission Controller when using virtual garden setup by setting .Values.global.virtualGarden.admission.user.name. (gardener/gardener#5386, @dimityrmirchev)
• [DEVELOPER] A new logcheck tool has been added: it aims at making logs across Gardener components more consistent and help detect programmer-level errors early on. Read more about it in the tool's documentation. (gardener/gardener#5442, @timebertt)
• [DEVELOPER] Functions RESTConfigFromKubeconfig and RESTConfigFromClientConnectionConfiguration in package /pkg/client/kubernetes now support an allowedFields parameter which can be used to allow additional fields in the kubeconfig when creating clients. (gardener/gardener#5386, @dimityrmirchev)

🐛 Bug Fixes

• [USER] Fixed a bug, that broken Shoot system components didn't cause failing Shoot health checks. (gardener/gardener#5453, @timebertt)
• [USER] The EveryNodeReady shoot condition is now correctly computed even if a worker pool overwrites the Kubernetes version. (gardener/gardener#5437, @rfranzke)
• [OPERATOR] Increase the ginkgo timeout for default shoot serial test suite to prevent timeouts on tests (gardener/gardener#5428, @BeckerMax)
• [DEPENDENCY] Fixes a bug that caused only one Machine object to be restored, and all others to be recreated during control plane migration. (gardener/gardener#5471, @plkokanov)
• [DEPENDENCY] The generic Worker actuator is now more resilient to status updates that fail because of conflicts. (gardener/gardener#5451, @ialidzhikov)

📖 Documentation

• [OPERATOR] Add a GEP for Shoot cluster CA rotation (gardener/gardener#5395, @BeckerMax)

🏃 Others

• [USER] The used PriorityClass for kube-proxy was changed from system-cluster-critical to system-node-critical. (gardener/gardener#5438, @rfranzke)
• [OPERATOR] Add a size based retention policy to the aggregate prometheus (gardener/gardener#5450, @istvanballok)
• [OPERATOR] Federate vpn and api server availability metrics (gardener/gardener#5448, @wyb1)
• [OPERATOR] Error messages containing duplicate zones and overlapping zones in their description that can happen when reconciling DNSProviders are now classified as ERR_CONFIGURATION_PROBLEM. (gardener/gardener#5447, @plkokanov)
• [OPERATOR] Change the Throttle factor metric to Throttle % in the Kubernetes Pods dashboard (gardener/gardener#5446, @istvanballok)
• [OPERATOR] These metrics are only available for the shoot with k8s v1.23 and upwards. (gardener/gardener#5445, @acumino)
• [OPERATOR] The audit policy configmap protection by the Shoot reference controller of gardener-controller-manager is now enabled by default (but still configurable). (gardener/gardener#5426, @ialidzhikov)
• [OPERATOR] Keep the _count and _sum series of the api server histogram metrics (gardener/gardener#5424, @istvanballok)
• [OPERATOR] The following golang dependencies have been upgraded, please consult the upstream release notes and this issue for guidance on upgrading your golang dependencies when vendoring this gardener version: (gardener/gardener#5421, @acumino)
  • k8s.io/* to v0.23.3
  • sigs.k8s.io/controller-runtime to v0.11.0
  • sigs.k8s.io/controller-tools to v0.8.0
• [OPERATOR] Add a dashboard for API Server request duration and response size (gardener/gardener#5419, @wyb1)
• [OPERATOR] Use max instead of sum when counting etcd objects (gardener/gardener#5418, @wyb1)
• [OPERATOR] Add new collector configs to node exporter (gardener/gardener#5396, @teturou8001)
• [OPERATOR] Topology spread constraints and anti affinity are now defined in the coredns deployment for zones to better spread coredns pods across multiple zones. (gardener/gardener#5393, @DockToFuture)
• [OPERATOR] A validation is added to validate that the configured worker pools maximum nodes count do not exceed maximum nodes count allowed by the Pods CIDR (spec.networking.pods). (gardener/gardener#5389, @shafeeqes)
• [OPERATOR] gardener-apiserver does now support a field selector for Bastions by spec.shootRef.name (gardener/gardener#5382, @shafeeqes)
• [OPERATOR] Add feature flag DisableDNSProviderManagement. This is part of Move DNSProvider capabilities out of g/g #5270. (gardener/gardener#5349, @MartinWeindel)
• [OPERATOR] Allows cluster owner to switch between horizontal and cluster-proportional autoscaling of coredns (gardener/gardener#5275, @ScheererJ)

📰 Noteworthy

• [DEVELOPER] A new document has been added describing the development tasks for supporting a new minor Kubernetes version. (gardener/gardener#5461, @rfranzke)

Docker Images

admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.41.0
apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.41.0
controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.41.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.41.0
landscaper-gardenlet: eu.gcr.io/gardener-project/gardener/landscaper-gardenlet:v1.41.0
scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.41.0
seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.41.0
resource-manager: eu.gcr.io/gardener-project/gardener/resource-manager:v1.41.0
landscaper-controlplane: eu.gcr.io/gardener-project/gardener/landscaper-controlplane:v1.41.0