[gardener]

Action Required

[USER] We are preparing a change that will lead to validation errors when the Project resource contains duplicates in the .spec.members[] list. For the time being, duplicates in this list are merged into a single member automatically by the Gardener API Server. In the future, this will no longer happen, instead, a validation error will be returned if a user sends a Project resource with duplicate members. Please adapt your API usage to not send any of such resources. (#3137, @rfranzke)
[USER] The project controller is now adapted to accept and maintain only the project namespace labels from the new API group - gardener.cloud/role=project and project.gardener.cloud/name=<project-name>. Until now the project controller was accepting and maintaining also the labels from the old API groups - garden.sapcloud.io/role=project and project.garden.sapcloud.io/name=<project-name>. With this change, the project controller removes the namespace labels associated to the old API group. If you are still using these deprecated labels, you need to adapt your machinery. (#3094, @ialidzhikov)
[USER] The already deprecated annotations shoot.garden.sapcloud.io/use-as-seed and shoot.garden.sapcloud.io/ignore-alerts are no longer respected by the corresponding Gardener components. If you are still using these deprecated annotations, you need to adapt your machinery to use respectively shoot.gardener.cloud/use-as-seed and shoot.gardener.cloud/ignore-alerts. You need to be careful with use-as-seed annotation as if you don't adapt to the new annotation and update your Gardener version to the one that no longer respects the deprecated annotation, this will be handled as deletion of the ShootedSeed and will mark the Seed for deletion. (#3094, @ialidzhikov)
[USER] A new webhook mutatingwebhookconfigurations.admissionregistration.k8s.io is deployed for all APIServerSNI- enabled clusters. It adds KUBERNETES_SERVICE_HOST environment variable pointing to the upstream Kube API Server. To disable this behavior: (#3082, @mvladev)
- label your Pods with apiserver-proxy.networking.gardener.cloud/inject: disable
- or label the entire namespace with apiserver-proxy.networking.gardener.cloud/inject: disable
- or label your Shoot resource with alpha.featuregates.shoot.gardener.cloud/apiserver-sni-pod-injector: disable to disable it cluster-wide.
[USER] For APIServerSNI-enabled clusters, Pods talking to the Kube API Server need to be allowed to connect to coredns running in kube-system namespace in order to resolve the hostname of the Kube API server. It also needs to have access to the IP from the default/kubernetes endpoint and the upstream IP of of the kube-apiserver. (#3082, @mvladev)
- If the Pod:
- is not matched by any NetworkPolicy - no action is required.
- is not injected with KUBERNETES_SERVICE_HOST, because the feature is disabled - no action is required.
- is matched by NetworkPolicies allowing ingress to coredns in kube-system and allows traffic top the default/kubernetes endpoint and the upstream upstream IP of of the kube-apiserver - no action is required.
- is matched by NetworkPolicies that do not allow access to coredns in kube-system and/or do not allows traffic top the default/kubernetes endpoint and/or the upstream upstream IP of of the kube-apiserver - a NetworkPolicy allowing such egress must be added e.g.:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-to-apiserver
spec:
  podSelector: {}
  egress:
  - to:
    - ipBlock:
        cidr: <IP from default/kubernetes endpoint>/32
    - ipBlock:
        cidr: <ip from apiserver FQDN e.g. nslookup api.foo.bar>/32
  - ports:
    - port: 8053
      protocol: UDP
    - port: 8053
      protocol: TCP
    to:
    - podSelector:
        matchExpressions:
        - key: k8s-app
          operator: In
          values:
          - kube-dns
      namespaceSelector:
        matchLabels:
          gardener.cloud/purpose: kube-system
  policyTypes:
  - Egress
  - Ingress
```
[DEVELOPER] Terraformer.SetVariablesEnvironment has been deprecated in favor of Terraformer.SetEnvVars. Please adapt your usage of the terraformer library accordingly. (#3204, @timebertt)
[DEVELOPER] The generic worker actuator interface does now include a new function MachineClass() runtime.Object that needs to be implemented. It is similar to MachineClassList() runtime.Object with the difference that it does not return the list object but the machine class object itself. (#3178, @rfranzke)
[DEVELOPER] The CleanupLeakedClusterRoles function has been removed from the generic worker actuator package. You can find more information about it here and here. (#3178, @rfranzke)
[DEVELOPER] A temporary workaround during the Cluster resource sync to the Seed by setting a fake Shoot status to prevent ShootNotFailed predicate in the extensions library from reacting false negatively is now cleaned up. Before upgrading to this version of Gardener, make sure that all of the extensions in your environment that use the ShootNotFailed predicate vendor github.com/gardener/gardener@v1.4.0 or above (that contains #2265). (#3097, @ialidzhikov)

Most notable changes

[USER] The Shoot now has a new constraint with type MaintenancePreconditionsSatisfied which indicates whether it's safe to maintain a shoot (see this document to get an overview what happens during maintenance). End-users should check this information to properly configure their clusters in order to avoid problems. (#3173, @rfranzke)
[USER] It is now possible to configure the kube-apiserver's --max-requests-inflight and --max-mutating-requests-inflight flags by setting the .spec.kubernetes.kubeAPIServer.requests.max{Non}MutatingInflight fields (default: {400}/200) in the Shoot specification. (#3141, @rfranzke)
[USER] It is now possible to configure the kube-controller-manager's --pod-eviction-timeout flag by setting the .spec.kubernetes.kubeControllerManager.podEvictionTimeout field (default: 2m0s) in the Shoot specification. (#3139, @rfranzke)
[OPERATOR] The controlplane Helm chart for Gardener does now expose a few more configuration options for the gardener-apiserver: (#3207, @rfranzke)
- .Values.global.apiserver.goAwayChance configures the --goaway-chance flag.
- .Values.global.apiserver.http2MaxStreamsPerConnection configures the --http2-max-streams-per-connection flag.
- .Values.global.apiserver.shutdownDelayDuration configures the --shutdown-delay-duration flag.
- .Values.global.requests.maxNonMutatingInflight configures the --max-requests-inflight flag.
- .Values.global.requests.maxMutatingInflight configures the --max-mutating-requests-inflight flag.
- .Values.global.requests.minTimeout configures the --min-request-timeout flag.
- .Values.global.requests.timeout configures the --request-timeout flag.
- .Values.global.watchCacheSizes.default configures the --default-watch-cache-size flag.
- .Values.global.watchCacheSizes.resources[] configures the --watch-cache-size flag.
[OPERATOR] A bug has been fixed that might led to orphaned machine resources in the shoot namespace in the seed that are stuck with the machine-controller-manager finalizer. (#3178, @rfranzke)
[OPERATOR] ManagedIstio and APIServerSNI can now be optionally configured via the new sni configuration in GardenletConfiguration, see the example configuration. This allows to use installation of Istio where the the ingressgateway is in another namespace. (#3143, @mvladev)
[OPERATOR] It is now possible to configure the TTL used for DNSEntry objects in the shoot controller via the gardenlet's component config (.controllers.shoot.dnsEntryTTLSeconds, default: 120). (#3142, @rfranzke)
[OPERATOR] It is now possible exclude specific Projects from the stale checks by annotating their related Namespaces with project.gardener.cloud/skip-stale-check=true. (#3136, @rfranzke)
[OPERATOR] The gardenlet rollout caused by shooted seed registrations is now spread by default within [0,5m]. You can overwrite this jitter period in the gardenlet's component configuration (.controllers.shootedSeedRegistration.syncJitterPeriod). (#3135, @rfranzke)
[OPERATOR] It is now possible to overwrite the feature gates in the gardenlet configuration for shooted seeds without the no-gardenlet option by setting featureGates.<name>={true,false}. (#3134, @rfranzke)
[OPERATOR] The audit policy config map reference protection controller introduced with v1.12.0 is now disabled by default. You can explicitly enable it in the gardener-controller-manager's component configuration by setting .controllers.shootReference.protectAuditPolicyConfigMaps=true. (#3117, @rfranzke)
[OPERATOR] A new webhook mutatingwebhookconfigurations.admissionregistration.k8s.io is deployed for all APIServerSNI-enabled clusters. It's running as a sidecar to the KubeAPI Server. (#3082, @mvladev)
[OPERATOR] The scheduler ensures that only candidate seeds with available capacity for shoots are considered during scheduling. The resource capacity and reservations can be configured in the gardenlet's component configuration. By default, the capacity for shoots in a seed is 200. (#3075, @stoyanr)
[OPERATOR] Operators can now define a default ResourceQuota resource which is automatically created in project related namespaces. Please consult the documentation (/docs/concepts/controller-manager.md) for more information. (#3072, @timuthy)
[OPERATOR] apiserver-proxy overload's manager is removed. (#3062, @mvladev)

Improvements

[USER] An issue has been fixed that can cause Shoots from being stuck in deletion due to invalid .spec.dns configuration. (#3168, @rfranzke)
[USER] A bug has been fixed that can cause Shoot resources from being stuck in Delete Succeeded state. (#3167, @rfranzke)
[USER] Containerd is supported in regions where gcr.io container registry can't be accessed. (#3164, @jia-jerry)
[USER] Fixed NetworkPolicy gardener.cloud--allow-dns to allow traffic from Pods with hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet. (#3162, @mvladev)
[USER] The gardenlet's shoot controller does now forget about the rate limiting when a shoot's deletion timestamp is set. This is to make starting deletion operations faster. (#3144, @rfranzke)
[USER] Project admins and viewers are now allowed to read corev1.ResourceQuota objects. (#3132, @timuthy)
[USER] Gardener now allows to edit a Shoot's metadata (including confirming the deletion), even if some referenced object (e.g. audit policy) has already been deleted. (#3116, @timebertt)
[USER] Missing audit policy ConfigMaps for Shoots are now ignored when trying to redeploy the kube-apiserver in the shoot deletion flow. (#3115, @rfranzke)
[USER] A bug that was preventing custom CA certificates to be installed on the shoot nodes is now fixed. (#3113, @vpnachev)
[USER] An issue causing CoreDNS dashboard to show always 'No Data' is now fixed. (#3089, @wyb1)
[USER] gardener-controller-manager's Shoot reference controller now also handles audit policy ConfigMap references. (#3071, @ialidzhikov)
[USER] An issue causing spec.kubernetes.kubelet.kubeReserved.pid field of the Shoot to be set for Kubernetes versions that don't support the corresponding feature is now fixed. (#3059, @ialidzhikov)
[USER] Node exporter properly reports filesystem size for operating systems that use an xfs filesystem (#3053, @wyb1)
[OPERATOR] Fix a bug where allow-to-seed-apiserver might not include the IP from the KUBERNETES_SERVICE_HOST environment variable of the gardenlet. (#3203, @mvladev)
[OPERATOR] Gardener triggers an infrastructure reconciliation during maintenance also for hibernated clusters. This ensures that the infrastructure is always up-to-date, even for long-term hibernated clusters. (#3196, @timuthy)
[OPERATOR] Gardenlet now logs the HTTP response of failed shoot health checks for checkAPIServerAvailability. (#3195, @timuthy)
[OPERATOR] Traffic is now allowed to cluster dns and the node local ipvs address to resolve a dns resolution issue with the NodeLocalDNS feature for dns names in control plane pods. (#3184, @DockToFuture)
[OPERATOR] When existing Namespaces are adopted for Projects then they will now be configured to remain even after the Project is being deleted later again. Earlier, such namespaces were also deleted together with the Project. Please note that this only takes effect for newly adopted project namespaces. (#3179, @rfranzke)
[OPERATOR] A bug in shoot deletion flow that was stuck waiting the kube-controller-manager to be scaled up when it is already deleted is now fixed. (#3176, @vpnachev)
[OPERATOR] The golang version is updated to 1.15.5. (#3175, @ialidzhikov)
[OPERATOR] metrics-server is upgraded to v0.4.1 and readiness and liveness probes now use http instead of tcp. (#3174, @mvladev)
[OPERATOR] Fluent-bit version upgraded to 1.6.4. (#3165, @vlvasilev)
[OPERATOR] Add Readiness and Liveness probe to the fluent-bit DaemonSet. (#3165, @vlvasilev)
[OPERATOR] Change the name of the gardener custom fluent-bit-to-loki plugin from loki to gardenerloki to avoid any plugin collisions with future version of the fluent-bit. (#3165, @vlvasilev)
[OPERATOR] Add minimum resource limit for the hvpa controller vpa (#3154, @BeckerMax)
[OPERATOR] The federated seed controller syncing the ShootState is no longer sends events for ShootState resources as they are not evaluated in a meaningful way anyways. (#3149, @rfranzke)
[OPERATOR] An issue has been fixed which caused the logging stack to skip logs for certain extension pods. (#3146, @Kristian-ZH)
[OPERATOR] An error has been fixed which caused the seed reconciliation (bootstrap) to fail if ManagedIstio is not enabled. (#3145, @timuthy)
[OPERATOR] A bug has been fixed that resulted in a stuck Seed deletion due to orphaned ManagedResources in the garden namespace. (#3133, @rfranzke)
[OPERATOR] When disabling APIServerSNI feature gate, existing LoadBalancer ports from ManagedIstio are not removed until all existing SNI-enabled Shoot clusters are migrated. (#3125, @mvladev)
[OPERATOR] A bug has been fixed that caused the gardenlet to deploy further instances of itself with its own self-generated server certificate. It prevents undesired redeployments of these further instances. (#3114, @rfranzke)
[OPERATOR] Increase kube-controller-manager VPA minAllowed (#3107, @ggaurav10)
[OPERATOR] gardenlet no longer sets the Shoots status to Failed too early when operation cannot be initialized (#3106, @ialidzhikov)
[OPERATOR] istio-ingressgateway now uses KEEPALIVE to downstream LoadBalancers to prevent idle timeout issues. (#3104, @mvladev)
[OPERATOR] The vpn deployment rolling strategy is improved so that the new pod is created before the old one is deleted. (#3100, @vpnachev)
[OPERATOR] A bug has been fixed that prevented the Loki HVPA recommendations from not being reverted. (#3098, @Kristian-ZH)
[OPERATOR] apiserver-proxy now uses tcp keepalive every 55 seconds to prevent idle timeouts between it and the SNI LoadBalancer. (#3092, @mvladev)
[OPERATOR] A race condition in Gardener's helm chart (/charts/gardener/controlplane) has been fixed. Earlier, the deployed ValidatingWebhookConfiguration potentially blocked the creation of Gardener ServiceAccounts. The validation is now excluded from namespaces with the label app: gardener. (#3088, @timuthy)
- ℹ️ Please make sure you either let /charts/gardener/controlplane also deploy the garden namespace, or add the label app=gardener to the namespace yourself.
[OPERATOR] A cache issue preventing Shoot reference controller of gardener-controller-manager to do not reconcile sometimes references for newly created Shoots when the CachedRuntimeClients feature gate is enabled is now fixed. (#3087, @ialidzhikov)
[OPERATOR] Fix a bug, where the shoot deletion controller tries to scale-up the already deleted or never created kube-controller-manager deployment. (#3077, @vpnachev)
[OPERATOR] Gardener now deletes (Cluster)RoleBindings of system components or addons, that were changed to an invalid state by endusers to be able to reconcile them back to the desired state. (#3074, @timebertt)
[OPERATOR] Remove egress restrictions for vpn-shoot pod as it was incompatible with the cilium network plugin. (#3073, @DockToFuture)
[OPERATOR] A bug that was preventing the ShootState resource to be updated with newly generated secrets is now fixed. (#3069, @vpnachev)
[OPERATOR] Remove pod panels from node-details dashboard. These panels only showed pods running in the kube-system namespace so the data is not very relevant and only leads to confusion. (#3068, @wyb1)
[OPERATOR] The kube-controller-manager VPA now has minAllowed values to prevent VPA from scaling it down too much. (#3057, @timebertt)
[OPERATOR] It is again allowed the seed taints seed.gardener.cloud/disable-capacity-reservation, seed.gardener.cloud/disable-dns and seed.gardener.cloud/invisible to be used. Note, these taints have been replaced by seed.spec.settings fields and there is no special semantic behind them anymore. (#2970, @vpnachev)
[OPERATOR] Migrating a Shoot from a Seed which has DNS disabled to a Seed which has DNS enabled will now generate a default domain name for the Shoot, if the Shoot's DNS section is not marked as unmanaged. (#2969, @plkokanov)
[OPERATOR] Attempting to migrating a Shoot with spec.dns != nil to a Seed which has DNS disabled is now forbidden and will return an error. (#2969, @plkokanov)
[DEVELOPER] github.com/gardener/gardener/extensions/pkg/predicate.Or (which was deprecated in favor of sigs.k8s.io/controller-runtime/pkg/predicate.Or) is now removed. (#3111, @ialidzhikov)

[gardener-resource-manager]

Action Required

[DEVELOPER] New api module is now available. Get it with go get github.com/gardener/gardener-resource-manager/api (gardener/gardener-resource-manager#86, @mvladev)

Most notable changes

[OPERATOR] The ManagedResource's .status.conditions[].lastUpdateTime is no longer continuously updated. This will greatly reduce the number of update calls to the kube-apiserver/etcd. (gardener/gardener-resource-manager#90, @rfranzke)
[OPERATOR] The gardener-resource-manager does now feature a /healthz endpoint that can be used as part of a liveness probe configuration. It listens on port 8081 by default (configurable via the --health-bind-address flag). (gardener/gardener-resource-manager#81, @rfranzke)
[DEVELOPER] The following Golang dependencies have been revendored: (gardener/gardener-resource-manager#80, @rfranzke)
- github.com/gardener/gardener (v1.9.0 -> v1.11.3)
- github.com/gardener/hvpa-controller (v0.2.5 -> v0.3.1)
- github.com/onsi/ginkgo (v1.12.1 -> v1.14.0)
- k8s.io/* (v0.16.8 -> v0.18.10)
- sigs.k8s.io/controller-runtime (v0.5.5 -> v0.6.3)
[DEVELOPER] The Golang version has been updated to v1.15.3. (gardener/gardener-resource-manager#80, @rfranzke)
[DEVELOPER] The base image for the Docker image has been updated to alpine v3.12.1. (gardener/gardener-resource-manager#80, @rfranzke)

Improvements

[OPERATOR] gardener-resource-manager is now using PATCH (instead of UPDATE) to add/remove a finalizer. (gardener/gardener-resource-manager#87, @ialidzhikov)
[OPERATOR] An issue has been fixed which caused failing health checks for CustomResourceDefinitions of version v1. (gardener/gardener-resource-manager#85, @timuthy)

Docker Images

gardener-apiserver: eu.gcr.io/gardener-project/gardener/apiserver:v1.13.0
gardener-controller-manager: eu.gcr.io/gardener-project/gardener/controller-manager:v1.13.0
gardener-scheduler: eu.gcr.io/gardener-project/gardener/scheduler:v1.13.0
gardener-admission-controller: eu.gcr.io/gardener-project/gardener/admission-controller:v1.13.0
gardener-seed-admission-controller: eu.gcr.io/gardener-project/gardener/seed-admission-controller:v1.13.0
gardenlet: eu.gcr.io/gardener-project/gardener/gardenlet:v1.13.0

gardener/gardener v1.13.0 on GitHub

[gardener]

Action Required

Most notable changes

Improvements

[gardener-resource-manager]

Action Required

Most notable changes

Improvements

Docker Images

gardener/gardener v1.13.0
on GitHub