Release notes for kOps 1.21 series

⚠ kOps 1.21 has not been released yet! ⚠

This is a document to gather the release notes prior to the release.

Significant changes

Service Account Issuer Discovery and AWS IAM Roles for Service Accounts (IRSA)

kOps now supports publishing an OIDC-compatible discovery document to an S3 bucket and configuring AWS to use it for IAM Roles for Service Accounts (IRSA).

See the Service Account Issuer Discovery documentation for more information.

Dedicated API Server nodes.

kOps now supports extending the control plane with dedicated apiserver nodes. These nodes run in dedicated instance groups that can be scaled horizontally.

In 1.21, this feature is behind a feature flag as node role name, labels, taints, and domains can change based on feedback from the community.

Warm Pool (AWS only)

A Warm Pool contains pre-initialized EC2 instances that can join the cluster significantly faster than regular instances. These instances run the kOps configuration process, pull known Docker images, and then shut down. When the ASG needs to scale out it will pull instances from the warm pool if any are available.

See the warm pool documentation for more information.

Other significant changes

• Protokube now runs as a systemd process rather than a docker container.

• Support for AWS launch configurations has been removed in favour of launch templates.

Breaking changes

• Support for Kubernetes versions 1.13 and 1.14 has been removed.

Required Actions

• To support Node Termination Handler's Queue Process mode, AWS cluster deletion now requires the kops CLI have sqs:ListQueues and events:ListRules permissions regardless of whether or not the addon is used.

Deprecations

• Support for Kubernetes versions 1.15 and 1.16 are deprecated and will be removed in kOps 1.22.

• The manifest based metrics server addon has been deprecated in favour of a configurable addon.

• The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.

• The node-role.kubernetes.io/master and kubernetes.io/role labels are deprecated and will be removed from control plane nodes in kOps 1.22

• Due to lack of maintainers, the Aliyun/Alibaba Cloud support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this cloud provider.

Full change list since 1.21.0-alpha.3 release

• fix a typo @yojay11717 #11232
• Release notes for 1.21.0-alpha.3 @hakman #11233
• Remove validations for EBS from cluster validation @h3poteto #11228
• Add support for Docker v20.10.6 @hakman #11236
• Don't start kubelet if instance is entering the warm pool @olemarkus #11216
• Correct typos @Akiros001 #11238
• Logging cleanup @rifelpet #11080
• Update kops_create_secret_dockerconfig.md @integrii,@hakman #11186
• Remove BLM banner @hakman #10672
• Run tests only in zones with increased limits @hakman #11240
• Give kOps CLI knowledge about ASG warm pools @olemarkus #11227
• Fix golint issue caused by typo @fenggw-fnst #11239
• Remove unused constants @johngmyers #11241
• Bump k8s versions with April 2021 releases in Alpha channel @MoShitrit #11245
• Update kOps recommended versions and images @hakman #11247
• Kubetest2 - Cleanup leaked resources from previous clusters @rifelpet #11250
• Run tests in all regions with increased limits @hakman #11249
• Don't set NeedUpdate on first addon install @olemarkus #11257
• Make it possible to detect field changes when mixedInstancePolicy is removed @h3poteto #11255
• Update rolling update documentation @johngmyers #11263
• Pre-pull cilium and kube-proxy in warming mode @olemarkus #11258
• [cilium] Add support for choosing resources @dntosas #11248
• Add install section to kubelet unit @olemarkus #11264
• Update terraform and cloudformation lint versions @rifelpet #11266
• Fix cilium template scoping typo @javipolo #11270
• Add Azure image to alpha/stable channel @kenji-cloudnatix #11271
• Exclude nodes from load balancers upon cordoning @johngmyers #11273
• Make it possible to enable/configure warm pool @olemarkus #11235
• If one tries to use eip with a public ip that doesn't exist, fail @olemarkus #11276
• Spotinst: Update spotinst/ocean-controller to v1.0.74 @liranp #11286
• Add NTH Queue Processor Mode @haugenj #10995
• Apiserver fixes @olemarkus #11293
• Spotinst: Prevent nil pointer dereference @liranp #11289
• fix: create.go doesnt add --name flag to the prompt: kops update cluster @ebarped #11296
• Make warm pool no ASG found error retryable @olemarkus #11285
• Document the newly required SQS permissions for NTH @rifelpet #11300
• fix permissions required for NTH Queue Processor @haugenj #11303
• bump NTH to 1.13.0 @haugenj #11301
• Add GCE Router task @kenji-cloudnatix #11184
• Add ability to set a default Issuer in certManager addon @javipolo #11281
• Make nodeup able to complete the warming life cycle hook @olemarkus #11259
• update deps @zetaab #11306
• Filter servers using cluster name in tags @zetaab #11305
• Add warm pool docs and release notes @olemarkus #11307
• Use the full operator instead of the generic one @olemarkus #11312
• Improve warm pool documentation @johngmyers #11313
• Disallow negative warmpool sizes @johngmyers #11317
• Promote channel alpha to stable @hakman #11318
• [metrics-server] Bump manifest to latest stable @dntosas #11319
• Allow disabling warm pool by setting WarmPool.MaxSize to 0 @johngmyers #11316
• Fix typo @johngmyers #11321
• [csi/aws] Bump templates + add support for warm pools @dntosas #11304
• Add a lifecycle test for GCE @kenji-cloudnatix #11291
• Add cluster-level warmPool settings @johngmyers #11322
• Fix arguments to csi-provisioner after bump to v2.2.0 @codablock #11326
• kubetest2: Infer the provider and zones from the kops cluster @justinsb,@rifelpet #10847
• Add support for configuring Cilium enable-host-reachable-services. @bjhaid #11333
• Fix lifecycle hook naming @olemarkus #11335
• Recognize Ubuntu 21.04 @hakman #11327
• Add enable-host-reachable-services to 1.8 and generic cilium. @bjhaid #11337
• Don't try to delete warm pool when creating the cluster @olemarkus #11331
• Update Calico to v3.18.2 @hakman #11339
• Fix SQS resource flapping @olemarkus #11336
• Update controller-runtime to v0.9.0-beta.0 @hakman #11342
• Set SAN for addon CAs @olemarkus #11328
• Update kubetest2 dependency and fix install method for upgrade scenario @rifelpet #11338
• Bump cilium to 1.9.6 @olemarkus #11344
• Fix upgrade scenario kubetest2 install @rifelpet #11350
• Fix kubetest2 panic inheriting env vars @rifelpet #11351
• Mount /run inside etcd-manager pods for systemd mounts @hakman #11352
• Update deps @zetaab #11357
• Ignore detached nodes when doing validate cluster @rajatjindal #11349
• Move firewall, iam, network and sshkey to awsmodel @hakman #11358
• [addons/nth] Add capability to define resources @dntosas #11360
• Split oidc_provider @olemarkus #11359
• Expose hubble agent when hubble is enabled @olemarkus #11314
• Configure aws oidc provider @olemarkus #11361
• Use VFS as service account issuer if configured @olemarkus #11362
• Allow cert-manager to be provisioned externally @codablock #11354
• Mark control-plane node for update when etcd volume size changes @hakman #11365
• Mark control-plane node for update when etcd manager config changes @hakman #11369
• user-configurable IAM roles for ServiceAccounts @olemarkus #11016
• add permission to create sa tokens @zetaab #11373
• Add more support for cilium 1.10 @olemarkus #11374
• Update Calico to v3.19.0 @hakman #11372
• Refactor terraform writing @johngmyers #11371
• Remove unused k8s version parsing @rifelpet #11375
• Fix upgrade of service-account key @johngmyers #11376
• Don't try to mount hubble TLS on the agent if we don't use hubble @olemarkus #11378
• Kubetest2 - Update k8s upgrade test + add kops upgrade test @rifelpet #11382
• Kubetest2 - Fix GNU mktemp syntax @rifelpet #11384
• Kubetest2 - fix wget flag in kops download @rifelpet #11385
• kubetest2 - remove unnecessary flags from upgrade scripts @rifelpet #11386
• Don't use PublicJWKS in TestAWSLBController @johngmyers #11391
• Don't add IRSA env vars if feature flag is not enabled @olemarkus #11392
• Recognize the ServiceAccountIssuerDiscovery featue gate @johngmyers #11395
• Quote grep patterns in docs/rotate-secrets.md @keithlayne #10656
• Documentation and release note for IRSA @johngmyers #11398
• Remove the PublicJWKS feature flag @johngmyers #11396
• Don't publish OIDC discovery if DiscoveryStore not set @johngmyers #11397
• Add elasticloadbalancing:ModifyTargetGroupAttributes to aws lb controller @olemarkus #11393
• Add another update cluster dryrun to upgrade tests @rifelpet #11401
• Update default volumes types in Cluster Documentation @allir #11405