Release notes for kOps 1.22 series

⚠ kOps 1.22 has not been released yet! ⚠

This is a document to gather the release notes prior to the release.

Significant changes

Instance metadata service version 2

On AWS, kOps will enable Instance Metadata Service Version 2 and require tokens on new clusters with Kubernetes 1.22. In addition, the following max hop limits will be set by default:

• worker and API server Nodes, and bastions, will have a limit of 1 hop.
• control plane nodes will have a limit of 3 hops to accommodate for controller Pods without host networking that need to assume roles.

This will increase security by default, but may break some types of workloads. In order to revert to old behavior, add the following to the InstanceGroup:

spec:
  instanceMetadata:
    httpTokens: optional

Other significant changes

• New clusters on AWS will no longer provision an SSH public key by default. To provision
  an SSH public key on a new cluster, use the --ssh-public-key flag to kops create cluster.

• The kOps Terraform support now renders managed files through the Terraform configuration instead
  of writing them to S3 directly. This defers changes to these files until the time of terraform apply.
  This feature may be temporarily disabled by turning off the TerraformManagedFiles feature flag
  using export KOPS_FEATURE_FLAGS="-TerraformManagedFiles".

• kOps now implements graceful rotation of its Certificate Authorities and the service
  account signing key. See the documentation on How to rotate all secrets / credentials

• New clusters running Kubernetes 1.22 will have AWS EBS CSI driver enabled by default.

Breaking changes

• Support for Kubernetes versions 1.15 and 1.16 has been removed.

• The legacy location for downloads s3://https://kubeupv2.s3.amazonaws.com/kops/ has been deprecated and will not be used for new releases. The new canonical downloads location is https://artifacts.k8s.io/binaries/kops/.

• The assets phase of kops update cluster has been removed. It is replaced by the new kops get assets --copy command.

• Support for importing and converting kubeup clusters has been removed.

Required actions

• The kOps Terraform support now renders managed files through the Terraform configuration instead
  of writing them to S3 directly. If, after upgrading kOps and applying a new Terraform plan,
  you subsequently downgrade to an earlier version of kOps, the generated plan will delete these
  files, breaking the cluster. Prior to applying the plan, you will need to orphan all the
  aws_s3_bucket_object objects the plan wants to destroy. Use terraform state rm on each of them.
  Then re-run terraform plan until there are no such objects in the plan.

  If you applied the plan without first orphaning all of these objects, fix the cluster by re-running
  kops update cluster --target terraform.

Deprecations

• Support for Kubernetes version 1.17 is deprecated and will be removed in kOps 1.23.

• Support for Kubernetes version 1.18 is deprecated and will be removed in kOps 1.24.

• Support for the Lyft CNI is deprecated and will be removed in kOps 1.23.

• Support for CentOS 7 is deprecated and will be removed in future versions of kOps.

• Support for CentOS 8 is deprecated and will be removed in future versions of kOps.

• Support for Debian 9 (Stretch) is deprecated and will be removed in future versions of kOps.

• Support for RHEL 7 is deprecated and will be removed in future versions of kOps.

• Support for Ubuntu 18.04 (Bionic) is deprecated and will be removed in future versions of kOps.

• The manifest based metrics server addon has been deprecated in favour of a configurable addon.

• The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.

• The node-role.kubernetes.io/master and kubernetes.io/role labels are deprecated and might be removed from control plane nodes in kOps 1.23.

• Due to lack of maintainers, the Aliyun/Alibaba Cloud support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this cloud provider.

• Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.

Other changes of note

• It is no longer necessary to set AWS_SDK_LOAD_CONFIG=1 in the environment when using AWS assumed roles with the kops CLI.

• There is a new command kops get assets for listing image and file assets used by a cluster.
  It also includes a --copy flag to copy the assets to local repositories.
  See the documentation on Using local asset repositories for more information.

Full change list since 1.22.0-alpha.2 release

• e2e upgrade-ab: fix a few errors @justinsb #11409
• Verify all versions are set correctly @johngmyers #11413
• Use etcd-manager built from etcdadm repo @justinsb,@hakman #11098
• Remove code for no-longer-supported k8s versions @johngmyers #11412
• Update the release process documentation @johngmyers #11419
• [addons/awscsidriver] Bump to GA release @dntosas #11418
• [Digital Ocean] Add an e2e job for DO @srikiz #10963
• Fix references to v1.20 in v1.21 release notes @hakman #11427
• Release notes for 1.21.0-beta.1 @johngmyers #11426
• e2e: only get ExternalIPRange if we need it @justinsb #11431
• e2e upgrade-ab: a few more fixes and notes on how to run locally @justinsb #11432
• Create new clusters without forcing a container runtime @hakman #11428
• Update verify-terraform to use 0.15.3 @rifelpet #11433
• Carry forward 1.20 deprecations to 1.21 release notes @johngmyers #11438
• Start release notes for 1.22 @johngmyers #11439
• Sort --extra-tags of ebs-csi-driver @codablock #11444
• Fix typo in 1.22 release notes @johngmyers #11448
• Add test scenario for aws ebs csi driver @olemarkus #11449
• Always install the latest plugin versions for Terraform tests @hakman #11447
• Set the output base for fitask @hakman #11411
• Simplify buildLaunchTemplateTask() part one @johngmyers #11452
• Add missing carryover items from 1.21 release notes @johngmyers #11451
• Add support for CAS 1.21.0 @olemarkus #11462
• Allow AWS instance types with multiple architectures @hakman #11463
• Fix KCM livenessProbe to use secure port @rifelpet #11454
• Simplify buildLaunchTemplateTask() part two @johngmyers #11461
• Use kubernetes.default for OIDC discovery in gossip clusters @rifelpet #11470
• Add instructions for updating the k8s versions periodic jobs @rifelpet #11473
• Release notes for 1.20.1 @justinsb #11475
• Release notes for 1.19.3 @justinsb #11474
• Update alpha channel with K8s releases from May-12 2021 @MoShitrit #11476
• upup: gcetasks: fix diffs in instance template and router @nicktrav #11460
• Discover what zone the cluster is in for the aws-ebs-csi driver tests @olemarkus #11472
• Use ginkgo to run the tests so we can run things in parallel @olemarkus #11479
• Kubetest2 - Increase validation time for DO jobs @rifelpet #11481
• upup: gcetasks: force send AutoCreateSubnetworks field when set to false @nicktrav #11457
• Add kOps and k8s 1.21 to alpha channel @MoShitrit #11482
• Reduce kOps supported version range @johngmyers #11485
• More release process documentation improvements @johngmyers #11434
• Set the test cluster-tag @olemarkus #11487
• Set canonical location for downloads to artifacts.k8s.io @hakman #11486
• [AWS CCM] Permission to create SA token @nckturner #11368
• Add link to release notes on first beta release @johngmyers #11488
• Remove etcd-manager certificate expiration advisory @hakman #11480
• Adjust deprecation announcements @johngmyers #11489
• Update cert-manager @olemarkus #11493
• Set priorityClassName on critical addons @olemarkus #11495
• fix(coredns/rbac): add permission to list and watch endpointslices @nettoclaudio #11459
• bump aws lb controller to 2.2.0 @olemarkus #11502
• Aws lb scenario fix flags @olemarkus #11506
• AWS LB controller requires multiple subnets to work @olemarkus #11507
• Cleanup some of the scenario scripts @rifelpet #11508
• Include new pipeline job in the release branch process @rifelpet #11509
• Spotinst: Update spotinst/ocean-controller to v1.0.75 @liranp #11512
• Subsume StatusStore into fi.Cloud @johngmyers #11498
• Split genkgo in two @olemarkus #11519
• [DigitalOcean] [WIP] Increase droplet size for e2e tests @srikiz #11520
• Add initial support for configuring IPv6 with AWS @hakman #11442
• Add default tags to LB controller and cilium eni resources @olemarkus #11517
• Remove dead code in bootstrap script @johngmyers #11521
• Set default fstype for ebs volumes to ext4 @olemarkus #11525
• Skip feature tests for ebs csi e2e @olemarkus #11530
• Update etcd_backup_restore_encryption.md @aberenshtein #11533
• Don't download nodeup if already in the AMI @johngmyers #11524
• [addons/networking.cilium.io] enable prometheus scraping @ulfox #11514
• feat(openstack): enable configuration of servergroup affinities @mitch000001 #11531
• Update containerd to v1.4.6 @hakman #11535
• Cleanup orphaned IAM service account roles in direct render @johngmyers #11497
• Support terraform 0.12+'s filebase64() in json output @rifelpet #11540
• Release images bundle instead of separate images @hakman #11522
• Bump CoreDNS manifests to latest stable version 1.8.3 @dntosas #11500
• Run the tests requiring snapshotcontroller again @olemarkus #11544
• Update CAS manifest @olemarkus #11491
• Make events etcd cluster optional @codablock #11330
• Add support for arbitrary terraform functions @rifelpet #11542
• Add snapshot-controller @olemarkus #10730
• Add etcd-server related tests @hakman #11552
• Bump default cilium to 1.9.7 @olemarkus #11554
• Document updating conformance is first stable minor release only @johngmyers #11556
• Add hubble documentation @olemarkus #11557
• Allow using insecure TLS for metrics-server with Kubernetes 1.19+ @hakman #11559
• Add snapshot-controller @olemarkus #11561
• Fix deletion of IAM roles and policies @johngmyers #11558
• Allow Spotinst to use comma separated instance types @hakman #11560
• Release notes for 1.21.0-beta.2 @johngmyers #11570
• Set flags on AWS CCM mimicking KCM @olemarkus #11566
• Enable cert-manager in the ebs csi e2e test @olemarkus #11569
• Only allow deletion of snapshots owned by the cluster @olemarkus #11571
• Avoid error when first creating VPC with IPv6 @justinsb #11575
• Improve some small issues with the release process @hakman #11572
• Cleanup InstanceProfile only that have ownership tags in delete cluster @h3poteto #11568
• Don't set the master address for aws ccm @olemarkus #11582
• Enable reading shared config when possibly from CLI @johngmyers #11387
• Only update kubeconfig user when we have user info @justinsb #11584
• Add release note for AWS shared config @johngmyers #11585
• Use latest CI build instead of building in the test @olemarkus #11588
• Remove unused files @johngmyers #11591
• Use the downloaded kops version for awslbc test @olemarkus #11593
• Use the OnDelete updateStrategy for AWS VPC CNI DaemonSet @johngmyers #11590
• Update Calico to v3.19.1 @hakman #11594
• Cleanup Docs @hakman #11595
• First addon operator integration: CoreDNS @justinsb #9374
• Add a note about NTH Queue Process mode @olemarkus #11600
• Enable AWS EBS CSI driver by default @olemarkus #11605
• Add documentation about snapshot-controller @olemarkus #11606
• Convert all indents to spaces in node bootstrap script @hakman #11611
• Use version marker for kops upgrade scenario @olemarkus #11612
• Add init image field for Amazon VPC CNI @ryan-dyer #11602
• Add to release process documentation @johngmyers #11581
• Change toolbox template flag for consistency @johngmyers #11616
• Fix duplicate CopyFile tasks @johngmyers #11619
• Don't stage kops as file assets @johngmyers #11620
• Dump all CP node logs to artifacts @olemarkus #11615
• Simplify release steps @johngmyers #11624
• Remove debug code and copy kops to PATH @olemarkus #11625
• Update Go to v1.16.4 @hakman #11626
• Add "kops get assets" command @johngmyers #11617
• Set lifecycle on WarmPool task @johngmyers #11618
• Label issue types in issue templates @johngmyers #11637
• Remove fallback support for legacy IAM @johngmyers #11641
• Rename CopyDockerImage to CopyImage @johngmyers #11640
• Update the service account issuer discovery documentation @olemarkus #11642
• Require all HasLifecycle tasks to have lifecycle set @johngmyers #11650
• Consolidate CSI livenessprobe images for multi-arch support @rifelpet #11652
• Protokube needs dns-controller IAM permissions @johngmyers #11645
• Remove docs on static addons @olemarkus #11653
• Skip some steps if not doing cluster lifecycle @johngmyers #11657
• Fix detection of virtual-hosted-style S3 urls in us-east-1 @johngmyers #11655
• Promote channel alpha to stable @johngmyers #11658
• Bump default cilium to 1.10 @olemarkus #11659
• [Digital Ocean] Code cleanup with no functional modifications @srikiz #11592
• Fix jwks object path in S3 for IRSA @h3poteto #11649
• Use version marker for kops ab scenario @olemarkus #11648
• Kubetest2 scenario script cleanup @rifelpet #11664
• Add more lifecycles to HasLifecycle tasks @rifelpet #11666
• Set lifecycle on Droplet task @johngmyers #11665
• Don't describe CloudLabels as being AWS-specific @johngmyers #11667
• Move common stuff in e2e scenarios to common.sh @olemarkus #11668
• Fix kubetest2 upgrade scripts @rifelpet #11670
• Clean up straggling autogenerated code @johngmyers #11671
• Remove dead code @johngmyers #11672
• Explicitly set kubeconfig flag where we want to use it @olemarkus #11676
• Add support for Docker v20.10.7 @hakman #11674
• Use release markers instead of releases @olemarkus #11679
• Drop trailing slash from oidc issuer @olemarkus #11682
• Make Lifecycle field non-pointer @johngmyers #11673
• Update Go to v1.16.5 @hakman #11686
• Fix set-version leaving backup files with "-e" suffix @johngmyers #11691
• Release notes for 1.21.0-beta.3 @johngmyers #11694
• Update release process documentation @johngmyers #11695
• Set IMDSv2 on by default for nodes and apiservers @olemarkus #11329
• Deprecate old OS versions @johngmyers #11696
• Fix panic in dryrun report @johngmyers #11698
• Add options for configuring IPv4 and IPv6 support with Calico @hakman #11688
• add e2e scenario script for testing cilium connectivity @olemarkus #11697
• Fix copying of images from docker.io @johngmyers #11656
• Fix the CSI EBS DS CRB. @olemarkus #11701
• Use v1 certificate for LB controller @olemarkus #11703
• Move asset copying out of apply_cluster @johngmyers #11700
• Remove documentation of legacy IAM permissions @johngmyers #11706
• Add some tests around channel adding needs-update annotation @olemarkus #11598
• Update kube-router to v1.2.3 @hakman #11124
• Create document on asset repositories @johngmyers #11654
• Make relnotes match the new max hop limit IMDS behaviour @olemarkus #11702
• Add proxy envs to calico to make possible usage of AWS source destination check @DOboznyi #11709
• Update controller-runtime to v0.9.0 @hakman #11713
• Generate AWSEBSCSIDriver model only when using AWS @hakman #11716
• Make AWS EBS CSI Driver default as of k8s 1.22 @olemarkus #11721
• Use quay images for cilium @olemarkus #11722
• Allow master to touch volumes tagged with kubernetes.io/cluster/:owned @wongma7 #11729
• Update release branch docs with kubetest2 presubmit job @rifelpet #11732
• Perform ClusterCIDR and ServiceClusterIPRange assignments for IPv6 @johngmyers #11724
• Spotinst: Support for API Load Balancer with AWS/NLB @liranp #11604
• Add support for setting latest k8s in ab scenario @olemarkus #11735
• Deprecate CloudFormation support @johngmyers #11630
• Calculate IPv6 subnet CIDR based on cluster CIDR @hakman #11523
• Only warm-pull images used by the CSI DS @olemarkus #11734
• Remove k8s-upgrade script as upgrade-ab is now used instead @olemarkus #11738
• Add small note about rotating cluster after backup restore @olemarkus #11733
• Make forwardToKubeDNS work in the NodeLocal DNSCache template @ederst #11743
• Add test scenario for if channels is able to delete dangling resources @olemarkus #11739
• Remove InstanceGroup from NodeupModelContext @johngmyers #9294
• Refactor keypair code in preparation for secret rotation @johngmyers #11219
• Remove unused field @johngmyers #11749
• Hyperlink Sprig reference @OutdatedVersion #11730
• Compare OpenStack security groups deterministically @ederst #11741
• Don't set Subnet dependency on AmazonIPv6CIDR for shared VPCs @hakman #11752
• Set BindAddress appropriately when in IPv6-only mode @johngmyers #11737
• Add --ipv6 experimental cli flag @hakman #11629
• Don't restrict nodeup download to IPv4 @johngmyers #11755
• Cilium: disable masquerade by default when in ENI IPAM mode @johngmyers #11753
• Set default ClusterCIDR through the PodCIDR @johngmyers #11756
• Enable IPv6 support for Cilium @johngmyers #11754
• Allow unsetting fields from the command line @johngmyers #11745
• Adjustments to SpecOverride @johngmyers #11761
• Make the AdminAccess default inclusive of IPv6 @johngmyers #11763
• Default the NodeCIDRMaskSize appropriately for IPv6 @johngmyers #11762
• Simplify Calico IPv6 configuration @johngmyers #11725
• Fix typo in IRSA docs @yurrriq #11770
• Fix typo in populate_instancegroup_spec.go @yurrriq #11769
• fix enable default SC when EBS driver is not installed @olemarkus #11771
• Set containerd config on nodeup.Config instead of clusterspec @olemarkus #11750
• Make it easy to run scenarios with irsa enabled @olemarkus #11758
• Trim unnecessary paths from worker node IAM @johngmyers #11775
• Allocate smaller IPv6 PodCIDRs by default @johngmyers #11772
• Update github.com/spf13/viper to v1.8.0 @hakman #11777
• [cni/cilium] Add support for additional config options @dntosas #11678
• Bump the cas addon version. @olemarkus #11780
• Also set haveUserInfo=true in case --user was provided in "kops export kubecfg" @codablock #11778
• Don't try to build etcd-manager secrets for cilium twice @olemarkus #11764
• [addons] Introduce NodeProblemDetector @dntosas #11381
• Enable ability to use IRSA for cluster autoscaler @olemarkus #11748
• Allow using IRSA for EBS CSI Driver @olemarkus #11747
• Delete all files in the provided discoveryStore on cluster deletion @olemarkus #11791
• Release notes for 1.20.2 @justinsb #11804
• Update alpha channel k8s versions and ec2 ami base image @MoShitrit #11803
• Seed the random number generator on AWS @johngmyers #11789
• Upgrade AWS CNI to latest release 1.8.0 @MoShitrit #11805
• bump the version of gophercloud @cardoe #11788
• Allow "kops create keypair" to stage next CA cert @johngmyers #11252
• Reduce policy size @olemarkus #11814
• Fix lbc permissions @olemarkus #11815
• doc: remove brew switch ref and simplify version bump @chenrui333 #11817
• brew: remove kops.rb @chenrui333 #11819
• Split out get, describe, and delete keypairs commands @johngmyers #11820
• Include multiple cluster CAs in trust stores @johngmyers #11809
• Fix validating presence of AWS EBS CSI @olemarkus #11795
• Pre-pull all container images used by components and addons @hakman #11717
• skip flaking ebs csi flakes @olemarkus #11821
• Set EnableExternalCloudController to true by default @hakman #11825
• Put versioned API of cluster into state store @johngmyers #9229
• Support creating new service-account keypairs @johngmyers #11822
• Add support for logging-format option (text/json) @dntosas #11583
• Add back createvolume to master + bump ebs driver @olemarkus #11811
• Improve the output of 'kops get keypairs' @johngmyers #11823
• Fix kOps version for managed flag on cert-manager @djablonski-moia #11828
• Run scenarios as presubmit tests @olemarkus #11801
• Include multiple CA certs in exported kubeconfigs @johngmyers #11831
• Remove support for importing and converting kubeup clusters @johngmyers #11824
• Ignore failing tests in upgrade scenario @rifelpet #11832
• Set priority class for AWS CCM addon @hakman #11834
• Limit concurrency of asset copy tasks @johngmyers #11708
• Add 'kops promote keypair' command @johngmyers #11835
• Kubetest2 fix periodic end to end tests @olemarkus #11838
• Kubetest2 - Add --skip-regex logic @rifelpet #11841
• Fix skip regex for ebs csi test @olemarkus #11840
• Mark nodes NeedsUpdate when keys they use change @johngmyers #11833
• Completely remove EnableExternalCloudController feature flag @hakman #11839
• Only set default --skip-regex if it hasn't been set @rifelpet #11842
• Clarify the limitations of Azure DNS support @kenji-cloudnatix #11844
• Refactor kube-controller-manager secrets @johngmyers #11847
• Escape --skip-regex pattern @rifelpet #11851
• Make aws-cni config more flexible and generalized @MoShitrit #11816
• Weaken some interfaces @johngmyers #11837
• Handle containerExec hooks when using containerd @hakman #11852
• Improve image copying @johngmyers #11854
• Update helm to v3.6.1 @olemarkus #11860
• Update CNI plugins to v0.9.1 @hakman #11846
• Don't include irrelevant bootstrap addons @johngmyers #11861
• Remove obsolete Spotinst manifest @johngmyers #11862
• Enable cross-subnet mode with Calico by default @hakman #11810
• Fix dryrun cluster creation @johngmyers #11863
• Push alpha channel to stable @MoShitrit #11864
• Add a note about running update-expected when updating base AMI @MoShitrit #11865
• Make it simpler to spot missing files in integration tests @olemarkus #11866
• fix: broken link @choeffer #11793
• Decrease default values for net.ipv4.tcp_rmem and net.ipv4.tcp_wmem @hakman #11868
• Remove version from addons @hakman #11867
• Move most nodeup.Config data to config store @johngmyers #11869
• Don't reconcile roles and policies if a profile is provided @olemarkus #11836
• Use DualStack API NLB for IPv6 @hakman #11870
• Simplify config server protocol @johngmyers #11871
• Refactor etcd-client-cilium secrets @johngmyers #11848
• Retain deleted keypairs @johngmyers #11845
• Write config as ManagedFile @johngmyers #11796
• Improve "kops distrust keypair" command @johngmyers #11876
• Avoid spurious changes for ASG InstanceProtection and LT InstanceMonitoring @hakman #11873
• Kubetest2 - set node-os-arch flag instead of skipping kubectl test on arm64 @rifelpet #11879
• Improve completion for kops root command @johngmyers #11880
• Spotinst: Update spotinst/ocean-controller to v1.0.76 @liranp #11885
• support large/slow downloads @aojea #11884
• Add support for darwin/arm64 on the client-side @hakman #11883
• Refactor nodeup APIServer builder, part one @johngmyers #11872
• Allow rotation of etcd-clients-ca-cilium @johngmyers #11877
• [DigitalOcean] Increase droplet size for e2e tests @srikiz #11887
• Set download timeout to 3 minutes @hakman #11886
• Implement completion for "kops create keypair" @johngmyers #11888
• Render managed files with Terraform @johngmyers #9621
• Implement completion for "kops promote keypair" @johngmyers #11892
• Fix nil-pointer dereference on dryrun @johngmyers #11894
• Implement completion for "kops distrust keypair" @johngmyers #11899
• Refactor etcd-clients-ca keyset for api-server @johngmyers #11897
• Allow overriding the ServiceAccountIssuer for IRSA @johngmyers #11853
• Remove unnecessary parameters from terraform finish methods @rifelpet #11900
• Include GCP Project in terraform HCL2 output @rifelpet #11901
• Use Cobra's built-in completion command @johngmyers #11905
• Refactor apiserver-aggregator-ca @johngmyers #11906
• Add support for IPv6 addresses to dns-controller @hakman #11907
• Improve "kops get keypairs" @johngmyers #11904
• Release notes for 1.21.0 @justinsb #11910
• Update pause image to 3.5 @rifelpet #11909
• Upgrade Cobra to 1.2.1 @johngmyers #11912
• Capture logs from the containerd service @hakman #11914
• Do not set both CIDR and IPv6CIDR on sg rules @olemarkus #11915
• Remove unused test files from legacy IAM @rifelpet #11918
• Reduce policy size further @olemarkus #11843
• Set KOPS_RUN_TOO_NEW_VERSION in scenario scripts @rifelpet #11923
• Update version support matrix for 1.21 @johngmyers #11922
• Rename the "ca" keyset to "kubernetes-ca" @johngmyers #11921
• Allow fsstore to be used for mock s3 rules @olemarkus #11916
• Implement completion for "kops rolling-update cluster" @johngmyers #11924
• Implement completion for "kops update cluster" @johngmyers #11926
• Update the status of cloud providers @johngmyers #11930
• Remove obsolete files @johngmyers #11932
• Implement completion for validate and upgrade @johngmyers #11927
• Continue if a single addon fails to be applied @olemarkus #11933
• Remove unused golden files from manyaddons test @olemarkus #11935
• Schedule certmanager webhook on control plane @olemarkus #11934
• [Digital Ocean] Remove PrivateNetworking option in droplet since it's deprecated @srikiz #11936
• Run cert-manager cainjector on CP nodes as well @olemarkus #11938
• Fix various CCM issues @olemarkus #11939
• Add podPidsLimit / --pod-max-pids support @uthark #11898
• Add log rotation for etcd-cilium.log @hakman #11943
• [Digital Ocean] Modify error message when multiple zones are specified @srikiz #11944
• Fix bullet point rendering in state doc @rothgar #11948
• Implement some completion for "kops create cluster" @johngmyers #11940
• check if the instance is under an asg @olivierpilotte #11958
• Use etcd v3.5.0 for Kubernetes 1.22+ @hakman #11941
• Unconditionally reenable KMS and Volume Limit tests @rifelpet #11966
• Suppress usage for errors returned from RunE @johngmyers #11969
• Implement completion for "kops create instancegroup" @johngmyers #11957
• Refactor keysets for etcd-manager @johngmyers #11964
• Cilium etcd fixes @olemarkus #11961
• Refactor service-account signing key @johngmyers #11974
• Add "all" variants of key rotation commands @johngmyers #11971
• Add documentation for keypair rotation @johngmyers #11972
• Implement completion for delete commands @johngmyers #11970
• Issue certs using CA KeypairID in NodeupConfig @johngmyers #11975
• Stop writing the certificate-only keyset.yaml @johngmyers #11977
• Provide more information on rotating secrets @johngmyers #11978
• Spotinst: Update spotinst/ocean-controller to v1.0.77 @liranp #11981
• [Digital Ocean] Fix sporadic volume detach error when volume is already detached @srikiz #11963
• Fix broken link to contributing @MoShitrit #11979
• Add "kops trust keypair" command @johngmyers #11973
• Implement completion for "kops edit" commands @johngmyers #11980
• Add missing IAM permissions to the NTH docs @olemarkus #11984
• Implement completion for "kops export kubeconfig" @johngmyers #11983
• Cobra cleanups @johngmyers #11985
• Fix "kops export kubeconfig" @johngmyers #11988
• Add region to aws lbc @olemarkus #11990
• Move containerd config from cloudup to nodeup @olemarkus #11986
• Change set and unset commands into flags on "kops edit cluster" @johngmyers #11987
• Remove dead code @johngmyers #11993
• Implement completion for "kops toolbox", part one @johngmyers #11992
• hack/upload: avoid ACLs for GCS buckets with UBLA enabled @spiffxp #11994
• Azure - support VMSS availability zones @rifelpet #11962
• Upgrade aws-sdk-go @rifelpet #11996
• remove references to kubernetes-release-dev @spiffxp #11997
• Clean up extra spaces @jayonlau #11989
• Verify CA keypair IDs for kops-controller-issued certs @johngmyers #11982
• Use keypair IDs for non-kops-controller-issued worker node certs @johngmyers #11998
• Update alpha channel with July k8s releases and bump Ubuntu EC2 AMI version @MoShitrit #12000
• Dedicated function for ccm permissons @olemarkus #11991
• Add keypair rotation test scenario @rifelpet #12001
• Fix file permissions on new keypair rotation test scenario @rifelpet #12005
• Implement completion for "kops toolbox", part two @johngmyers #11999
• Issue kubelet cert on apiserver nodes for k8s before 1.19 @johngmyers #12002
• Refactor more kube-apiserver credentials @johngmyers #12003
• Accommodate older destination kops versions in upgrade-ab scenario @johngmyers #12008
• Fix kops binary references in keypair rotation scenario @rifelpet #12009
• Don't provision SSH key by default on AWS @johngmyers #12011
• Deprecate the Lyft CNI @johngmyers #12010
• Remove apiserver's access to controller-manager secrets @johngmyers #12006