Release notes for kOps 1.20 series

(The kOps 1.20 release has not been released yet; this is a document to gather the notes prior to the release).

Significant changes

• Added experimental Azure support. To get started check the docs.

• Added template funtions for kubernetes version based on channel data.

• kOps now use helm3 functions for merging template --set and --values arguments. This has slightly different behaviour than previous helm2-like logic.

• Following kubeadm, control plane nodes are now labelled with node-role.kubernetes.io/control-plane=""

• Default node image for GCE changed from COS to Ubuntu for K8s versions >= 1.17.0. This is to more closely align with the AWS implementation (the most mature support) and because COS limits the ability to modify files on its disk.

Breaking changes

• Support for Kubernetes 1.11 and 1.12 has been removed.

• Support for Terraform version 0.11 has been removed.

• Support for the feature flag Terraform-0.12 has been removed. All generated Terraform HCL2/JSON files will support versions 0.12.26+ and 0.13.0+.

Required Actions

• If you are running kops toolbox template in an airgapped environment, you have to set --channel to point to a local channel file.

• If your workload targets control plane nodes, you need to change them to select the node-role.kubernetes.io/control-plane="" label. You should also add the node-role.kubernetes.io/control-plane:NoSchedule toleration to these workloads. This taint will not be added to control plane nodes before kOps 1.22.

Deprecations

• Support for Kubernetes versions 1.13 and 1.14 are deprecated and will be removed in kOps 1.21.

• The manifest based metrics server addon has been deprecated in favour of a configurable addon.

• The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.

• The node-role.kubernetes.io/master and kubernetes.io/role labels are deprecated and will be removed from control plane nodes in kOps 1.22

• The experimental node-authorizer that could be enabled using nodeAuthorization has been removed. Setting this value is now forbidden.

Partial change list since 1.19.0 release

1.19.0-beta.3 to 1.20.0-alpha.1

• Update docs for cutting new release branches @rifelpet #10084
• Update security_groups.md @yurrriq #10078
• Take node labels from cloud tags on AWS @johngmyers #9575
• Update Office Hours Zoom link @johngmyers #10087
• Update zoom links on the spanish README @rdrgmnzs #10088
• Ignore changes to ForAPIServer field @justinsb #10086
• Update Flannel CNI to v0.13.0 @hakman #10064
• kubetest2 - Implement create/validate/delete cluster functionality @rifelpet #10083
• Cert circular deps @olemarkus #10092
• Fix cilium template by specifying boolean as a string for enable-metrics @h3poteto #10094
• Release notes for 1.18.2 @justinsb #10097
• Update Kops Go build supported versions 1.15 @bmelbourne #10099
• Spotinst: Bump the Spot Cluster Controller to 1.0.68 @liranp #10103
• Remove hack/workaround from etcd-manager certificate expiration advisory @hakman #10102
• Install container runtime packages as assets @hakman #10048
• Default to exporting a kubecfg, even without credentials @justinsb #10105
• Remove dependency of TerraformJSON feature flag @johngmyers #10106
• Makefile and hack script cleanup @rifelpet #10112
• Update channels @hakman #10117
• Update Calico config for eBPF mode @hakman #10115
• Add random AWS zone logic + specify build stage location @rifelpet #10121
• Update AWS VPC CNI to 1.7.5 @MoShitrit #10124
• Add nodeLocalDNSCache.kubeDnsOnly option @javipolo #10111
• Align AWS VPC CNI manifest with upstream @hakman #10126
• Fix release notes links to point to https://kops.sigs.k8s @hakman #10118
• Add verify-cloudformation script @rifelpet #10130
• Fix cloudformation lint errors @rifelpet #10131
• Update shell style for CLI docs for better compatibility @hakman #10128
• Prevent unintended resource updates to LB attatchments @rdrgmnzs #9794
• Make verify-cloudformation job fail when issues are found @rifelpet #10133
• Set minimum Terraform version to 0.12.26/0.13.0 @bmelbourne #10109
• ELB/TargetGroup/ASG attachment fixes @rifelpet #10138
• Prepare for version 1.20 @johngmyers #10101
• Rebrand kops to kOps @hakman #10077
• Remove code for no-longer-supported k8s releases @johngmyers #10141
• allow reauth for openstack client @zetaab #10144
• Simplify etcd options builder @hakman #10145
• Update AWS Cloudmock for complex and externallb integration test clusters @rifelpet #10140
• Deprecate field calico.majorVersion @hakman #10143
• [Digital Ocean] Use Debian10 as default image @srikiz #10098
• Fix NLB naming for terraform and cloudformation targets @rifelpet #10158
• Move NLB's VPC CIDR security group rule logic into model @rifelpet #10161
• Fix additionalSecurityGroups support for NLB @rifelpet #10162
• Some typos @Hellcatlk #10160
• Fix output for CF and TF @hakman #10164
• Avoid waiting on validation during rolling update for inapplicable instance groups @bharath-123 #10065
• OpenStack Reset deviceID status if needed @zetaab #10178
• Remove unused bearer token field from kubeconfig builder @rifelpet #10181
• Compare KubernetesAPIAccess to OpenStack allowedCIDRs deterministically @havulv #10186
• Consistent naming of security group rules @olemarkus #10179
• Upgrade Hashicorp HCLv2 Go module v2.7.0 @bmelbourne #10189
• Fix auto scaling group changes when using spot instances @hakman #10187
• Upgrade sprig to v3 @olemarkus #10191
• Upgrade helm to 2.17 and use the helm.sh reference @olemarkus #10192
• Fix AWS NLB reconciliation @hakman #10199
• Fix disabling spot instances when using launch templates @hakman #10198
• Add ACM cert permalink @rifelpet #10156
• Setup a second NLB listener when an AWS ACM certificate is used @rifelpet,@hakman #10157
• Update Go to v1.15.4 @hakman #10209
• Upgrade docker client @olemarkus #10193
• Spotinst: Configure Resource Limits in Ocean Auto Scaler @liranp #10190
• Release notes 1.19.0-beta.1 @hakman #10213
• Use LaunchTemplate versions instead of timestamped LaunchTemplates @hakman #10151
• Update kOps version after 1.19.0-beta.1 release @hakman #10216
• Remove components from cluster validation @johngmyers #10214
• Allow to use custom csi plugin image and enable topology support @zetaab #10215
• Update validate cluster cli docs @johngmyers #10219
• Fix cluster autoscaler docs @djablonski-moia #10225
• Make etcd-manager log verbosity configurable @elblivion #10194
• Update k8s versions nov 2020 @MoShitrit #10227
• Update Ubuntu ami to latest version @MoShitrit #10195
• Fix various nits @hakman #10217
• Switch ARM64 CI to Graviton2 CPU @hakman #10230
• Update docs related to audit logging @hakman #10231
• Don't install the misc packages for k8s 1.20+ @johngmyers #10222
• Fix readme @karancode #10228
• Update kops as kOps and remove extra spaces from .md files @axpraka,@hakman #10235
• Add default runtime and runtimes fields in the docker config @bharath-123 #10238
• Fix cluster validation dependency on local kubeconfig @eddycharly #10221
• Associate instance group to pod validation failures in cluster validation. @bharath-123 #10237
• Add HPA Flags for horizontal-pod-autoscaler-initial-readiness-delay & horizontal-pod-autoscaler-cpu-initialization-period @JoelBCarter #10241
• Remove more code specific to unsupported etcd v2 @johngmyers #10245
• GCE: ignore (output-only) networkInterface.name @justinsb #10242
• Make it possible to use OnDelete update strategy on addon daemonset @olemarkus #10167
• Fix version of storage-aws addon manifest @johngmyers #10247
• Fix cloudformation lint job @rifelpet #10256
• Update etcd-manager to 3.0.20201117 @justinsb #10257
• Use separate domain for kops-controller bootstrap @johngmyers #10239
• Revert "Switch ARM64 CI to Graviton2 CPU" @hakman #10262
• Update Bazel rules for Go to v0.24.7 @hakman #10240
• Update k8s dependencies to 1.20.0-beta.2 @rifelpet #10266
• Push multi-arch images @hakman #10265
• alpha channel: update legacy images @justinsb #10269
• Fix multi-arch image pushing @hakman #10270
• Add sslPolicy for NLB to change listener's security policy @FrankYang0529 #9666
• Optimize Bazel builds by os and arch @hakman #10267
• Fix incorrect URLs in kops cluster documentation @bycEEE #10274
• Use etcd v3.4.13 for k8s v1.19+ @hakman #10277
• Parse TargetGroup names from ARNs @hakman #10276
• Add Go code-generator v0.20.0-beta.2 crypto hash @bmelbourne #10285
• Add ACM/NLB instructions to 1.19 release notes @rifelpet #10292
• Release notes for 1.19.0-beta.2 @hakman #10293
• Add more NLB release notes and documentation @rifelpet #10294
• Can check cert expiry using openssl @alok87,@hakman #10282
• [weave] Add support for default version override @dntosas,@hakman #10273
• Add support of Azure Blob storage to VFS @kenji-cloudnatix #10258
• Update kOps version after 1.19.0-beta.2 release @hakman #10295
• Remove support for using legacy ELB name @hakman #10296
• Remove dead code @hakman #10297
• Remove support for disabling manifest normalization @johngmyers #10298
• Upgrade cloud-provider-openstack to 1.19.2 @rifelpet #10303
• Fix a typo in an error message returned from buildAzureBlobPath @kenji-cloudnatix #10305
• Allow setting CPU limit and Mem request / limit for kube API server @rdrgmnzs #10275
• Optimize Bazel dev builds by arch @hakman #10309
• Update Calico to v3.17.0 @hakman #10310
• [Digital Ocean] Upgrade godo sdk to v1.54 @srikiz #10320
• Tolerate missing detached EC2 instances @hwoarang #10319
• Don't try to detach masters @olemarkus #10328
• Remove copyright notice from nodeup scripts to reduce the user-data size. @rdrgmnzs #10333
• Add docs for metrics server @olemarkus #10332
• Push alpha to stable @MoShitrit #10336
• Add paramaeters related to Taint based Evictions in kube-apiserver @h3poteto #10339
• Allow using gp3 for root volumes @olemarkus #10345
• Update containerd and Docker versions @hakman #10341
• Update aws-sdk-go to v1.36.0 @hakman #10347
• Bump aws-vpc-cni version to 1.7.6 @MoShitrit #10337
• Update etcd-manager to 3.0.20201202 @justinsb #10351
• Update DigitalOcean cloud-controller-manager to v0.1.30 @timoreimann #10352
• Add aws-cloud-controller-manager config to addons @nckturner #9704
• Allow attaching same external target group to multiple instance groups @hakman #10335
• Add fuzzer and OSS-fuzz build script @AdamKorcz #10326
• Set --service-account-issuer for k8s 1.20+ @johngmyers #10284
• Promote addon docs to first level menu item @olemarkus #10355
• [Digital Ocean] Promote to Beta @srikiz #10312
• Give users the option to gzip and base64 encode the heredocs in the nodeup.sh user-data @rdrgmnzs #10357
• Add integration test for creating an HA cluster in shared zone @hakman #10365
• Add minimal cert-manager addon @olemarkus #10318
• Add option to reuse existing Elastic IPs for NAT gateways @hakman #10374
• Remove resource limits from cluster autoscaler @olemarkus #10375
• Remove dependency on TravisCI @hakman #10366
• fix cluster-autoscaler README url from cluster_spec -> addons @isaachui #10373
• Rename duplicate ci target to quick-ci @hakman #10378
• Use custom-configured ServiceAccountIssuer when present @johngmyers #10364
• Add option for setting the volume encryption key in AWS @hakman #10359
• Add support for AWS IMDS v2 @bharath-123 #10324
• Update k8s dependencies to v1.20.0 @hakman #10390
• Update docs for CentOS 8 @hakman #10368
• Move tools into separate hack go module @rifelpet #10308
• Update etcd-manager to 20201209 @justinsb #10394
• Mount /lib64 for Protokube only on AMD64 @hakman #10396
• Explicitly specify http_endpoint in terraform launch template @bharath-123 #10398
• Update alpha channel with December 2020 k8s releases and bump Ubuntu AMI version @MoShitrit #10401
• Hack script improvements @rifelpet #10407
• hack/goimports - Replace mapfile with read @rifelpet #10410
• Allow override of registry and tag for Calico images @hakman #10316
• Update Calico to v3.17.1 @hakman #10408
• Bump aws-cni to 1.7.7 @MoShitrit #10416
• Add support for containerd v1.4.3 ARM64 @hakman #10418
• Add release note for terraform launch template migration @rifelpet #10423
• Expose metrics port when PrometheusMetricsEnabled set to true in Calico @avdhoot #10414
• Bump etcd client to 3.4.13. Use go modules @olemarkus #10425
• Use the kubernetes-sigs version of yaml @olemarkus #10427
• Bump heredoc to v2 @olemarkus #10429
• Update container runtime service files @hakman #10428
• Template functions for recommended kubernetes versions @olemarkus #10369
• Make CoreDNS the default DNS server @rajansandeep #7919
• Delay defaulting to CoreDNS to k8s v1.20 @hakman #10435
• Bump go-bindata and use go module @olemarkus #10421
• Bump sftp to 1.12 @olemarkus #10436
• IAM ServiceAccount Roles: truncate name at 64 characters @justinsb #10437
• Bump helm to v3 @olemarkus #10426
• cloudmock - guard the VPC CIDR association calls with a mutex @rifelpet #10440
• Upgrade mkdocs dependencies to latest @rifelpet #10433
• Spotinst: Schedule Ocean Controller to Linux nodes only @liranp #10444
• Bump AWS-CNI to version 1.7.8 @MoShitrit #10447
• protokube - query host by label when setting tags @rdrgmnzs #10413
• Allow Calico to run on systems with loose reverse path forwarding @hakman #10442
• Bump k8s versions on alpha and bump Ubuntu AMI version on stable @MoShitrit #10464
• Remove gjtempleton as reviewer @gjtempleton #10466
• Calico: Allow operators to choose which encapsulation mode to use @seh #10404
• Spotinst: Ignore volume type case sensitivity to prevent unnecessary updates @liranp #10450
• Spotinst: Expose Ocean Headroom percentage and autoconfig labels @liranp #10449
• Spotinst: Support for multiple subnets per zone @liranp #10452
• Add new-pod-scale-up-delay in Cluster Autoscaler spec @akshedu #10471
• Replace (some) deprecated ResourceHolder with Resource @justinsb #10472
• Remove ResourceHolder: remove last usages and remove code @justinsb #10478
• Refactor MirroredAsset into mirrors package @justinsb #10475
• Refactor nodeUpConfigBuilder to be standalone @justinsb #10476
• Avoid recursive type definitions in schema @justinsb #10482
• Drop support for containerd 1.2 @hakman #10483
• Update CNI plugins to v0.8.7 @hakman #10481
• Add Azure support @kenji-cloudnatix #10114
• Refactor GCE InstanceTemplate @justinsb #10477
• Use Region method of fi.Cloud @justinsb,@rifelpet #10474
• Spotinst: Bump the Ocean Controller to 1.0.69 @liranp #10487
• Added event-qps and event-burst flags to kubelet @DOboznyi #10486
• Add config options for container runtime package URL and Hash @hakman #10473
• Fix cluster setup when KOPS_ARCH is set @hakman #10496
• Docs: Rename "Development" section to "Contributing" and add instructions to update the base AMI version of Ubuntu @MoShitrit #10455
• Release notes for 1.19.0-beta.3 @hakman #10497
• Use containerd.sock for AmazonVPC CNI with containerd @hakman #10502
• Remove support for Kubenet with containerd @hakman #10501
• Add containerd option for registry mirrors @hakman #10507
• Treat InvalidDhcpOptionsId.NotFound as already-deleted @wongma7 #10508
• Add required toleration to gpu documentation @silashansen #10509
• AWS IAM Role Tagging @rifelpet #10488
• Update stable channel with recent k8s releases @MoShitrit #10514
• Run k/k's e2e suite via new kubetest2 make target @rifelpet #10504
• Remove copyright YEAR from generated Go files @bmelbourne #10520
• e2e - dump cluster manifests into artifacts and add --kubernetes-version @rifelpet #10522
• kubetest2: Pass through some AWS env vars @justinsb #10525
• kubetest2: add initial support for GCE @justinsb #10524
• Add gp3 Volume Type to etcd @msidwell #10453
• Only include API server additional security groups in InstanceGroups for masters @seh #10519
• Update kube-router to v1.1.1 @hakman #10512
• IRSA - continue adding route53 permisions to masters @rifelpet #10529
• Add possibility to set volume throughput for gp3 volumes @hakman #10530
• Prefix etcd cluster names with letters @hakman #10361
• Recognize ubuntu 20.10 @justinsb #10278
• Don't allow ebs volume TF resource names to begin with digit @rifelpet #10424
• Add K8s Docker runtime support deprecation release note @bmelbourne,@hakman #10371
• Make it possible to change the etcd volume type and iops @olemarkus #10461
• Promote Ole Markus to approvers list @hakman #10542
• Add containerd config file to Flatcar based instances @hakman #10540
• Add control-plane node role label to cp nodes @olemarkus #10397
• Move bootstrapchannelbuilder to a dedicated package @olemarkus #10409
• kubetest2: support specifying admin-access value @justinsb #10526
• GCE: Don't warn about NVME @justinsb #10548
• Simple upgrade test using kubetest2 framework @justinsb #10523
• Refactor and centralize distribution logic @justinsb #10538
• Fix to handle exit code of gazelle command in hack/verify-bazel.sh @h3poteto #10182
• COS/GCE: exec on kubelet/flexvolume dirs @justinsb #10547
• Fix typo in comment @fenggw-fnst #10541
• Openstack: Prevent data race in servergroup member list @justinsb #10553
• Updates GCE channels to use ubuntu over COS @geojaz #10554
• Kubetest2 - use our own tester that wraps kubetest2's ginkgo tester @rifelpet #10549
• Spotinst: Specify Spot percentage per Instance Group @liranp #10551
• update gophercloud dependency @zetaab #10556
• Upgrade Go v1.15.6 / Bazel v3.4.1 @bmelbourne #10550
• Remove node-authorization @olemarkus #10439
• [addons/CA] Add support for specifying resources and metrics @dntosas #10281
• Spotinst: Iterate over metadata labels only once @liranp #10560
• Default cgroup driver to systemd from k8s 1.20 @bharath-123 #10419
• AWS CSI driver @olemarkus #10467
• Upgrade cfn-lint to 0.44.3 @rifelpet #10565
• Fix file not found error detection in fs:// @rifelpet #10566
• Fix NLB listener -> target group association for TF & CF @rifelpet #10567
• Spotinst: Bump the Ocean Controller to 1.0.70 @liranp #10573
• Spotinst: Specify whether scale-down activities should be restricted @liranp #10561
• [OpenStack] Use new hash format in instance names @zetaab #10557
• kubetest2 - Add manifest template support @rifelpet #10559
• Updates to Alpha versions - k8s & kOps @MoShitrit #10576
• Use Bazel 3.4.1 for postsubmit jobs @hakman #10578
• Give kubetest2 its own makefile @rifelpet #10577
• Use consistent naming for the remaining SGRs part two @olemarkus #10188
• [DigitalOcean] add e2e tests @srikiz #10575
• Allow nodeup (and others) to replace in-use files @justinsb #10581
• Dial-down logging on flagbuilder @justinsb #10582
• Fix default make target @rifelpet #10584
• containerd: Add /etc/crictl config to enable crictl @justinsb #10585
• Add CF integration test for gp3 volumes @hakman #10569
• Release 1.20.0-alpha.1 @hakman #10591