Release notes for kOps 1.19 series
(The kOps 1.19 release has not been released yet; this is a document to gather the notes prior to the release).
Significant changes
Changes to kubernetes config export
kOps will no longer automatically export the kubernetes config on kops update cluster
. In order to export the config on cluster update, you need to either add the --user <user>
to reference an existing user, or --admin
to export the cluster admin user. If neither flag is passed, the kubernetes config will not be modified. This makes it easier to reuse user definitions across clusters should you, for example, use OIDC for authentication.
Similarly, kops export kubecfg
will also require passing either the --admin
or --user
flag if the context does not already exist.
By default, the credentials of any exported admin user now have a lifetime of 18 hours. The lifetime of the exported
credentials may be specified as a value of the --admin
flag. To get the previous behavior, specify --admin=87600h
to either kops update cluster
or kops export kubecfg
.
kops create cluster --yes
exports the admin user along with rest of the cluster config, as was the previous behaviour (except for the 18-hour validity).
OpenStack Cinder plugin
kOps will install the Cinder plugin for kOps running kubernetes 1.16 or newer. If you already have this plugin installed you should remove it before upgrading.
If you already have a default StorageClass
, you should set cloudConfig.Openstack.BlockStorage.CreateStorageClass: false
to prevent kOps from installing one.
Other significant changes by kind
General
-
New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.
-
On AWS kOps now defaults to using launch templates instead of launch configurations.
-
There is now Alpha support for Hashicorp Vault as a store for secrets and keys. See the Vault state store docs.
-
The lifetimes of certificates used by various components have been substantially reduced.
The certificates on a node will expire sometime between 455 and 485 days after the node's creation.
The expiration times vary randomly so that nodes are likely to have their certs expire at different times than other nodes. -
kOps now supports using an AWS Network Load Balancer (NLB) for API access.
See the documentation for more info.
CLI
-
The
kops update cluster
command will now refuse to run on a cluster that
has been updated by a newer version of kOps unless it is given the--allow-kops-downgrade
flag. -
New command for deleting a single instance: kops delete instance
CNI
-
Clusters using the Amazon VPC CNI provider now perform an
ec2.DescribeInstanceTypes
call at instance launch time. In large clusters or AWS accounts this may lead to API throttling which could delay node readiness. If this becomes a problem please open a GitHub issue. -
Clusters using Calico with
CrossSubnet
enabled will switch to the new awsSrcDstCheck for disabling the AWS source/destination checks. The previous implementation using k8s-ec2-srcdst is now deprecated. -
Clusters using Calico can now enable the eBPF dataplane mode for Ubuntu 20.04 (Focal) hosts. Add
spec.networking.calico.bpfEnabled: true
andspec.kubeProxy.enabled: false
to the cluster spec to enable. -
Clusters using Calico can now encrypt pod-to-pod traffic with WireGuard for Ubuntu hosts. Add
spec.networking.calico.wireguardEnabled: true
to the cluster spec to enable. -
New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.
Addons
-
Metrics Server is now available as a configurable addon. Add
spec.metricsServer.enabled: true
to the cluster spec to enable. -
Cluster Autoscaler is now availalble as a configurable addon. Add
spec.clusterAutoscaler.enabled: true
to the cluster spec to enable. -
AWS Node Termination Handler is now available as a configurable addon. Add
spec.nodeTerminationHandler.enabled: true
to the cluster spec to enable.
Breaking changes
-
Support for Kubernetes 1.9 and 1.10 has been removed.
-
Support for the Romana networking provider has been removed.
-
Support for legacy IAM permissions has been removed. This removal may be temporarily deferred to kOps 1.20 by setting the
LegacyIAM
feature flag.
Required Actions
-
See note about Openstack Cinder plugin above.
-
Terraform users, in order to prevent downtime you will have to remove the state of any existing ELB or TargetGroup attatchments from your Terraform state file. This is due to migrating the attachments to the in-line
aws_autoscaling_group
fields. See the terraform documentation for more information about the difference. This migration is required due to a bug described in #9913.
To prevent downtime, follow these steps with the new version of Kops:
kops update cluster --target terraform ...
terraform plan
terraform state list | grep aws_autoscaling_attachment | xargs -L1 terraform state rm
terraform plan
# Ensure these resources are no longer being destroyed and recreated
terraform apply
-
If you are using Terraform with an additional .tf file and using "aws_autoscaling_attachment" to attach additional Load Balancers or ALB/NLB Target Groups you'll need to migrate to attaching them through the InstanceGroup spec instead.
-
AWS clusters using an ACM Certificate on the API ELB (
.spec.api.loadBalancer.sslCertificateID
) will need to migrate from Classic LoadBalancer (CLB) to Network LoadBalancer (NLB) prior to upgrading to Kubernetes 1.19 by setting.spec.api.loadBalancer.class: Network
.
Any kubeconfig files using kOps' admin client credentials will need to be regenerated withkops export kubecfg --admin
.
For more information see this page.
Deprecations
-
Support for Kubernetes versions 1.11 and 1.12 are deprecated and will be removed in kOps 1.20.
-
Support for Terraform version 0.11 has been deprecated and will be removed in kOps 1.20.
-
Support for feature flag
Terraform-0.12
has been deprecated and will be removed in kOps 1.20. All generated Terraform HCL2/JSON files will support versions0.12.26+
and0.13.0+
. -
The manifest based metrics server addon has been deprecated in favour of a configurable addon.
-
The manifest based cluster autoscaler addon has been deprecated in favour of a configurable addon.
Partial change list since 1.19.0-beta.1 release
- Remove components from cluster validation @johngmyers #10214
- Update validate cluster cli docs @johngmyers #10219
- Switch ARM64 CI to Graviton2 CPU @hakman #10230
- Fix various nits @hakman #10217
- Make etcd-manager log verbosity configurable @elblivion #10194
- Use LaunchTemplate versions instead of timestamped LaunchTemplates @hakman #10151
- Add default runtime and runtimes fields in the docker config @bharath-123 #10238
- Add HPA Flags for
horizontal-pod-autoscaler-initial-readiness-delay
&horizontal-pod-autoscaler-cpu-initialization-period
@JoelBCarter #10241 - Fix version of storage-aws addon manifest @johngmyers #10247
- Update etcd-manager to 3.0.20201117 @justinsb #10257
- Use separate domain for kops-controller bootstrap @johngmyers #10239
- Revert "Switch ARM64 CI to Graviton2 CPU" @hakman #10262
- Push multi-arch images @hakman #10265
- Use etcd v3.4.13 for k8s v1.19+ @hakman #10277
- Fix cloudformation lint job @rifelpet,@FrankYang0529 #10256
- Parse TargetGroup names from ARNs @hakman #10276
- Release 1.19.0-beta.2 @hakman #10291