Release notes for kOps 1.23 series
⚠ kOps 1.23 has not been released yet! ⚠
This is a document to gather the release notes prior to the release.
Significant changes
Other significant changes
Breaking changes
-
Support for Kubernetes version 1.17 has been removed.
-
Support for the Lyft CNI has been removed.
-
Cilium now has
disable-cnp-status-updates: true
by default. Set this to false if you rely on the CiliumNetworkPolicy status fields.
Required actions
Deprecations
-
Support for Kubernetes version 1.18 is deprecated and will be removed in kOps 1.24.
-
Support for Kubernetes version 1.19 is deprecated and will be removed in kOps 1.25.
-
Support for CentOS 7 is deprecated and will be removed in future versions of kOps.
-
Support for CentOS 8 is deprecated and will be removed in future versions of kOps.
-
Support for Debian 9 (Stretch) is deprecated and will be removed in future versions of kOps.
-
Support for RHEL 7 is deprecated and will be removed in future versions of kOps.
-
Support for Ubuntu 18.04 (Bionic) is deprecated and will be removed in future versions of kOps.
-
All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.
-
The
node-role.kubernetes.io/master
andkubernetes.io/role
labels are deprecated and might be removed from control plane nodes in future versions of kOps. -
Due to lack of maintainers, the Aliyun/Alibaba Cloud support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this cloud provider.
-
Due to lack of maintainers, the CloudFormation support has been deprecated. The current implementation will be left as-is until the implementation needs updates or otherwise becomes incompatible. At that point, it will be removed. We very much welcome anyone willing to contribute to this target.
Other changes of note
Full change list since 1.22.0 release
1.23.0-alpha.1 to 1.23.0-alpha.2
- Release 1.22.0-beta.1 @johngmyers #12211
- Revert "Release 1.22.0-beta.1" @johngmyers #12213
- Release 1.23.0-alpha.1 @johngmyers #12210
- [addons/metrics-server] Bump minor version @dntosas #12198
- Release notes for 1.22.0-beta.1 @johngmyers #12216
- Remove code for unsupported features @johngmyers #12218
- Bump cloudbuild images to latest @rifelpet #12222
- Fix new staticcheck errors @burntcarrot #12215
- Move bootstrap RBAC from protokube to core bootstrap addon @johngmyers #12221
- Update staticcheck to latest @rifelpet #12224
- [Documentation update] Note for the metrics configuration for etcd @recollir #12223
- fix parse semver @zetaab #12227
- check does iface has associations @zetaab #12232
- Bump cert-manager to 1.5.3 @olemarkus #12229
- Fix etcd3-migration docs link @RetWolf #12238
- Update metrics-server tag used in e2e scenario @rifelpet #12239
- Bump node termination handler to 1.13.3 @olemarkus #12237
- [addons/node-local-dns] Bump version and make image field configurable @dntosas #12219
- Set kube-apiserver as default logs container @olemarkus #12246
- Enable protect-kernel-defaults by default and set the correct sysctls in nodeup @olemarkus #12193
- Use node internal IP for metrics-server @olemarkus #12260
- Set ipv6 nameservers on aws @olemarkus #12259
- Set NodeIPFamilies in ipv6 mode @olemarkus #12258
- Disable masquerade means disable masquerade if ipv6 too @olemarkus #12265
- Update controller-runtime to v0.9.7 @hakman #12264
- Try to bootstrap when at least one IP is available @hakman #12270
- Fix core manifest @olemarkus #12268
- update openstack CSI @zetaab #12251
- Fix bootstrap when at least one IP is available @hakman #12273
- Fix kernel parameter for IPv6 forwarding @hakman #12271
- Dump external-dns and dns-controller pod logs @rifelpet #12276
- Add .log suffix to dumped pod logs @rifelpet #12278
- Add support --kernel-memcg-notification Kubelet flag @hakman #12267
- Set explicit fsType to be able to mount volumes @pa1op #12280
- dev: hack/update-expected script should clear KOPS_ARCH @justinsb #12283
- Add helper to convert kubemanifest.Object to unstructured @justinsb #12284
- Add notes and docs on external-dns @olemarkus #12279
- use ipip Always by default in OpenStack @zetaab #12286
- Include kops- prefix in external-dns TXT record @rifelpet #12289
- External dns fixes @olemarkus #12288
- Do not set ClusterCIDR for KubeProxy when using CNI networking and kubeProxy.clusterCIDR is not set @dezmodue #12207
- Update AWS CCM tags @rifelpet #12104
- Enable IMDS IPv6 endpoint @rifelpet #12290
- Note about breaking change for apiserver files @olemarkus #12294
- Use MasterInternalName for gossip cluster SA issuer @rifelpet #12297
- [kubelet] Add validation for cpuCFSQuotaPeriod after k8s>=1.20 @dntosas #12292
- Recognize Ubuntu 21.10 (Impish Indri) @hakman #12300
- Bump snapshot-controller to 4.2.1 @olemarkus #12303
- Update Go to v1.16.8 @hakman #12306
- Bump aws ebs csi driver to 1.2.1 @olemarkus #12310
- Add ability to yaml-encode unstructured @justinsb #12282
- Pre-install nvidia container runtime + drivers on GPU instances @olemarkus #11628
- Update Bazel to v4.2.1 @hakman #12311
- Do not precreate dns record for api lbs @olemarkus #12308
- Default to latest staging image for AWS CCM @hakman #12291
- Make AWS CCM NodeIPFamilies configurable @hakman #12305
- Use sg rule ids and tags where possible @olemarkus #12314
- Allow arbitrary length terraform literals @rifelpet #12316
- Report the power state of cloud instances @yadneshk #12326
- include new required permissions in documentation @yeus #12328
- Use TLS for kubescheduler health check as of k8s 1.23 @olemarkus #12333
- Update ALPHA k8s versions with Sept releases @MoShitrit #12337
- Implement support for AWS ipv6 prefixes @olemarkus #12112
- Add IMDS IPv6 endpoint in terraform @rifelpet #12298
- Deprecate TerraformJSON feature flag @rifelpet #12341
- Skip SCTP e2e tests on cilium + k8s 1.23 @rifelpet #12348
- Use EC2 and Metadata IPv6 endpoints in IPv6 mode for EBS CSI Driver @hakman #12349
- fix: remove inconsistent comment @eddycharly #12351
- fix: etcd backup docs @eddycharly #12352
- Fix EC2 IPv6 endpoint for EBS CSI Driver controller @hakman #12353
- Create a second Terraform provider for managed files @rifelpet #12322
- Recognize pending EC2 instances as needed deletion @justinsb #12357
- Script for iterating development on AWS @justinsb #12356
- Only add IPv6 IAM permissions if using IPv6 @justinsb #12355
- Fix controller defaults for both bootstrap tokens and ipv6 @justinsb #12354
- Update Calico to v3.20.1 @hakman #12360
- Fix version check in cilium SCTP test skipping @rifelpet #12365
- Only configure IMDSv2 on AWS @justinsb #12369
- AWS: Move some subnet functions into AWS model @justinsb #12367
- Fix nil pointer error where containerd is not in use @justinsb #12374
- kubetest2 - add support for dumping k8s resources via SSH @rifelpet #12372
- Better logging in setLifecycleOverride @justinsb #12376
- Fix parsing of kops toolbox dump yaml output @rifelpet #12377
- Move cidrmap to subnet package @justinsb #11578
- GCE: Don't create utility subnets in private topology @justinsb #12373
- GCE: Fix subnet deletion @justinsb #12370
- GCE: Always have IPv6 rules in "ipv6 mode" @justinsb #12368
- [channels] Bump AWS/GCP/Azure Ubuntu AMIs to latest @dntosas #12334
- GCE: For IPAlias or Custom Routes, we must recognize source by CIDR @justinsb #12371
- kubetest2 - fix parsing output from toolbox dump @rifelpet #12379
- Update Go to v1.17.1 @hakman #12375
- GCE: When using calico, need to open up ipip protocol @justinsb #12384
- Remove unneeded network related sysctls @hakman #12385
- Add specific taints to dns-controller. @olemarkus #12389
- Add sleep to upgrade-ab scenario @olemarkus #12391
- Make channels target phony @olemarkus #12392
- feat: add support for wildcard in roles generated for IRSA @eddycharly #12342
- Support zone autocompletion @justinsb,@hakman #12366
- Decrease connection timeout when dumping logs from instances @rifelpet #12397
- Fix list of supported cloud providers in CLI docs @rifelpet #12396
- Fix link to addons documentation @johngmyers #12402
- Upgrade terraform to 1.0.7 @rifelpet,@hakman #12403
- Add support to configure Cilium CNI chaining @choutone #12407
- Run verify-cloudformation in host network @rifelpet #12410
- Upgrade cnf-lint to 0.54.2 @hakman #12411
- Add option to create an internal load balancer for the bastion @dezmodue #12321
- Remove critical-pod scheduler annotation. @rifelpet #12398
- Revert "Remove unneeded network related sysctls" @olemarkus #12415
- Add bidirectional BPF mount for Cilium >= 1.9.10 or >= 1.10.4 @ReillyBrogan #12394
- Allow adding more subnets to an NLB @olemarkus #12412
- release-process.md: Update references to artifact promotion @justaugustus #12386
- Update google.golang.org/api to 0.57.0 @justinsb #12421
- protokube: don't try to connect to apiserver if not control-plane @justinsb #12424
- Refactor bootstrap verifier/authenticator into its own package @justinsb #12422
- Add kubetest2-kops template for testing dedicated APIServer nodes @rifelpet #12428
- Have toolbox dump include contents of /etc/hosts @rifelpet #12427
- Allow aws-iam-authenticator to be scheduled onto dedicated apiserver nodes @rifelpet #12426
- Mount cgroupv2 for cilium at a custom location @olemarkus #12431
- Update Amazon Linux 2 documentation to mention the 5.10 kernel @rifelpet #12430
- Add ability to provide custom CoreDNS tolerations and affinity @hierynomus #12234
- Document cloud-init behaviour @dezmodue #12438
- Skip certain e2e tests in GCE @rifelpet #12434
- Fix AWS IAM Authenticator support for k8s 1.22 @rifelpet #12425
- feat: add support for custom audience in aws oidc provider @eddycharly #12419
- Fix typo in name of new 'UseServiceAccountExternalPermissions' variable. @ev-hines #12440
- Truncate cluster name in NTH EventBridgeRules @rifelpet #12439
- Use separate cloud.config file for in-tree vs out-of-tree components @rifelpet #12435
- Add fixed version to all addons @olemarkus #12416
- Add support for YAML/JSON output to 'kops get instances' @hierynomus #12442
- Update stable k8s versions @MoShitrit #12454
- Bump aws-cni version to 1.9.1 @MoShitrit #12455
- Skip load balancer test in IPv6 clusters @rifelpet #12452
- Update containerd to v1.4.10 @hakman #12459
- Remove unnecessary sysctl "net.ipv6.conf.all.accept_ra=2" @hakman #12461
- Bump Cluster Autoscaler images @olemarkus #12463
- Create v1alpha3 apiVersion @johngmyers #12406
- v1alpha3 API fixup @johngmyers #12466
- Release notes for 1.22.0-beta.2 @johngmyers #12468
- Fail if an apimachinery conversion is missing @johngmyers #12469
- kube-controller-manager also doesn't mount /srv/kubernetes @johngmyers #12473
- Cleanup ClusterSpec code @johngmyers #12472
- Remove Docker overlayfs upgrade code @johngmyers #12471
- Spotinst: Update
spotinst/ocean-controller
to v1.0.78 @liranp #12476 - Update containerd to v1.4.11 @hakman #12479
- Update Docker to v20.10.9 @hakman #12481
- Update k8s dependencies to v1.22.2 @hakman #12483
- Add Cilium agent pod annotations support to improve personalization @sterchelen #12414
- [DigitalOcean] Incorporate existing vpc support for kops @srikiz #12485
- Add kubescheduler.config.k8s.io/v1beta2 for k8s 1.22+ @hakman #12486
- Update controller-runtime to v0.10.2 @hakman #12490
- Update Calico to v3.20.2 @hakman #12491
- Replace klog flags with go-runner in k8s 1.23 @rifelpet #12494
- Support GCE TPM verification @justinsb #12420
- Remove unused fields cluster APIs @johngmyers #12475
- Update Bazel rules_go to v0.29.0 @hakman #12496
- Update etcd-manager to 3.0.20211007 @hakman #12497
- Remove some unused fields from v1alpha3 componentconfig @johngmyers #12500
- Add Cilium + RHEL8 release note @rifelpet #12504
- Update Go to v1.17.2 @hakman #12502
- Convert go.mod files to Go 1.17 @hakman #12505
- Release notes for 1.20.3 @hakman #12512
- Release notes for 1.21.2 @hakman #12513
- Bump channels @hakman #12516
- Update release process docs @hakman #12514
- Remove more unused fields from v1alpha3 API @johngmyers #12517
- Update coredns to v1.8.5 @hakman #12518
- Fix addon-resource-tracking e2e scenario @rifelpet #12520
- Spotinst: Support for RI commitments @liranp #12522
- tests: create-cluster integration tests should validate additional objects @justinsb #12285
- Update Bazel rules_docker to v0.20.0 @hakman #12523
- Enable ingress hostname feature for OpenStack @zetaab #12525
- Upgrade EBS CSI Driver to v1.4.0 @rifelpet,@olemarkus #12529
- Release notes for 1.22.0 @johngmyers #12533
- Spotinst: Permission to create
bigdata.spot.io/bigdataenvironments
@liranp #12521 - Add missing "a" to 1.22 release notes @yurrriq #12536
- Bump cert-manager dependency @olemarkus #12537
- Don't hard-code the SQS Queue ARN partition @rifelpet #12540
- Minor updates to releases document @johngmyers #12546
- Add rolling updates doc to menus @johngmyers #12550
- [cilium] Add support for bpf-lb-sock-hostns-only field @dntosas #12524
- kops-controller: register coordination scheme @justinsb #12553
- Add capacity-optimized-prioritized as a valid spot allocation strategy @ripta #12560
- Disable CNP status updates by default @olemarkus #12564
- Update google/go-containerregistry to v0.6.0 @hakman #12566
- Re-add ec2:DescribeLaunchTemplateVersions to CA IAM policy @rifelpet #12568
- Upgrade tests - cleanup previous cluster with newer kops version @rifelpet #12570
- Upgrade AWS VPC CNI to 1.9.3 w/ k8s 1.22 support @rifelpet #12573
- Update channels @hakman #12548
- Make it possible to set CAS max-node-provision-time @olemarkus #12437
- Update release documentation @johngmyers #12578
- Release notes for 1.22.1 @johngmyers #12579
- Prune addons via labels @justinsb #12156
- Support BYO IPv6 @johngmyers #12582
- Mention KOPS_ARCH in contributor documentation @johngmyers #12586
- Allow AWS LBC to attach certificates @olemarkus #12309
- gossip: Seed /etc/hosts in nodeup @justinsb #12554
- GCE: use chrony on Ubuntu + GCE @justinsb #12587
- Upgrade Canal to v3.20 with k8s 1.22 support @rifelpet #12584
- GCE: Allow network to be marked as shared @justinsb #12590
- Add calico-kube-controllers for Canal @hakman #12593
- Support BYO IPv6 @johngmyers #12592
- gce: allow router to refer to network object @justinsb #12591
- Update Canal based on Calico @hakman #12594
- Spotinst: Get instance types from
mixedInstancesPolicy
field @liranp #12549 - Rename BAZEL_BIN to BAZEL in Makefile @hakman #12599
- gce: open kops-controller port from nodes @justinsb #12556
- Handle keypair items without certificates @johngmyers #12601
- Respect any MaxPods value the user sets explicitly @hakman #12603
- Add permissions needed for KCM to provision NLBs @olemarkus #12611
- dns-controller: Treat IPv6 node addresses as both internal and external @johngmyers #12608
- GCE: improve network & subnet terraform support @justinsb #12382
- GCE: support egress specification @justinsb #12600
- Fix GCE router terraform reference @rifelpet #12618
- Use the SQS Queue's ARN reference @rifelpet,@hakman #12571
- Remove vestigial Cilium ContainerRuntimeLabels code @johngmyers #12615
- Improve default CIDR assignments for IPv6 @johngmyers #12617
- Make dns-controller delete placeholder addresses for IPv6 cluster @johngmyers #12605
- Use .bazel-bin to help gopls & VSCode @justinsb #12498
- GCE: Delete routes with long cluster names @justinsb #12619
- Use instance metadata to find local IPv6 prefix @hakman #12622
- Create placeholder DNS records of correct type for IPv6 clusters @johngmyers #12616
- kubetest2: remove duplicate admin-access flag @justinsb #12625
- kubetest2: force printing of the plan on cluster creation @justinsb #12624
- Use server-side apply for addons. Identify as kops @olemarkus #12583
- Use InternalIP as preferred kubelet address only in ivp6 mode @olemarkus #12626
- Clarify the deployment responsible for API DNS in error message @rifelpet #12277
- Dump more resource types from kubectl into cluster-info directory @rifelpet #12631
- Configure aws-iam-authenticator using identityMappings defined in cluster.yaml @hierynomus #12538
- Never masquerade IPv6 with Cilium @johngmyers #12623
- Add more IPv6 integration tests @hakman #12634
- Log
kops toolbox dump
output to artifacts dir rather than stdout @rifelpet #12639 - Integration test cleanup @rifelpet #12637
- Allow kops-controller to describe network interfaces @hakman #12641
- Fix hardcoded ARN partitions @rifelpet #12638
- Check for hardcoded partitions in integration tests @rifelpet #12635
- Enable Router Advertisements for Debian 11 @hakman #12642
- Use prefixes for IPv6 with Calico @hakman #12643
- doc: Number subsections that are procedural steps @johngmyers #12645
- Revert "Make dns-controller delete placeholder addresses for IPv6 cluster" @johngmyers #12646
- dns-controller: use aliases for internal host-network pods @johngmyers #12640
- Prohibit masquerading in IPv6 clusters @johngmyers #12647
- Validate CNI can support IPv6 @johngmyers #12650
- Remove unused Cilium fields from v1alpha3 API @johngmyers #12610
- Remove obsolete documents @johngmyers #12648
- Enable Router Advertisements for Debian 11 on ens* interfaces @hakman #12652
- IPv6 requires external CCM @johngmyers #12651