Release notes for kOps 1.28 series

Significant changes

AWS

• Node Termination Handler is now enabled by default.

GCP

• metadata-proxy is no longer deployed on GCP clusters for Kubernetes 1.29+.

Breaking changes

AWS

• The kops get assets --copy command no longer sets object-level public-read ACLs in the destination fileRepository.

Other breaking changes

• Support for Kubernetes version 1.22 has been removed.

• Support for Ubuntu 18.04 is has been removed.

• Support for Canal, Flannel, and Kube-Router has been removed for Kubernetes 1.28 and later.

• RHEL-based distros will no longer have wget, curl, python2, and git packages installed. Install them with hooks if needed.

Deprecations

• Support for Kubernetes version 1.23 is deprecated and will be removed in kOps 1.29.

• Support for Kubernetes version 1.24 is deprecated and will be removed in kOps 1.30.

• Support for AWS Classic Load Balancer for API is deprecated and should not be used for newly created clusters.

• All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.

What's Changed

• Add create cluster flag for specifying the list of etcd clusters by @hakman in #15552
• Add option for specifying the list of etcd metrics urls by @hakman in #15553
• Add CL2 test command to scalability scenario by @prateekgogia in #15538
• make cni plugin configurable in scaling test scenario by @prateekgogia in #15557
• Upgrade Karpenter to v0.27.5 by @anthonyhaussman in #15144
• Allow overriding uint values by @hakman in #15551
• Update dependencies by @hakman in #15562
• azure: Enable support for public load balancer by @hakman in #15563
• Revert "Remove obsolete etcd versions" by @hakman in #15564
• azure: Fix finding load balancers without subnets by @hakman in #15567
• Update etcd-manager to v3.0.20230630 by @hakman in #15568
• azure: Add support for network security groups by @hakman in #15570
• Don't download container runtime assets when skipping the installation by @hakman in #15579
• Promote alpha channel to stable by @hakman in #15581
• hetzner: Update CCM to v1.16.0 by @hakman in #15577
• aws: Avoid spurious changes in EBSVolume for KmsKeyId by @hakman in #15573
• docs(cilium): fix several broken links by @agilgur5 in #15325
• docs: remove kube-dns-autoscaler when upgrading to CoreDNS by @agilgur5 in #15584
• docs(cilium): update links to latest v1.13 by @agilgur5 in #15583
• doc: Added documentation about loadbalancer and security group configuration by @valentin-ricard in #15588
• Update Karpenter to v0.28.1 by @hakman in #15585
• kops-controller: create IPAM controller for GCE by @justinsb in #15591
• Increase client-side throttling limits by @hakman in #15593
• ipv6: containerd routes support for IPv6 by @justinsb in #15594
• Validate additionalNetworkCIDRs only set on AWS by @johngmyers in #14921
• Use private topology for apiserver e2e test by @johngmyers in #14905
• scaleway: add scaleway zones to autocompletion by @Mia-Cross in #15603
• gce: Add support for bastions by @hakman in #15602
• Fix Karpenter failure to start on IPv6 clusters by @johngmyers in #15605
• gce: Update logic for internal LB by @hakman in #15332
• Move GCE networkCIDR prohibition to validateNetworking() by @johngmyers in #15610
• v1alpha3: Rename GCE networking to GCP by @johngmyers in #15612
• Remove references to ClusterSpec from nodeup sysctls.go by @johngmyers in #15613
• gce: Set firewall rules for Internal LBs also by @justinsb in #15611
• gce: Rename firewall SSH rules for bastion by @hakman in #15614
• scaleway: switched credentials reading order by @Mia-Cross in #15618
• Remove more references to ClusterSpec from nodeup by @johngmyers in #15620
• Update Go to v1.20.6 by @hakman in #15621
• Update aws-sdk-go to support new AWS SSO profile by @avdhoot in #15616
• scaleway: refactoring: utils functions to get info from tags by @Mia-Cross in #15626
• aws: Allow using the same instance ID as egress for multiple subnets by @hakman in #15628
• scaleway: documentation improvement by @Mia-Cross in #15604
• Deprecate Canal, Flannel, and Kube-router by @johngmyers in #15634
• openstack: Open hubble port 4244 by @zetaab in #15635
• Add support for using swap memory by @hakman in #15632
• gce: Use user-data instead of startup-script metadata key by @hakman in #15607
• Add VFSContext to various clientsets by @johngmyers in #14960
• add removeAll to vfs by @Codelax in #15395
• kops-controller: load objects with version conversion by @justinsb in #15608
• Remove references to ClusterSpec.API from nodeup by @johngmyers in #15615
• azure: Add support for dns=none by @hakman in #15627
• spot: update docs about setting the VNG Size Limits in Launch Spec by @IdanShohamNetApp in #15631
• Remove references to more ClusterSpec fields from nodeup by @johngmyers in #15645
• Remove dead code for non-kops-controller bootstrap by @johngmyers in #15646
• Remove support for bootstrap tokens by @johngmyers in #15648
• Fix comment on patchNodePodCIDRs by @justinsb in #15651
• Support removal of managed node labels by @justinsb in #15650
• feat(karpenter): Variabilize image, logFormat and logLevel by @anthonyhaussman in #15601
• azure: Add mode dependency logic to deletion by @hakman in #15617
• Refactor out references to global vfs.Context by @johngmyers in #15640
• Fix modifying backupRetentionDays by @hakman in #15655
• Update release nodes for kOps 1.27 by @hakman in #15653
• Update scaleway-sdk-go to v1.0.0-beta.19 by @hakman in #15658
• Update dependencies by @github-actions in #15661
• More VFSContext refactoring by @johngmyers in #15662
• Update channels by @hakman in #15660
• azure: Verify node identity using VMSS name instead of tags by @hakman in #15659
• docs: revise the bastion ssh guideline to mitigate permission denied errors by @techieforfun in #15657
• Fix long auth helper cache file name by @norseto in #15547
• More VFSContext refactoring by @johngmyers in #15663
• verify-boilerplate: stricter error checking by @justinsb in #15665
• Add golden-output test for cacheFilePath by @justinsb in #15664
• Determine default API access method by IG subnet type by @johngmyers in #14996
• Improve validation of PodCIDR and ServiceClusterIPRange by @johngmyers in #15623
• azure: Avoid spurious changes in NetworkSecurityGroup by @hakman in #15668
• Continue skipping SCTP tests for cilium until we upgrade to 1.13 by @rifelpet in #15671
• Bump k8s and kops versions used in scenario scripts by @rifelpet in #15672
• Add 1.27 release notes to docs menu by @rifelpet in #15673
• azure: Populate node labels from tags by @hakman in #15667
• v1alpha3: Remove no-longer-used topology fields by @johngmyers in #15676
• Skip ssh-to-all-nodes test in private topology by @johngmyers in #15683
• kubetest2-kops: rename control-plane-size flag to control-plane-count by @justinsb in #15674
• etcd-manager: support symlinking versions by @justinsb in #15565
• Update dependencies by @github-actions in #15685
• Fix addon-resource-tracking scenario for new starting kops version by @johngmyers in #15688
• Print error message when digest image fails by @hakman in #15689
• v1alpha3: remove redundant ConfigStore by @johngmyers in #15678
• Promote alpha channel to stable by @hakman in #15695
• v1alpha3: move state store location config under its own sub-struct by @johngmyers in #15693
• update logs with "cannot render instance groups" instead of "cannot render nodes" in validate_cluster.go by @haojue in #15694
• Use release version of k8s 1.27 in integration test by @johngmyers in #15699
• Update dependencies by @github-actions in #15701
• Upgrade cluster-autoscaler by @johngmyers in #15703
• Promote July 2023 K8s releases to stable by @hakman in #15700
• Enable NTH by default on AWS by @johngmyers in #15666
• gce: fix message around getting firewall rule by @justinsb in #15710
• Log error when PutWarmPool fails by @rifelpet in #15712
• gce: Set labels on ForwardingRules by @justinsb in #15709
• Upgrade AWS CCM by @johngmyers in #15706
• gce load balancers: set LoadBalancingScheme to EXTERNAL explicitly by @justinsb in #15708
• Remove more references to ClusterSpec fields from nodeup by @johngmyers in #15647
• Remove code for unsupported k8s version by @johngmyers in #15716
• upgrade-ab test should use old (deprecated) flags by @justinsb in #15717
• azure: Add support for application security groups by @hakman in #15677
• Rename eventbridge rule in toolbox dump output by @rifelpet in #15721
• Fix WarmPool with --target direct by @rifelpet in #15722
• kubetest2: Mark --control-plane-size as deprecated by @hakman in #15725
• scaleway: handle changes to volumes by @Mia-Cross in #15727
• Update references to control-plane-count by @rifelpet in #15733
• Add prefix delegation for amazon vpc cni in scale tests by @prateekgogia in #15736
• OpenStack: add server group name override annotation by @zetaab in #15735
• azure: Add support for NAT gateway by @hakman in #15737
• Bump cilium to v1.13.5 by @zadjadr in #15730
• aws: Add instance group tag to subnets only with Karpenter by @hakman in #15740
• aws: Don't use ap-northeast-2d and us-east-1e for testing by @hakman in #15741
• Use --master-count in testing for backwards compatibility by @hakman in #15742
• Use Ubuntu 20.04 in load-balancer-controller E2E scenario by @rifelpet in #15747
• E2E - Dont set --master-count if --control-plane-count is provided by @rifelpet in #15748
• feature: Add cluster-id for Cilium by @zadjadr in #15746
• Use arm64 AMI for LBC scenario by @rifelpet in #15749
• Fix AMI SSM parameter for podidentitywebhook scenarioo by @rifelpet in #15754
• Remove references to cloudconfig-related fields from ClusterSpec in nodeup by @johngmyers in #15715
• Fix a bug in setting env variable for amazon vpc cni by @prateekgogia in #15757
• Remove use of ClusterSpec in nodeup by @johngmyers in #15755
• Release 1.28.0-alpha.2 by @johngmyers in #15758
• Fixing a typo in Hetzner Firewall Model management by @marcopalmisano in #15762
• Fix hubble certificate dnsname by @zadjadr in #15756
• Allow setting env vars from the command line by @hakman in #15767
• Fix amazonvpc string casing by @prateekgogia in #15773
• Trim e2e skip regexes for Cilium by @johngmyers in #15753
• Use dns=none for scale tests by @hakman in #15774
• Pass error when failing to get SSM parameter by @hakman in #15775
• openstack: Add missing security groups for cilium etcd by @zadjadr in #15766
• Use the same LBC version for e2e tests as what is deployed by @rifelpet in #15777
• Don't set object-level public ACL in S3 FileRepository by @johngmyers in #15726
• Bump actions/dependency-review-action from 3.0.6 to 3.0.7 by @dependabot in #15780
• Bump actions/setup-go from 4.0.1 to 4.1.0 by @dependabot in #15779
• aws: Ignore volumes set to delete on instance termination by @hakman in #15782
• Update dependencies by @hakman in #15781
• Use us-east-2 region for scale tests by @hakman in #15783
• aws: implement paginator for DescribeLaunchTemplate on buildKarpenterGroup by @ltellesfl in #15785
• Mark flags as deprecated instead of normalizing by @hakman in #15743
• Fix AWS CCM defaults for IPAM to match KCM by @johngmyers in #15670
• Skip failing ProxyTerminatingEndpoints test by @hakman in #15792
• Add a new field for using a custom registry for Cilium by @jandersen-plaid in #15787
• Stop installing misc utils on RHEL distros by @rifelpet in #15797
• kcm: Add support for --endpoint/slice-updates-batch-period by @hakman in #15798
• Allow setting metav1.Duration from the command line by @hakman in #15799
• gce: don't logspam when next-route-hop is starting by @justinsb in #15802
• Bump actions/dependency-review-action from 3.0.7 to 3.0.8 by @dependabot in #15803
• fix: error message typo by @0o001 in #15804
• Continue skipping SCTP HostPort tests in older k8s versions by @rifelpet in #15807
• Create clusters with bigger default subnets by @hakman in #15791
• Bump AWS CNI to 1.13.4 by @moshevayner in #15809
• Bump Cert Manager to 1.12.3 by @moshevayner in #15810
• Use AWS CCM 1.28.1 on k8s 1.28+ by @johngmyers in #15813
• Fix gossip on DigitalOcean by @justinsb in #15815
• add mirror by @justinsb in #15816
• update alpha channel with k8s emergency release and ubuntu ami versions by @moshevayner in #15817
• Promote alpha to stable by @moshevayner in #15819
• Skip UDP LoadBalancer test on k8s 1.26 by @rifelpet in #15822
• Split network CIDR into even bigger subnets by @hakman in #15814
• aws: Create subnets for additional network CIDRs by @hakman in #15805
• Add support for --experimental-allocatable-ignore-eviction kubelet flag by @hakman in #15824
• cleanup: Fix comment on GetExternalNetwork by @justinsb in #15827
• Fix small typo in documentation by @mihow in #15796
• Scale config changes for scale prow job on AWS by @prateekgogia in #15599
• Bump actions/checkout from 3.5.3 to 3.6.0 by @dependabot in #15833
• Find containerd package hash from release sha256sum by @hakman in #15834
• Don't rely on kubectl being installed by @justinsb in #15828
• scaleway: rolling-update feature by @Mia-Cross in #15835
• aws: Add dependency on additional network cidrs for subnets by @hakman in #15841
• Let us spread across all AZ(s) in us-east-2 by @dims in #15843
• Allow custom service account issuer without public bucket by @hakman in #14991
• Update dependencies by @hakman in #15842
• Dependency analysis: include the direct task by @justinsb in #15846
• Use cluster-autoscaler 1.28.0 on k8s 1.28+ by @johngmyers in #15850
• Update dependencies by @github-actions in #15851
• Release 1.28.0-beta.1 by @hakman in #15855
• Automated cherry pick of #15848: Fix warmpool to expose dependencies for dependency analysis by @johngmyers in #15863
• Automated cherry pick of #15866: Default to 100.64.0.0/13 as IPv4 service cluster IP range by @hakman in #15870
• Automated cherry pick of #14893: Don't expose v1alpha3 API by @johngmyers in #15873
• Automated cherry pick of #15868: Only run one replica of controller pods on non-HA by @hakman in #15880
• Automated cherry pick of #15879: Update ko to v0.14.1 by @hakman in #15882
• Automated cherry pick of #15878: Update Calico to v3.25.2 by @hakman in #15881
• Release 1.28.0-beta.2 by @johngmyers in #15888
• Automated cherry pick of #15910: Add Cognito permissions for AWS LBC. by @danports in #15914
• Automated cherry pick of #15919: Update kubelet API with SeccompDefault option. by @hakman in #15924
• Release 1.28.0 by @johngmyers in #15954

New Contributors

• @valentin-ricard made their first contribution in #15588
• @Codelax made their first contribution in #15395
• @norseto made their first contribution in #15547
• @haojue made their first contribution in #15694
• @marcopalmisano made their first contribution in #15762
• @0o001 made their first contribution in #15804
• @mihow made their first contribution in #15796

Full Changelog: v1.28.0-alpha.1...v1.28.0