kubernetes-sigs/kubespray v2.28.0

on GitHub

Announcement

⚠️ This is the last version of RHEL 8 that we support, and it will be deprecated in the next release, see #11872 for a discussion and reasons why.
⚠️ We have removed the Weave CNI test in previous versions and will remove it in the next release because the project has been deprecated.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Krew installation support is removed (#11824, @VannTen)
• Action required
  You should remove the leading 'v' of all explicit version of components deployed by kubespray (most notably kube_version) (#11890, @VannTen)
• Action required
  etcd_kubeadm_enabled (was deprecated) is removed. You should remove it from your inventory (#11901, @VannTen)
• gateway_api_experimental_channel is deprecated, please use gateway_api_channel and set experimental. (#11763, @tico88612)

Changes by Kind

Feature

• Add Kubernetes 1.32.x hash (#12161, @tmurakam) (#11885, @yankay) (#12003, @mzaian) (#12052, @0ekk)
• Add containerd 2.0.x hash (#11845, @mzaian) (#12011, @mzaian)
• Update runc binary to v1.2.4
  Set containerd_limit_open_file_num to 1048576 so it's configurable. (#11845, @mzaian)
• Update runc binary to v1.2.5 (#12011, @mzaian)
• Make nerdctl 2.0.3 default (#11913, @mzaian)
• Add deploy_coredns: bool (true by default), to let kubespray deploy or not coredns in kube-system (#12218, @ant31)
• Add option ubuntu_stop_unattended_upgrades to stop Ubuntu unattended upgrades (#12174, @0ekk)
• Add support for ranges: (start‑stop or single start) as an additional way to define Cilium LoadBalancer IP pools, alongside the existing cidrs: field. (#12140, @Kimcheolhui)
• Adds the script controb/offline/upload2artifactory.py for offline environments. (#11886, @bbaassssiiee)
• ArgoCD updated to version 2.14.5 to maintain compatibility with Kubernetes version 1.31. (#12041, @farshadasadpour)
• Automatically publish ingress-nginx service address if manual address is not specified and ingress-nginx is not using host network (#11879, @ThisIsQasim)
• Bump node-local-dns (k8s-dns-node-cache) image (#11981, @sathieu)
• Cilium CNI installation replaces Jinja template with Cilium CLI
  cilium_agent_custom_args and cilium_operator_custom_args are deprecated, please use cilium_agent_extra_args and cilium_operator_extra_args.
  cilium_identity_allocation_mode default change to crd.
  cilium_enable_host_legacy_routing default change to false.
  Add CIlium hubble export advanced flow log settings (cilium_hubble_export_file_max_backups, cilium_hubble_export_file_max_size_mb, cilium_hubble_export_dynamic_enabled and cilium_hubble_export_dynamic_config_content)
  Deprecated cilium_ipsec_node_encryption, replace it with cilium_encryption_node_encryption (#12101, @tico88612)
• Default etcd snapshot count to 10000 (#11997, @ErikJiang)
• Enable_dual_stack_networks deprecated, refact network stack with separate ipv4 and ipv6 (#11953, @borislitv)
• Ensure metrics port exists for nodelocaldns/nodelocaldns-second daemonsets (#11998, @Rickkwa)
• Fix cilium network plugin config issue deploying cilium 1.17 (#11986, @pedro-peter)
• For RHEL hosts, checking for subscription status timeout after rh_subscription_check_timeout (default to 3 minutes) (#12115, @VannTen)
• Gateway API can be brought forward before the CNI installation. (#12189, @tico88612)
• Improve ntp package conflict handling (#12212, @ErikJiang)
• Increase the control plane memory requirement to 2GB (#11864, @yankay)
• Network: Fix calico-kube-controller can't list the tiers resources (#12169, @cyclinder)
• Setting up a Docker image service for offline installation on a Mac (#11960, @diguage)
• Support containerd registry mirror certificate configuration (#11857, @KubeKyrie)
• Support kube-proxy nftables mode (#12060, @yankay)
• Terraform upcloud: Add possibility to setup cluster using nodes with no public IPs (#11696, @Xartos)
• Terraform: Added support for UpCloud routers and gateways (#11386, @Xartos)
• The external_cloud_provider support manual option lets users install the cloud controller manager themselves. (#11883, @tico88612)
• Tolerations of cilium-operator deployments can be defined using the cilium_operator_tolerations group_var (#12200, @felipe88alves)
• Update default crio capabilities to allow rancher to start (#11989, @jvkassi)
• Update CI test from AlmaLinux8 to AlmaLinux9 (#11889, @yankay)
• Update kube-vip to v0.8.9 (#11983, @sathieu)
• Upgrade OpenStack Cloud Controller Manager to v1.32.0 (#12121, @tico88612)
• Upgrade ingress-nginx to version v1.12.1 to resolve critical vulnerabilities (CVE-2025-1974 and others) and webhook certgen to v1.5.2. (#12075, @farshadasadpour)
• Upgrade kube-router to 2.1.1 (#12066, @VannTen)
• Upgrade load balancers image version to Nginx 1.27, Haproxy 3.1. (#11928, @guoard)
• Upgrade the default Docker version to 28.0 (#12070, @tico88612)
• Users can now configure hubble-export-file-max-backups and hubble-export-file-max-size-mb through the Kubespray inventory. (#12072, @ErmolenkoMaxim)
• [calico] Update default calico to v3.29.2 (#12012, @mzaian)
• [kubernetes/control-plane] Added support for structured AuthorizationConfiguration files. (#11852, @chadswen)

Documentation

• Fix documentation for offline usage by adding the 'v' prefix in download urls (#12166, @tmurakam)
• Fix path to facts.yml in node facts refresh section (#12177, @guoard)
• Fix sample inventory for the reserved resource (#11895, @anshuman-agarwala)
• No longer reserve outdated cephfs-provisioner installation and documentation (#12113, @tico88612)
• No longer reserve outdated rbd-provisioner installation and documentation (#12114, @tico88612)
• Our CRI-O default capabilities remove NET_RAW and SYS_CHROOT. (#12018, @tico88612)

Failing Test

• Add dns_autoscaler_affinity and remove in-place values. (#12165, @tico88612)
• Fix CI by exclude the .ansible in .ansible-lint
  Remove ctr image pull workaround for nerdctl (#11948, @yankay)

Bug or Regression

• Add support for control plane reconfiguration on upgrades
  Add support for kubeadm-config v1beta4 UpgradeConfiguration.apply and UpgradeConfiguration.node
  Use kubeadm upgrade node during secondary control plane node upgrades (#12015, @chadswen)
• Enable NRI by default on containerd (following containerd defaults) (#12152, @ShinyaIshitobi)
• File download.url's are masked unless the extra var unsafe_show_logs is true. (#11959, @bbaassssiiee)
• Fix a bug where kubeadm_certificate_key was not defined if control plane nodes were not in correct order (#11875, @Xartos)
• Fix a bug where custom TCP/UDP ports were not exposed by the ingress-nginx-controller container and service. (#11850, @commx)
• Fix broken calico Typha template when using both calico_ipam_host_local and typha_secure (#11917, @c-romeo)
• Fix broken dhclient hooks when using resolvconf (#11946, @kyrbrbik)
• Fix control plane pods deletion with proper shell quoting (#11943, @iptizer)
• Fix coredns deployment with coredns_pod_disruption_budget: true or enable_nodelocaldns_secondary (#11952, @RaulButuc)
• Fix hubble-ui deployment to not renders tls volume when the cilium_hubble_tls_generate option not configured. (#12143, @atobaum)
• Fix scale.yml problems with cached IP facts (#12020, @0ekk)
• Fix: Using the ./manage-offline-container-images.sh register command does not create a new container but registers the image in the existing container registry. (#11964, @DearJey)
• Fix: arm64 checksums for youki and kata-containers (#12173, @ErikJiang)
• Fix: missing 'v' prefix in offline image tags (#12086, @ErikJiang)
• Fix: prevent kubeadm to override coredns configuration/deployment on upgrade (#12028, @sathieu)
• Fixed an issue where the second and subsequent parameters in kubelet_cpu_manager_policy_options were ignored due to incorrect indentation. (#12123, @HoKim98)
• Fixed kube-vip to use kube-vip/kube-vip-iptables image instead of kube-vip/kube-vip when lb_fwdmethod or kube_vip_lb_fwdmethod is set to masquerade (#12145, @aviral-agarwal)
• Install symlinks parroting as other control plane nodes etcd certificates (and key) on all control plane nodes, to make kubeadm works (#12181, @VannTen)
• Kubelet-csr-approver moves to regular application installation (#12141, @tico88612)
• New Boolean default variable leave_etc_backup_files: true, set to false for uncluttered /etc directory on target nodes. (#11937, @bbaassssiiee)
• [calico] Fix kubecontrollersconfigurations list permission (#12035, @darkobas2)

Other (Cleanup or Flake)

• Binary checksums are no longer overridable from inventories or host facts (#12234, @VannTen)
• Calico-node pods no longer have a cpu limit by default (#11914, @VannTen)
• Enhance safety and validation mechanisms in the node removal process (#12085, @farshadasadpour)
• Heketi playbook (contrib) is removed. (#12091, @VannTen)
• Kubectl bash completion and alias available for Suse operation systems family (#11860, @noama-nv)
• Kubespray-defaults role is renamed to kubespray_defaults (#12202, @VannTen)
• Remove contrib/kvm-setup and contrib/mitogen. (#12093, @VannTen)
• Rename role bootstrap-os to bootstrap_os (#12203, @VannTen)
• Update KUBESPRAY_VERSION for v2.27.0 (#11854, @yankay)
• Update containerd.options key name (#12170, @flpanbin)
• Upgrade CI for openSuse 15.6 (#12074, @yankay)

Component versions

• kubernetes 1.32.5
• etcd 3.5.16
• docker 28.0
• containerd 2.0.5
• cri-o 1.32.0
• cni-plugins 1.4.1
• calico 3.29.3
• cilium 1.17.3
• flannel 0.22.0
• kube-ovn 1.12.21
• kube-router 2.1.1
• multus 4.1.0
• weave 2.8.7
• kube-vip 0.8.0
• cert-manager 1.15.3
• coredns 1.11.3
• ingress-nginx 1.12.1
• argocd 2.14.5
• helm 3.16.4
• metallb 0.13.9
• registry 2.8.1
• aws-ebs-csi-plugin 0.5.0
• azure-csi-plugin 1.10.0
• cinder-csi-plugin 1.30.0
• gcp-pd-csi-plugin 1.9.2
• local-path-provisioner 0.0.24
• local-volume-provisioner 2.5.0
• node-feature-discovery 0.16.4