⚠️ Upgrade Notes
- IPv6/dual-stack clusters: static control-plane and agent nodes now render
node-ipvalues that match the configured cluster CIDR families whencluster_ipv6_cidr/service_ipv6_cidrare set. Do not enable dual-stack in-place on an existing IPv4-only k3s/RKE2 cluster; cluster/service CIDR families are bootstrap-time choices and require a new cluster or blue/green migration. This adds new plan-time rejections for configurations that previously planned but could not boot correctly: missing effective public IPv6 on any static HCloud node, IPv6-only pod/service CIDRs on the standard private-network path, active autoscaler nodepools on the standard private-network path,nat_routerwith IPv6 cluster CIDRs,extra_robot_nodeswith IPv6 cluster CIDRs,node_transport_mode = "tailscale"with IPv6 cluster CIDRs, public-overlay transport families that omit an enabled cluster CIDR family, and Cilium native routing with IPv6 cluster CIDRs. Use dual-stack, the default Cilium tunnel mode, static HCloud nodes, and public IPv6-enabled nodepools for the standard path. - Experimental public-overlay node IPs: existing
multinetwork_mode = "cilium_public_overlay"clusters may see control-planenode-ipchange from private IPv4 to public overlay addresses and a k3s/RKE2 config restart, even with IPv4-only cluster CIDRs. This lab-only path is still not production-supported. - openSUSE SSH access on new/replaced nodes: cloud-init no longer writes the
ssh_pwauthoverride that shadowed openSUSE's vendorsshd_config, and kube-hetzner's SSH drop-in now explicitly keepsUsePAM yeswhile disabling keyboard-interactive authentication. This applies to newly provisioned or replaced openSUSE nodes; already affected nodes may need manual SSH rescue remediation or replacement. - Autoscaler node IPs on existing nodes: the standard autoscaler now writes private
node-ipduring bootstrap for new/replaced autoscaled nodes, replacing the previous public IPv4 auto-detection on default public-route pools. Already-running autoscaled nodes keep their existing kubelet config until the autoscaler recycles them or you drain/delete them intentionally; top-levelnode-ipvalues inagent_nodes_custom_configremain unsupported and are overridden on autoscaler-created nodes. - Custom Leap Micro Packer mirrors: custom
opensuse_leapmicro_*_mirror_linkvalues now require the matching publisher-signed checksum/signature sidecars and an independentopensuse_leapmicro_*_expected_sha256pin. Query-bearing image URLs must also set explicit*_checksum_linkand*_signature_linkvalues. Put credentials in the architecture-specificopensuse_leapmicro_x86_mirror_authorization_headeroropensuse_leapmicro_arm_mirror_authorization_headeronly when that architecture's three URLs share one HTTPS origin; authenticated redirects, cross-origin sidecars, URL userinfo credentials, and unsigned/unpinned custom images are rejected. - Custom MicroOS Packer mirrors: custom
opensuse_microos_*_mirror_linkvalues require explicit matching checksum/signature sidecars and an independentopensuse_microos_*_expected_sha256pin. Use the architecture-specific sensitive mirror authorization header only for a single HTTPS origin; redirects, cross-origin sidecars, URL userinfo credentials, and unsigned/unpinned custom images are rejected. - Transactional OS SELinux packages: new k3s/RKE2 nodes with SELinux enabled now require the matching
k3s-selinuxorrke2-selinuxpackage and policy to be baked into the selected Leap Micro or MicroOS snapshot. RKE2 bootstrap uses its tar installer so it cannot silently replace the reviewed image package through a live RPM repository. Rebuild custom/legacy snapshots with the current distro-specific Packer matrix before replacing or adding nodes.
🚀 New Features
- Static control-plane and agent nodes now advertise dual-stack
node-ipvalues that match configured cluster CIDR families, making the existing Cilium IPv6 CIDR inputs plan-validated on the standard private-network topology where validation passes (#2170, #2244, #2245; thanks @mgazza, @bkero).
🐛 Bug Fixes
- Fixed non-empty
registries_configrendering with Terraform/OpenTofu by keeping the conditional result type consistent (#2241, #2242; thanks @prochac). - Fixed private-only clusters whose public control-plane host is intentionally absent so plans no longer fail with a
coalesceerror (#2248, #2249; thanks @elkh510). - Fixed new/replaced openSUSE nodes losing SSH access when cloud-init generated a minimal
/etc/ssh/sshd_configfromssh_pwauth: false, which hid the vendorUsePAM yessetting (#2252; thanks @steache). - Fixed IPv6/dual-stack validation so dormant
count = 0control-plane or agent nodepools do not block otherwise valid plans. - Fixed private-only NAT-router/control-plane-load-balancer topologies so
join_endpoint_type = "public"now fails at plan time instead of rendering a null Kubernetes server URL. - Fixed standard autoscaler-created nodes so they render their private
node-ipbefore k3s/RKE2 starts, restoring metrics-server scrapes byInternalIP. - Fixed experimental public-overlay autoscaler bootstrap so it retries and fails closed unless every required public IP family is discovered before writing
node-ip. - Fixed repeat Leap Micro snapshot builds by giving each generated image a UTC timestamp plus collision-resistant build ID, allowing operators to refresh stale OS images without deleting snapshots used by running nodes.
- Fixed
scripts/create.shso the Leap Micro Packer template and every required trust artifact come from one consistent source snapshot, existing user files remain untouched, and download/Packer failures stop the workflow instead of falling through to success guidance. - Prevented Packer's Leap Micro provisioning trace from printing the generated root bootstrap password and hash into build logs.
- Authenticated Leap Micro appliances and Rancher SELinux RPMs against vendored full-fingerprint trust anchors, fail-closed key/signature lifecycle checks, exact artifact identity, and reviewed exact-byte SHA-256 pins before Packer writes or installs them.
- Authenticated rolling MicroOS appliances with the same vendored-trust-anchor and independent-digest model, baked the verified distro-specific Rancher SELinux RPM into a committed transactional snapshot, and made automatic image lookup prefer the matching
kube-hetzner/k8s-distrolabel while retaining only unlabeled legacy snapshots as fallback. - Committed final Leap Micro and MicroOS image cleanup inside transactional snapshots so SSH host-key removal, NetworkManager state, and timezone survive image capture without baking regenerated host keys into clones.
- Fixed post-install readiness failing when Helm replaced an addon deployment between the existence and availability checks; the gate now retries that race against its original timeout.
- Prevented RKE2 bootstrap from downloading
rke2-selinuxat runtime; both distributions now fail closed before installation unless the matching verified policy package is already baked into the selected transactional OS snapshot. - Replaced mutable root execution of
get.k3s.io/get.rke2.ioacross HCloud, autoscaler, Robot, and generated external-node join paths with pinned official installer bytes. Reviewed channel payloads and explicit operator digests are independently pinned; existing custom exact-version configurations remain compatible through strict parsing of that exact official release's checksum publication. Payloads are verified before installation, inherited installer-path controls are neutralized, baked SELinux preflights remain mandatory, and RKE2 stays on its tar path. The generated external helper also fails closed on missing SELinux policy/context setup and starts the verified agent service. - Rejected reviewed RKE2 channel releases at plan time when they do not publish artifacts for every active node architecture, while excluding dormant autoscaler pools and preserving custom exact-version checksum fallback.
- Replaced the moving-ref/raw-script setup bootstrap with one immutable Codeload archive whose reviewed SHA-256 is verified before extraction or execution, independently pinned the Packer manifest, and neutralized inherited source-directory overrides.
🔧 Changes
- Removed stale
stagingbranch guidance from contributor docs, documentation CI triggers, and agent skills; release work now targets explicit release-candidate branches before merge (#2246; thanks @bkero). - Updated
dflook/terraform-fmt-checkfrom 2.2.3 to 3.0.0 (#2243; thanks @dependabot). - Added least-privilege PR gates for Packer formatting/validation, trust-anchor lifecycle, adversarial appliance and installer fixtures, shell analysis, a pinned/checksummed tokenless tfsec scan, and real signed Rancher SELinux RPMs; PR-controlled jobs no longer inherit a writable token or persisted checkout credentials.
- Added optional
k3s_artifact_sha256andrke2_artifact_sha256maps for operators who want independent payload pins on custom exact Kubernetes releases, without adding a plan-time upgrade requirement to existing configurations. - Protected the release control plane outside mutable workflow files:
masternow requires maintainer PR integration, the HCloud secret environment accepts only default-branch runs with maintainer approval and no admin bypass,v*tags are administrator-protected, and release publishing is tag-only. - Added adversarial live-control contracts that reject release-tag exclusions, extra bypass actors, additional HCloud reviewers or secrets, alternate repositories, and incomplete branch/tag rules instead of accepting merely present controls.
- Enforced the immutable release tree in pull-request CI and tag publication: the tagged tree must descend from the canary-tested functional commit and may differ only at the three reviewed README bootstrap pins.
- Regenerated the public quick-start site from the verified README bootstrap, added a drift/moving-source CI rejection, and routed operator snapshot refreshes through the manifest-bound generated bundle and verified Packer plugin matrix.
📚 Documentation
- Cleaned up stale README/site wording around Ansible, MetalLB, and Hetzner firewall defaults (#2247; thanks @SnoozeFreddo).
- Updated v2 → v3 migration and LLM guidance to target v3.1.0/current v3 releases instead of the known-regression v3.0.0 baseline.
- Clarified that the retired KH Assistant Custom GPT is no longer the supported assistant channel; users should install the
/kh-assistantagent skill suite.
👥 Contributors
Thanks to all contributors who made this release possible:
What's Changed
Other Changes
- release: prepare v3.1.0 intake by @mysticaltech in #2253
- release: prepare v3.1.0 by @mysticaltech in #2254
- fix(ci): install hcloud CLI for trusted smoke by @mysticaltech in #2255
Full Changelog: v3.0.1...v3.1.0