⚠️ Upgrade Notes
- Force cleanup scope and timeout:
cleanupkhnow treats the HCloud project selected by the Terraform token as dedicated to one cluster and proposes deleting every runtime resource in that project, including unrelated resources. It defaults to a dry run; persistent data remains opt-in. Existing v3.1.0 K3s and RKE2 clusters also show an expected in-place update to the ingress load balancer destroy-cleanupterraform_dataon the next apply. That apply must persist the new 45-second SSH timeout before a later destroy can use it. - Additional firewall ownership:
extra_firewall_idsare authoritative through each module-managed server. If the same server/firewall relationship is currently managed by a standalonehcloud_firewall_attachment, follow the state-only ownership handoff indocs/operations.mdbefore applying; two Terraform resources must not manage the same attachment. - Label-selected SSH keys: every key matched by
ssh_hcloud_key_labelmust be a valid RSA, Ed25519, ECDSA NIST P-256/P-384/P-521, or OpenSSH FIDO public key. Unsupported legacy DSA keys, OpenSSH certificates, and malformed key lines now fail during plan instead of failing node bootstrap. - Existing autoscaler nodes: new autoscaler nodes restore both
health-checker.serviceand the configuredtransactional-update.timerstate after bootstrap. Existing autoscaler nodes retain their original cloud-init; use the in-place service repair indocs/operations.mdor recycle them gradually. Static nodes are repaired automatically on the next apply. Restoringhealth-checker.servicealso restores the next-boot transactional snapshot health validation and automatic rollback semantics that are deliberately suppressed during first-boot Kubernetes provisioning. - Existing metadata routes: the metadata-route correction is cloud-init behavior for new or replaced nodes. Existing affected nodes must be repaired in place or replaced one at a time; see
docs/operations.md. - Existing K3s Calico clusters:
calico_valuesnow reaches the K3s Kustomization and the patch is applied on the next run, rolling the cluster-widecalico-nodeDaemonSet. Use a maintenance window and verify node networking. An existing Calico IPPool CIDR is not changed automatically; use Calico's controlled IPPool migration procedure rather than deleting an in-use pool. RKE2 does not consumecalico_values. - Secrets encryption: existing control planes repair the EncryptionConfiguration owner, mode, and SELinux context during the next configuration run. In-place key replacement or disabling encryption is rejected because a one-key swap can make existing Secrets unreadable. This release has no ownership handoff or multi-key input; keep the original Terraform state/key or migrate to a new cluster when rotation is required.
🚀 New Features
- Static agent nodepools and individual map-backed agent nodes can now add scoped
extra_firewall_ids. The effective list is the union of global, nodepool, and node IDs, with plan-time enforcement of Hetzner's five-firewall-per-server limit (#2271; thanks @tiran133). - Added an opt-in per-nodepool
delete_protectionsetting for static agent servers. It enables Hetzner delete and rebuild protection without changing existing clusters; it is an API safety control, not a backup or Terraformprevent_destroyreplacement (#2264; thanks @stefan-sommer-osp).
🐛 Bug Fixes
- Refreshed the independently reviewed SHA-256 pins for the publisher-signed rolling MicroOS x86 and ARM appliances, fixed their first boot hanging in cloud-init after
dhcpcdacquired its Hetzner lease, and label verified RKE2 tar installs under/optfor Rancher's confined runtime domain. Snapshot builds now select cloud-init's bundledudhcpcclient for metadata discovery before first boot without weakening the fail-closed signature, filename, or exact-byte checks; fresh MicroOS/RKE2 nodes no longer leave HCloud CSI Unix-socket traffic blocked under enforcing SELinux. - Kept autoscaler-created server
user_databelow Hetzner Cloud's 32 KiB API limit by compressing large cloud-init payloads before embedding them. Oversized autoscaler configuration now fails during plan; static-node and NAT-router cloud-init has the same exact-size guard before the HCloud server API call, once rendered network values are known. - Fixed fresh Leap Micro/MicroOS nodes rejecting Terraform's first SSH connection when an image snapshot retained the same root public key with a forced-command restriction. First boot now replaces stale options by key identity while preserving unrelated operator keys, both Packer finalizers remove bootstrap authorized keys from the persistent
/rootsubvolume before image capture, and unsupported label-selected Hetzner keys fail during plan instead of node bootstrap. - Fixed K3s agents with floating IPs receiving the server-only
flannel-external-ipflag, which prevented the agent service from starting. Existing static agents reconcile their config and restart in place without HCloud server replacement (#2270, #2272; thanks @redcapcloud and @tiran133). - Restored
health-checker.serviceand the requestedtransactional-update.timerstate after static and autoscaler bootstrap, with fail-closed verification instead of silently suppressing service errors. Existing static nodes are repaired persistently on the next apply, including legacy cloud-init that masked the checker on every boot (#2266, #2267; thanks @LilDjaga). - Kept the Hetzner metadata service reachable when private-network DHCP advertises a black-holed direct
/32route for169.254.169.254. New nodes reconcile the exact NetworkManager route through the public gateway, reapply the profile, verify the selected route, and verify metadata access; indirect/private-only paths remain untouched (#2268, #2269; thanks @karsten42). - Applied
calico_valuesas a K3s-only Kustomization patch instead of uploading and then ignoring it, with plan-time rejection of malformed Kubernetes patch documents only when K3s Calico consumes the input. RKE2 continues to use its bundled Calico chart and ignores legacycalico_values(#2265; thanks @beslovas). - Hardened RKE2 and K3s Secrets EncryptionConfiguration handling: explicit control-plane/agent roles keep key material off agents; control planes enforce root-only staging and destination permissions, SELinux-aware labeling, key-change/disablement rejection, cleanup on failure, and config rollback when a restart fails. This also fixes RKE2 bootstrap on SELinux-enforcing nodes (#2263; thanks @antony-jr).
- Made the generated-site contract test portable to clean GitHub Actions runners instead of requiring undeclared
rg. CI installs Zsh and Fish and fails closed when a documented shell verifier is missing; local runs print an explicit skip when an optional shell is unavailable.
🔧 Changes
- Rebuilt
cleanupkhas a fail-closed, project-aware CLI. It ignores ambient HCloud credentials, pins every API call to the Terraform token, verifies and activates the cluster-named context, inventories all runtime resources and snapshots before mutation, filters empty CLI rows, handles delete protection, separates persistent-data choices, and verifies the final inventory. Best-effort ingress cleanup now gives up on unreachable SSH after 45 seconds instead of retrying for ten minutes. - Kept GitHub CI focused on cheap required lint, documentation drift, and tag publication; HCloud smoke, credentials, cluster inspection, and teardown now remain local. Tag publication rejects missing release notes.
- Restored the one-command
createkhandcleanupkhflows for Bash/Zsh and Fish.scripts/create.shagain works when downloaded directly, while manifest verification and atomic Packer bundle publication stay behind the simple entrypoint.
📚 Documentation
- Reorganized the README into a concise visual overview, four-step Quick Start, and documentation index; moved upgrades, support details, day-2 operations, recipes, and troubleshooting into focused guides while preserving the project-support and Hetzner acknowledgement footer.
- Added the K3s certificate-expiry recovery step for retrieving a renewed admin kubeconfig over SSH from a healthy control-plane node and replacing its loopback API endpoint locally.
What's Changed
Other Changes
- chore: complete v3.1.0 post-release maintenance by @mysticaltech in #2256
- docs(skills): preserve merged PR records by @mysticaltech in #2257
- Restore the simple createkh workflow by @mysticaltech in #2258
- docs: make README a focused project entry point by @mysticaltech in #2259
- docs: restore project support and Hetzner footer by @mysticaltech in #2260
- Prepare the v3.2.0 release candidate by @mysticaltech in #2273
- Deletion protection for agent nodes by @stefan-sommer-osp in #2264
- fix: keep metadata service off the private network when DHCP advertises a /32 for 169.254.169.254 by @karsten42 in #2269
- feat: support per-nodepool extra firewall IDs by @tiran133 in #2271
- fix: omit server-only flag from floating IP agents by @tiran133 in #2272
- fix(rke2): restore SELinux context for encryption-config by @antony-jr in #2263
- fix(autoscaler): re-enable transactional-update.timer after agent install by @LilDjaga in #2267
- Fix project-aware cluster cleanup by @mysticaltech in #2274
New Contributors
- @stefan-sommer-osp made their first contribution in #2264
- @karsten42 made their first contribution in #2269
- @antony-jr made their first contribution in #2263
- @LilDjaga made their first contribution in #2267
Full Changelog: v3.1.0...v3.2.0