⚠️ v3.0.0 Upgrade Notes
This is the v3 major-release line. Before upgrading from any v2.x release:
- Pin and review:
- Set module version to
3.0.0(or your targeted v3 tag). - Read
docs/v2-to-v3-migration.mdandMIGRATION.mdend-to-end. - Fastest path: with Claude Code or a compatible agent, run the
/migrate-v2-to-v3agent skill from a checkout of this repo — it applies the variable rename map, runs the protected-infrastructure plan gate, and reviews the upgrade plan with you (see the README section "AI-assisted migration").
- Set module version to
- Run safe upgrade flow:
terraform init -upgradeterraform plan- Apply only after reviewing all resource actions.
- If you use private-network NAT routers created before v2.19.0, check for primary IP replacement and perform state migration first (see migration notes).
- Networking behavior changed in v3: nodepool
network_idis active and control-plane attachment behavior is explicit. Cilium public overlay remains an experimental preview gated byenable_experimental_cilium_public_overlayuntil live cross-network datapath validation passes. Prefer blue/green migration for custom/private/Robot/multinetwork topologies; do not apply plans that unexpectedly destroy or recreate network subnets. - New clusters and normal in-place v2 upgrades use
network_subnet_mode = "per_nodepool", matching the released v2 subnet topology. Optionalnetwork_subnet_mode = "shared"is for new clusters or intentional topology changes only. - Several public inputs were renamed or removed in v3 to clean up the module contract. See
MIGRATION.mdfor the old-to-new variable map, especially the inverted positive booleans (enable_hetzner_csi,enable_placement_groups,allow_inbound_icmp,enable_kube_proxy,enable_network_policy,enable_selinux, nodepoolenable_public_ipv4/enable_public_ipv6, autoscaler public-IP flags, and load-balancer enable flags). - The v2
k3s_channeldefault wasv1.33; v3 defaults to the upstreamstablechannel while automatic Kubernetes upgrades still default on. Before the first v3 apply, either pink3s_version, setk3s_channel = "v1.33"to keep the v2 minor channel intentionally, or consciously accept followingstable. - Addon version defaults are deterministic in v3. Unset addon version variables now use a reviewed module matrix instead of upstream latest/floating behavior; set
latestto keep fetching upstream latest releases or following latest Helm charts intentionally.
Version Requirements
- Minimum Terraform/OpenTofu version:
1.10.1 - Minimum hcloud provider version:
1.62.0
💥 Breaking Changes
- Public input cleanup - Renamed Kubernetes distribution, install, audit, load-balancer, Robot, CCM, WireGuard, firewall, placement group, public-IP, kube-proxy, SELinux, and storage inputs to a consistent v3 contract. Removed obsolete inputs such as
enable_iscsid,extra_kustomize_*,autoscaler_labels,autoscaler_taints, and the old CCM deployment-mode switch. SeeMIGRATION.md. - Hetzner CCM HelmChart only - Removed the legacy raw-manifest Hetzner CCM path. v3 always renders the CCM HelmChart manifest and removes old non-Helm CCM objects during addon reconciliation.
- Existing Network shape - Replaced
existing_network_idwithexisting_network = { id = 1234567 };network_id = 0is no longer a valid user value, and omitted/nullnetwork_idmeans the primary kube-hetzner Network. - Nodepool network behavior - Agent and autoscaler
network_idvalues are now active. Control planes stay on the primary Network and do not acceptnetwork_id. - Subnet allocation modes - New v3 clusters and normal in-place v2 upgrades use
network_subnet_mode = "per_nodepool", matching the released v2 subnet topology. Added optionalnetwork_subnet_mode = "shared"for compact new-cluster layouts that intentionally use one shared agent subnet and one shared control-plane subnet. - Minimum tool versions - Terraform/OpenTofu
>= 1.10.1and hcloud provider>= 1.62.0are required. - Default/architecture changes - New nodes default to Leap Micro, architecture selection is consolidated into
enabled_architectures, and default behavior moved to explicit positive booleans.
🚀 New Features
- Declarative Node Annotations - Added create-only Kubernetes Node annotations on control-plane, agent, per-node agent/control-plane overrides, and autoscaler nodepools. Non-empty maps render a node-local cloud-init systemd oneshot that waits for k3s/RKE2, uses the node kubelet kubeconfig, and patches only its own Node; empty maps render no unit or payload, preserving existing cluster user-data. This covers Longhorn default-disk bootstrap annotations and autoscaler-created nodes without adding an ongoing reconciler (#2198, requested by @clemlesne).
- Optional Upgrade Tooling Deployment - Added
enable_kuredandenable_system_upgrade_controllertoggles for clusters that manage reboot orchestration or system-upgrade-controller externally. Disabling these flags omits the resources from future kustomization applies but does not prune already-deployed kured/system-upgrade-controller objects from existing clusters; remove those manually if needed. The k3s and RKE2 kustomization trigger state includes these toggles and the rendered addon payload, so future toggle flips rerun the relevant provisioners (#2223). - Per-Set User Kustomization Apply Flags - Added
user_kustomizations[*].apply_optionsfor passing validatedkubectl applyflags such as server-side apply to individual ordered user kustomization sets (#2218). - Raw kube-apiserver Args Passthrough - Added
kube_apiserver_args(list of strings, default[]) to append arbitrary apiserver flags to the control-planeconfig.yamlkube-apiserver-arg(e.g.service-account-issuer/service-account-jwks-urifor OIDC workload identity) for options without a dedicated module variable. Applied in-place via the existing config-update provisioner (k3s/rke2 service restart, no control-plane node recreation); entries must omit the leading--, enforced by input validation. - Leap Micro Support (Stable Default OS) - Added
osselector for control plane, agent, and autoscaler nodepools (plus per-node agent overrides). New nodepools default toleapmicro; existing nodepools remain on MicroOS by default on upgrade to avoid recreation. New variables:leapmicro_x86_snapshot_id,leapmicro_arm_snapshot_id. Added packer templatepacker-template/hcloud-leapmicro-snapshots.pkr.hcland automatic OS detection via thekube-hetzner/osserver label. - Agent Floating IP Family Selection - Added
floating_ip_type(ipv4/ipv6) to agent nodepools and node overrides, including IPv6-aware NetworkManager reconfiguration logic. - Cilium Egress Gateway HA Reconciler - New
cilium_egress_gateway_ha_enabledoption to deploy a lightweight controller that keeps labeledCiliumEgressGatewayPolicyobjects pinned to a Ready egress node. - Cilium v3 Dual-Stack Defaults - Cilium now renders IPv4/IPv6 Helm values from the configured cluster CIDRs and keeps kube-proxy replacement tied to
enable_kube_proxy = false(#2170, #2178). - Cilium Gateway API Support - Added
cilium_gateway_api_enabledto install standard Gateway API CRDs for the selected Cilium line, enable CiliumgatewayAPI.enabled, and wire cert-manager Gateway API support. Addedexamples/cilium-gateway-api. - Cilium Multinetwork Public Overlay Preview - Added gated
multinetwork_mode = "cilium_public_overlay"plumbing for Cilium-only clusters spanning multiple Hetzner Networks, including public IPv4/IPv6/dual-stack transport selection, WireGuard/tunnel defaults, public load-balancer targeting, control-plane fanout removal, and one Cluster Autoscaler Deployment per effectivenetwork_id. This path now requiresenable_experimental_cilium_public_overlay = trueand is not production-supported until the live datapath E2E passes. - Tailscale Node Transport - Added opt-in
node_transport_mode = "tailscale"for secure single-network clusters and supported private multinetwork scale-out. The module can bootstrap Tailscale, use MagicDNS for Terraform/kubeconfig access, optionally advertise each node's Hetzner private/32route with subnet-route SNAT disabled, keep Kubernetes node IPs on Hetzner private addresses, validate Tailnet/firewall/CNI/load-balancer constraints at plan time with explicit nodepoolnetwork_scope, and render autoscaler nodes with per-Network Tailscale bootstrap. Flannel is the first supported CNI; Cilium remains gated as experimental for this transport until live datapath coverage promotes it. - Embedded Registry Mirror - Added
embedded_registry_mirrorfor trusted large clusters, enabling k3s/RKE2's embedded Spegel mirror while preserving userregistries_configentries. - Placement Group Auto-Sharding - Count-based nodepools without an explicit
placement_groupnow shard implicit Hetzner spread placement groups every 10 servers; explicit placement groups still fail validation above Hetzner's 10-server limit. - Large-Scale Tailscale Examples - Added +100-node and 10,000-total-node Tailscale node-transport reference examples that account for Hetzner Network attachment limits, placement-group limits, autoscaler shards, and the public-IP/Tailnet exposure model.
- Endpoint Introspection Outputs - Added outputs for the effective kubeconfig API endpoint, node join endpoint, node transport mode, and Tailscale MagicDNS hostnames.
- v3 Topology Recommendations - Added
docs/v3-topology-recommendations.mdcovering the recommended dev, HA, NAT, Tailscale, +100 node, 10k reference, RKE2, Cilium dual-stack, Gateway API, Robot/vSwitch, and registry mirror shapes. - Multiple Attached Volumes Per Node - Added
attached_volumessupport for control plane and agent nodepools (including per-node overrides) to provision and mount multiple Hetzner Volumes per node. - NAT Router Customization - Added NAT-router
extra_runcmdanduse_private_nat_router_bastionsupport for private-network bastion hardening (#2165, #2166). - External Overlay Access Hooks - Added and documented the provider-agnostic
node_connection_overridespattern for Tailscale, ZeroTier, Cloudflare WARP, and similar overlays. Use this for user-owned operator access or post-bootstrap overlay features; usenode_transport_mode = "tailscale"when Tailscale should be the official Kubernetes node transport. - Per-Nodepool Snapshot Overrides - Added
os_snapshot_idoverrides to control-plane and agent nodepools and node overrides (#2158). - Plan-Time Configuration Guardrails - Added Terraform/OpenTofu cross-variable validation for architecture toggles, network regions/CIDRs, nodepool topology, load balancers, autoscaler settings, CCM/Robot, Cilium-only features, firewall sources, and multi-volume attachments so invalid combinations fail during
terraform plan. - Robot vSwitch Route Exposure Control - Added
expose_routes_to_vswitchto manage Hetzner Cloud route exposure to coupled Robot vSwitches when kube-hetzner creates the primary Network. - v2-to-v3 Migration Assistant - Added a read-only audit script, project skill, and migration playbook for guided v2 configuration rewrites, plan review, and state-safety checks.
- OpenTofu Support - Documented OpenTofu as a supported engine and expanded Hetzner CI presets to run both Terraform and OpenTofu apply/health/destroy paths.
🐛 Bug Fixes
- Shell-Safe User Input Hardening - SSH keys, Kubernetes install environment values, install exec/version inputs, Cluster Autoscaler extra args, and user kustomization template filenames are now validated or YAML-encoded before root bootstrap/rendering. Unsafe quotes, newlines, shell metacharacters, YAML syntax, and path traversal in those inputs are intentionally rejected.
- Kubeconfig Structural Rename Safety - Rewrote generated kubeconfig endpoint and identity renames through parsed YAML so certificate blobs and other fields containing
defaultare no longer mutated by global string replacement. - Ingress Load Balancer Destroy Cleanup - Added fail-open destroy-time cleanup for module-managed ingress LoadBalancer Services across k3s and RKE2 so Hetzner CCM removes the adopted ingress load balancer before Terraform tears down nodes and network attachments. Found by the v3 live gate after a surviving nginx LB blocked network/subnet destroy.
- Ingress Load Balancer Destroy Retry - CI now retries
terraform destroyonce to absorb the known already-detaching race between Hetzner CCM deleting an adopted ingress LoadBalancer and Terraform detaching the same LB network attachment; plan 011 tracks the longer-term single-ownership design. - Ingress Hook Bootstrap Scheduling - Added hook-scoped bootstrap tolerations to ingress-nginx admission patch jobs and the HAProxy CRD hook so managed ingress Helm installs can finish when v3 bootstraps kustomizations before agent nodes join. Controller Deployments intentionally keep their existing scheduling semantics. Found by the live CI gate via the nginx admission hook deadlock.
- Ingress Load Balancer Annotation Rendering - Fixed ingress-nginx, Traefik, and HAProxy Helm values templates so Hetzner Load Balancer adoption annotations stay nested at the chart-specific Service annotation path instead of being stripped to the values document root by Terraform template trim markers. Found by the v3 CI gate, with a new plan-time semantic contract for the rendered values.
- NAT Router Reconciliation Safety - Reworked the NAT router reconcile provisioner so fresh v3 routers no longer write script text into
sshd_config.d/kube-hetzner.confthrough unterminated nested heredocs, preventing sshd from failing and permanently cutting off the router. Found by the v3 live gate. - Size-Aware Control-Plane Kubelet Reservations - Control-plane nodepools that still use the legacy kubelet reservation default now compute
kube-reservedmemory from the selected Hetzner server type, preventing small RKE2 control planes such ascx23from rejecting scheduler static pods during bootstrap. Added an RKE2 validation guardrail for parseable reservations above 50% of server RAM. Credits the live-gate finding from fresh v3 RKE2 cluster validation. - SSH Authorized Keys Upgrade Safety - Host SSH reconciliation now preserves out-of-band root authorized keys by default while revoking module-managed keys removed from
ssh_public_keyorssh_additional_public_keys. Setssh_authorized_keys_exclusive = truefor strict replacement with only module-managed keys. - NAT Router Failover Peer Scoping - NAT routers now carry the standard cluster identity labels and redundant failover peer discovery filters by
role=nat_router,cluster=<cluster_name>, preventing foreign NAT routers in the same Hetzner project from being selected. - Agent Bootstrap Ordering - Ordered agents after k3s/RKE2 kustomization bootstrap, moved post-install readiness waits after agent join, and kept observable agent start failures for default multi-node clusters (#2215, #2220, #2221).
- Control Plane LB Health Check - Kept the control-plane load balancer health check on HTTP protocol with TLS enabled for the Kubernetes
/readyzendpoint, avoiding invalid Hetznerhttpsprotocol validation failures (#2188, #2199, #2200, #2205). - Autoscaler Large Configs and DRA RBAC - Cluster Autoscaler now reads the generated Hetzner cluster config from a Secret-backed file, uses server-side apply for its manifest, and has read-only RBAC for Kubernetes Dynamic Resource Allocation resources (#2194, #2195, #2202).
- Kured on Tainted Nodes - Added a universal toleration to Kured so OS reboot management still runs on tainted nodes (#2196).
- Subnet Validation Contract - Kept NAT router and vSwitch subnet-index upper bounds conditional on those features being enabled, preserving small
subnet_countvalidation compatibility while still failing invalid enabled configurations (#2197). - External Manifest Fetch Resilience - Added retry blocks to GitHub and public-IP HTTP data sources so transient TLS handshake timeouts do not fail plans, applies, or destroys.
- Autoscaler CA Root Loading - Removed the
/etc/ssl/certshostPath mount from Cluster Autoscaler so RKE2/Leap Micro clusters use the image's bundled CA roots instead of hitting host certificate directory permission failures. - Terraform 1.15 Validation Compatibility - Moved cross-variable and local-dependent module contract checks from input-variable validation blocks into a hard
terraform_data.validation_contractprecondition surface, preserving plan-time failures while allowing Terraform 1.15.0 to initialize and validate the module. - Tailscale Volume Provisioning Ordering - Agent Longhorn and attached-volume configuration now waits for Tailscale agent bootstrap before using Tailnet MagicDNS SSH targets.
- Tailscale Auth-Key Ergonomics -
auth_keymode no longer advertises kube-hetzner tags by default, so simple pre-auth keys work without TailnettagOwners; tagged nodes remain an explicit opt-in and OAuth mode now validates that tag-scoped auth is configured. - Tailscale Single-Network Ergonomics - Tailscale mode now cleanly supports ordinary single-network clusters: node-private route advertisement can be disabled when no
network_scope = "external"nodepools are used, private control-plane Load Balancers are allowed, and private managed ingress Load Balancers are rejected only for external-network scale-out. - Tailscale Same-Root Network Validation - Tailscale static agent and autoscaler nodepools now use explicit
network_scope = "primary" | "external"intent, so invalid same-root external Network configurations fail duringterraform planeven whennetwork_idis not known until apply. - Placement Group Disable/Limit Semantics -
enable_placement_groups = falsenow stops creating unused placement-group resources, and plan-time validation enforces Hetzner's 50-placement-group project limit before large static topologies hit provider errors. - Same-Root Tailscale External Networks - In Tailscale transport mode, nodepool
network_idvalues can come from Hetzner Network resources created in the same Terraform root because control planes no longer need apply-time fanout attachments to every external agent Network. - Cloud-Init Health-Checker Race - Host and autoscaler cloud-init now masks Leap Micro/MicroOS
health-checker.servicebeforecloud-finalto prevent a systemd ordering-cycle race that can skip first-boot Kubernetes bootstrap on autoscaled nodes. - Cilium Multinetwork Bootstrap - Public-overlay clusters now allow restricted outbound Kubernetes API traffic and keep Hetzner CCM network-aware while route reconciliation stays disabled, so control planes can remain on their private node identity and external-network agents can join over the public overlay.
- Cilium Default Bootstrap - Cilium now enables eBPF masquerading only when kube-proxy replacement is enabled, matching Cilium's BPF NodePort dependency and preventing default Cilium clusters from CrashLooping on startup.
- Interface Rename Self-Heal - Added a boot-time
kh-rename-interface.serviceand stale udev MAC refresh so private NIC renames survive MAC changes and reboots (#2182). - User Kustomization Redeploys - User kustomization uploads and deploys now rerun after first control-plane replacement (#2160).
- Custom Ingress Mode -
ingress_controller = "custom"now skips managed ingress Service lookup/wait logic (#2173). - Autoscaler Without Public IPv4 - Autoscaler cloud-init now routes IPv4 through the private gateway when public IPv4 is disabled, while keeping public IPv6 routing when enabled (#2154).
- Hetzner CCM Dual-Stack Address Family - Hetzner CCM now keeps route reconciliation on the IPv4 pod CIDR and sets
HCLOUD_INSTANCES_ADDRESS_FAMILYfor IPv6/dual-stack clusters (#2170). - Cilium Egress Gateway Validation - Enforces
enable_kube_proxy = falsewhen Cilium Egress Gateway is enabled, matching Cilium's kube-proxy replacement requirement (#2178). - Cilium Egress Gateway HA Reconciler - Treats
CiliumEgressGatewayPolicyas cluster-scoped when retargeting labeled policies (#2178). - Upgrade-Safe Ingress Namespace Defaults - Restored legacy nginx default namespace (
nginx) to avoid Helm ownership conflicts during upgrades from v2.19.x clusters. - CCM Ownership Compatibility - Keeps Hetzner CCM on the existing HelmChart manifest flow, avoiding release-name collisions with already-installed CCM chart instances.
- CCM Helm Migration Cleanup - v3 now removes the full legacy non-Helm Hetzner CCM RBAC surface before installing the Helm-managed CCM, while preserving Helm-owned CCM resources on later applies.
- Upgrade-Safe Hetzner SSH Key State - Preserves the v2-managed
hcloud_ssh_key.k3sresource during v3 upgrades instead of auto-adopting the same key through a data source and planning key deletion. - Addon Manifest Fetch Stability - Terraform now fetches kured and system-upgrade-controller release manifests and uploads them as local kustomize resources, avoiding control-plane kustomize failures on GitHub release-asset URLs.
- Subnet Topology Compatibility - Restored per-nodepool control-plane/agent subnet resources and nodepool subnet attachment while keeping auto-assigned private IPv4 behavior.
- RKE2 SELinux Apply Parity - Wired RKE2 server/agent install flows to apply the
rke2-selinuxpolicy module when available and added safe post-installrestoreconrelabeling for RKE2 binaries. - LeapMicro SELinux Snapshot Matrix (k3s/rke2 x x86/arm) - LeapMicro packer now builds distro-specific SELinux snapshots (
selinux_package_to_install), labels snapshots withkube-hetzner/k8s-distroand architecture, and auto-selection now matcheskubernetes_distributionto prevent k3s/rke2 SELinux RPM conflicts. - MicroOS Packer SELinux Scope - Removed
rke2-selinuxpreinstall from the MicroOS packer template; it now only preinstalls and locksk3s-selinux. - LeapMicro SELinux Policy De-duplication - Moved
k8s_custom_policiesinto a shared template consumed by both host and autoscaler cloud-init paths to prevent policy drift. - RKE2 SELinux Enforcing Guardrail - Added an explicit enforcing-mode validation that fails provisioning if the
rke2SELinux module is still not loaded. - Longhorn iSCSI SELinux Capability - Added
iscsid_tdac_overridepermission to the shared kube-hetzner SELinux module. - RKE2 TLS SAN Endpoint Parity -
control_plane_config_rke2now includeslocal.kubeconfig_server_addressintls-san, preserving SAN coverage for NAT-router/private-LB kubeconfig endpoints. - Hetzner CI LeapMicro Snapshot Gate - Hetzner test prerequisites now accept LeapMicro snapshot secrets (with MicroOS fallback) instead of requiring MicroOS-only secrets.
- Autoscaler Nodepool Parity/Validation - Added autoscaler nodepool validation guards (unique names, integer min/max bounds, taint effect and swap/zram format checks) and aligned RKE2 autoscaled node labeling/taint rendering with the k3s autoscaler path.
- RKE2 User Kustomizations - Switched user kustomization apply path to distribution-aware
kubectl_cli, fixing apply failures in RKE2 clusters. - extra_network_ids Attachment - Wired
extra_network_idsinto host provisioning so additional Hetzner networks are actually attached to control-plane and agent nodes. - Connection Override Consistency - Unified control-plane and agent
node_connection_overridesresolution so provisioning and follow-up operations honor the same override key strategy (including suffixed node names). - RKE2 TLS SAN Parity (No LB) - Added kubeconfig/control-plane advertised endpoints to RKE2 non-LB TLS SAN generation to prevent certificate mismatch on custom kubeconfig server addresses.
- Control Plane Bootstrap Config Files - First-node k3s/RKE2 bootstrap now installs authentication and audit policy config files before starting the API server when the matching API-server args are enabled.
- RKE2 First Bootstrap Parity - RKE2 first bootstrap now respects
enable_selinuxand uses the effective kubeconfig/control-plane endpoints in its initialtls-sanlist, matching steady-state config. - Attached Volume Mount Safety - Attached control-plane and agent volumes now rerun mount configuration on size changes, resize XFS via mount path, and persist fstab entries by filesystem UUID instead of mutable device paths.
- K3s Channel Guardrail - Default
k3s_channelnow uses the livestablechannel, and plan-time validation rejects minor live channels except the explicitv1.33v2-preservation path unless an exactk3s_version/rke2_versionis set, avoiding broken upstream minor-channel installer resolution. - API Port Consistency - k3s first bootstrap now honors
kubernetes_api_port, the control-plane LB health check/backend follows the configured listener port, IPv6 kubeconfig endpoints are bracketed correctly, and RKE2 now rejects unsupported non-6443 API port settings. - RKE2 Apply Parity - RKE2 kustomization triggers now include CCM values and system-upgrade drain/eviction/window settings, readiness waits evaluate dynamically, deployment/job waits match k3s, and RKE2 secret deployment uses the shared file-based secret path instead of shell argv literals.
- Node Route Robustness - Host cloud-init now handles public-IPv6-only nodes by routing IPv4 through the private gateway while preserving public IPv6 routing, matching autoscaler behavior.
- Leap Micro First-Boot Readiness - Host provisioning now resets and tolerates the known non-critical
transactional-update.servicefirst-boot failure while still blocking on other failed systemd units. - Upgrade-Safe Detection/Provisioning - Existing hyphenated nodepool names without random suffixes are detected correctly for OS defaults, NAT routers no longer recreate for image/user-data drift, and RKE2 autoscaler SSH keys use the normalized authorized-key list.
- RKE2 Replacement/Reapply Safety - RKE2 first-node bootstrap now retriggers on first control-plane replacement, and RKE2 addon application hashes the rendered kustomization payload so template/resource toggles are reapplied.
- Per-Network Route/Floating IP Detection - Install-time private-route repair and floating IP public-NIC detection now use each node's actual private network CIDR/gateway instead of the primary cluster network.
- NAT Router Config Reconciliation - Existing NAT routers now reconcile cloud-init-owned SSH, DNS, iptables, and keepalived config through Terraform provisioners while connection-critical SSH/user/key changes force router replacement.
- Hetzner CI Presets - CI preset tests now use a compact local-checkout fixture, sanitize preset-derived cluster names, and cover Terraform apply, Kubernetes health checks, and Terraform destroy for default, nginx, and RKE2 presets.
- RKE2 Registry Bootstrap - Initial cloud-init now writes
registries_configto/etc/rancher/rke2/registries.yamlfor RKE2 clusters instead of the k3s path, so custom registries are available before first RKE2 start.
🔧 Changes
-
SSH Port Validation - Tightened
ssh_portinput validation so port0and fractional values fail early. -
SMB CSI Chart Pinning - Pinned the
csi_driver_smb_versiondefault to reviewed chart1.20.3while preserving"*"as an explicit floating opt-in, and removed the unused NFS CSI template scaffold. -
🔧 Disabled Addon Fetch Gating -
enable_kured = falseandenable_system_upgrade_controller = falsenow skip the corresponding GitHub release/manifest HTTP data sources entirely; disabled clusters may see one idempotent kustomization re-run as addon trigger state normalizes disabled remote manifest inputs to empty strings. -
Sensitive Kustomization Backup - The generated kustomization backup now uses Terraform's sensitive local-file resource so Rancher registration manifest URLs are treated as sensitive on disk; rotate any Rancher registration token that may have been written by an earlier backup.
-
Hetzner CI Orphan Sweeper - Added a scheduled/manual sweeper for stale
kh-ci-*Hetzner CI resources. It skips active Hetzner test runs, defaults to dry-run, and requires prefix-anchored names plus age gates before deletion. -
Render Harness CI Gate - Added a hermetic rendered-template harness and negative validation-contract plan fixture so Helm values, ingress LB annotations, cloud-init YAML, shell templates, and key v3 validation preconditions are checked before live cluster gates.
-
Explicit Provider Constraints - Pinned the previously implicit Kubernetes, Helm, Random, and CloudInit provider requirements and expanded CI validation across Terraform 1.10.5, 1.14.9, 1.15.0, and OpenTofu 1.11.6.
-
iSCSI Daemon Defaults -
iscsidis now enabled on all nodes by default, and theenable_iscsidinput was removed. -
Cilium Default Version - Updated the default Cilium version to
1.19.3so v3 defaults align with the current Gateway API-supported Cilium line. -
Gateway API CRD Version Override - Restored the
gateway_api_versioninput for users who need to pin the standard Gateway API CRD bundle independently ofcilium_version, matching the v2 pinning capability. The default empty value keeps v3's Cilium-derived CRD version behavior. -
Primary IP Provider Cleanup - Removed now-unused
assignee_type = "server"attributes from hcloud Primary IP resources and raised the hcloud provider minimum to1.62.0. -
Cloudflare Zero Trust Support Boundary - Documented Cloudflare Access/Tunnel as a user-managed external access pattern for kube API, SSH, Rancher, and ingress, while explicitly keeping Cloudflare Mesh/WARP out of the v3 node-transport support contract. Use Tailscale for supported secure node transport.
-
Release Attribution Robustness - Release workflow now maps commits to associated PR authors (including squash merges) when generating contributor credits, so original implementers are preserved.
-
v3 Migration Skill Update - Updated the repo-local migrate-v2-to-v3 agent skill with live-upgrade lessons, the destroy retry caveat, and the final Tailscale support wording.
👥 Contributors
Thanks to all contributors who made this release possible:
Attribution includes pull request authors mapped from commits, so squashed PR implementers are credited.
Pull Request Authors
- @github-actions[bot]
- @knuurr
- @markus-seidl
- @mikeywuu
- @mysticaltech
- @nikolauspschuetz
- @orlovmyk
- @pat-s
- @stani-o
- @vsalomaki
Full Contributor List
- @github-actions[bot]
- @knuurr
- @markus-seidl
- @mikeywuu
- @mysticaltech
- @nikolauspschuetz
- @orlovmyk
- @pat-s
- @stani-o
- @vsalomaki
- Aleksa Siriški
- Ben Kero
- Dorian Brmbota
- Jacek Tomaszewski
- K. N.
- Karim
- Karim Naufal
- Marek Pikuła
- Markus Seidl
- Michael Ehrlich
- Mike Uspelkat
- Nikolaus Schuetz
- Stanisa Obradovic
- Ville Salomäki
- mysticaltech
- pat-s
- Claude Opus 4.6
- Claude Sonnet 4.6
- Mykyta Orlov
- gemini-code-assist[bot]
- knuurr
What's Changed
📚 Documentation
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2057
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2061
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2064
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2065
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2072
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2142
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2143
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2144
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2145
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2146
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2148
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2149
- [AUTO] Update Terraform Documentation by @github-actions[bot] in #2150
Other Changes
- Moving to LeapMicro by @stani-o in #1628
- feat: default new nodes to Leap Micro (keep MicroOS for upgrades) by @pat-s in #1936
- Add
myipv4as possible firewall setting input by @vsalomaki in #1932 - Provide bastion host if no nat_router is provided by @markus-seidl in #2046
- Add user_kustomizations-map for specifying Kustomization-"sets" to be applied sequentially by @vsalomaki in #1924
- Add test setup for hetzner testing by @vsalomaki in #1931
- refactor: generic external overlay hooks (supersedes #1760) by @mysticaltech in #2055
- Add RKE2 as an additional kubernetes distribution by @mikeywuu in #1784
- feat: make audit-policy updates distribution-aware for k3s/rke2 by @mysticaltech in #2066
- feat: add ssh public key format validation by @mysticaltech in #2068
- feat: support TCP/UDP in traefik_additional_ports by @mysticaltech in #2069
- feat(v3): support per-node cloud-init extra write_files and runcmd (#1039) by @mysticaltech in #2070
- feat: support post-deploy SSH key rotation by @mysticaltech in #2067
- feat(v3): add RKE2 upgrade window support (#1591) by @mysticaltech in #2071
- feat(v3): configurable cluster autoscaler tolerations (#1727) by @mysticaltech in #2073
- feat(v3): per-nodepool keep_disk overrides (#1344) by @mysticaltech in #2074
- feat(v3): add RDNS for NAT router and control-plane LB (#223) by @mysticaltech in #2075
- feat(v3): add custom ingress mode and LB IPv4 output (#284) by @mysticaltech in #2076
- feat(v3): add k3s/rke2 config outputs and warning note (#1435) by @mysticaltech in #2077
- feat(v3): default to blocking inbound ICMP ping (#1730) by @mysticaltech in #2078
- feat: add configurable secrets encryption mode (idea 368) by @mysticaltech in #2081
- idea #1554: configure helm and kubernetes providers using dynamically fetched kubeconfig by @mysticaltech in #2129
- fix: derive host private IPv4 directly from node address (idea 541) by @mysticaltech in #2082
- feat: support optional external private IPv4 on hosts (idea 1041) by @mysticaltech in #2083
- feat: always enable iscsid on all node pools (idea 1172) by @mysticaltech in #2086
- feat: add autoscaler metrics service templates (idea 797) by @mysticaltech in #2087
- idea #517: introduce extra robot nodes variable terraform by @mysticaltech in #2091
- idea #1838: rename master branch main github settings by @mysticaltech in #2092
- idea #2053: update modules host templates cloudinit yaml by @mysticaltech in #2094
- idea #405: create tests directory configuration presets tfvars by @mysticaltech in #2095
- idea #1927: initialize mkdocs material theme python script by @mysticaltech in #2096
- idea #1997: ensure cluster autoscaler resource values is by @mysticaltech in #2100
- idea #976: update readme md officially state opentofu by @mysticaltech in #2101
- idea #899: update nodepool object definitions variables tf by @mysticaltech in #2102
- idea #1926: create new example directory at examples by @mysticaltech in #2104
- idea #872: refactor firewall rules terraform remove default by @mysticaltech in #2106
- idea #1418: migrate all remaining helm based manifest by @mysticaltech in #2107
- idea #801: create distribution aware k8s kubelet config by @mysticaltech in #2110
- idea #1446: introduce cluster ipv6 cidr service ipv6 by @mysticaltech in #2111
- idea #1359: add clear troubleshooting note fix selinux by @mysticaltech in #2112
- idea #1383: init tf add explicit kubectl delete by @mysticaltech in #2113
- idea #1042: add ingress controller use system namespace by @mysticaltech in #2114
- idea #1357: standardize explicitly export full module control by @mysticaltech in #2115
- idea #1038: extend control plane nodepools variable schema by @mysticaltech in #2118
- idea #1447: add kubeapi port variable variables tf by @mysticaltech in #2121
- idea #1219: update k3s rke2 configuration update scripts by @mysticaltech in #2126
- idea #1728: update default value firewall ssh source by @mysticaltech in #2127
- idea #630: add explicit security section to README by @mysticaltech in #2130
- idea #1311: verify robot credentials support is already implemented by @mysticaltech in #2131
- idea #1132: verify kubernetes distribution support is already implemented by @mysticaltech in #2132
- idea #911: verify cilium routing mode support is already implemented by @mysticaltech in #2133
- idea #1839: verify longhorn size/mount extension is already implemented by @mysticaltech in #2134
- idea #1258: verify system-upgrade-controller updates are already implemented by @mysticaltech in #2135
- idea #1729: blocker and implementation plan by @mysticaltech in #2136
- idea #634: blocker and implementation plan by @mysticaltech in #2137
- idea #816: blocker and implementation plan by @mysticaltech in #2138
- idea #1568: blocker and implementation plan by @mysticaltech in #2139
- idea #1526: blocker and implementation plan by @mysticaltech in #2140
- feat: add enable_load_balancer variable (idea 1833) by @mysticaltech in #2079
- feat: add extra network IDs for extra firewall rules (idea 1388) by @mysticaltech in #2080
- feat: add optional Cilium egress-gateway HA support (idea 1076) by @mysticaltech in #2084
- refactor: split control plane/agent extra args by pool (idea 1196) by @mysticaltech in #2085
- feat: add optional floating IP type variable (idea 1769) by @mysticaltech in #2088
- idea #1606: introduce k3s encryption at rest variable by @mysticaltech in #2090
- idea #1129: add terraform data resource modules host by @mysticaltech in #2093
- idea #1261: add enable x86 enable arm boolean by @mysticaltech in #2099
- idea #1485: extend control plane nodepools agent nodepools by @mysticaltech in #2103
- idea #500: introduce combine load balancers boolean variable by @mysticaltech in #2108
- idea #1215: automatically append fail swap false kubelet by @mysticaltech in #2120
- idea #1999: add append random suffix boolean defaulting by @mysticaltech in #2124
- idea #1778: implement join script external output that by @mysticaltech in #2128
- chore: add execute-hard-plan skill and local ignores by @mysticaltech in #2141
- idea #2044: introduce join endpoint type public private by @mysticaltech in #2089
- idea #1360: add etcd snapshot retention variable variables by @mysticaltech in #2097
- idea #936: add chrony disable steps boolean variable by @mysticaltech in #2098
- idea #315: add templates external dns yaml tpl by @mysticaltech in #2105
- idea #1518: create templates velero yaml tpl add by @mysticaltech in #2109
- idea #1384: complete helm values merging support hetzner by @mysticaltech in #2116
- idea #878: add enable csi driver nfs associated by @mysticaltech in #2117
- idea #1358: add longhorn volume size longhorn mount by @mysticaltech in #2119
- idea #1385: add new optional variable egress floating by @mysticaltech in #2122
- idea #1527: transition kured static manifest helmchart crd by @mysticaltech in #2123
- idea #1131: define timeouts variable variables tf e by @mysticaltech in #2125
- feat: drain v3 issue and PR queue by @mysticaltech in #2185
- fix(packer): expose server type/location overrides for snapshot builds (v3) by @mysticaltech in #2230
- fix(agents): order agents after kustomization and surface start failures (v3) by @nikolauspschuetz in #2221
- feat: kube_apiserver_args passthrough for raw apiserver flags by @nikolauspschuetz in #2222
- release: promote staging to master for v3 by @mysticaltech in #2147
New Contributors
Full Changelog: v2.21.0...v3.0.0