saschagrunert/kubernix v1.0.0

on GitHub

Highlights

KuberNix 1.0 is the first stable release. From now on, the command line, the exit codes, the environment file, the status and JSON log output and the stable files of the run directory follow semantic versioning, see Stability.

New commands

• kubernix stop stops the cluster of a run directory from another terminal or script, also while it is still starting.
• kubernix status shows the state, mode, nodes, runtimes, API server endpoint and node readiness, also as JSON via --output json.
• kubernix shell -- <command> runs a single command in the cluster environment and exits with its exit code.
• --addons none starts a cluster without CoreDNS.

Faster and simpler environments

• The evaluated Nix environment is cached as a profile in the run directory, so starts and new shells skip the flake evaluation. The profile is a garbage collector root as well.
• In multinode root mode, the node containers run the binaries from the Nix store of the host instead of building an image with Nix inside, which removes the floating nixos/nix:latest base image.
• KuberNix generates the certificates and kubeconfigs itself, so cfssl is not needed anymore.

Runtimes

• CRI-O monitors the containers of a pod with one conmon-rs instance.
• crun and runc are registered as runtime handlers for CRI-O and containerd, so pods can select the other one via a RuntimeClass.
• Both CRI runtimes share the same defaults and ignore the configuration files, OCI hooks and CDI specs of the host. The registry credentials and certificate directories of the host still apply.

Networking

• Every run directory gets its own bridges (kx<id>.<n>), and in root mode the node IP is the bridge gateway, so it does not change with the network of the host.
• CoreDNS listens on the node IP and port 10053, so it no longer collides with local resolvers like systemd-resolved.
• KuberNix checks the component ports before it changes the system and restores the sysctls it changed once the cluster stops, under a host-wide lock.
• Traffic between pods of different nodes is not masqueraded anymore, and hostPort works with both CRI runtimes.

Rootless mode

• KuberNix checks the systemd user session, subordinate IDs, newuidmap and newgidmap, user namespaces, cgroup v2 and port 6443 before it starts, and the troubleshooting section describes how to fix them.
• Multinode rootless clusters work: every kubelet gets its own mount namespace and pod logs directory.
• kubernix shell enters the namespaces of the running cluster and keeps the working directory.

Security

• Separate CAs for etcd, the kubelet serving certificates and the front proxy, key usages per role, the NodeRestriction admission plugin, no anonymous API server access and an audit policy.
• Root mode refuses run directories which other users could tamper with. Files get written atomically without following symbolic links, and the kubeconfigs and the PKI are only readable by their owner.

Lifecycle

• Termination signals are handled from the start of the bootstrap, and another signal force quits. The next start removes the leftovers of a killed instance, like processes, node containers, bridges, mounts and firewall rules.
• Certificates and keys get validated when a cluster restarts, and only the affected ones get regenerated.
• KuberNix reports the cluster as running only once the default service account exists, so pods can be created right away.

Upgrading from 0.4

• Run directories of 0.4 cannot be used anymore, since the network layout and the PKI changed. Remove them before starting a new cluster.
• The short options -e, -f and -u are hidden aliases now, use --no-shell, --log-format and --container-runtime instead.
• --packages and --addons take comma separated or repeated values, but no space separated values anymore. The same applies to KUBERNIX_PACKAGES and KUBERNIX_ADDONS.
• make run and the other Makefile targets run KuberNix rootless by default.
• Release binaries only exist for x86_64 and aarch64.
• A custom --dockerfile uses the directory of the Dockerfile as build context.

See Upgrading from 0.4 for the details.

Release artifacts

Every release now ships an SPDX 2.3 SBOM per binary and a SHA256SUMS file, which is signed keyless with cosign (SHA256SUMS.sigstore.json). The binaries carry GitHub build provenance and SBOM attestations, and an OpenVEX document tracks whether known vulnerabilities of dependencies affect KuberNix. See Verifying Releases for how to check them.

Dependencies

Kubernetes v1.37.1, CRI-O v1.37.2 with conmon-rs v1.1.1, containerd v2.4.1, etcd v3.7.2, crun v1.30.1, runc v1.5.2, podman v5.8.9, rootlesskit v2.3.6 and CoreDNS v1.14.6 (see What is inside for the full list).

Install

Download a static binary for your architecture below, or install from crates.io:

cargo install kubernix

Full Changelog: v0.4.3...v1.0.0