Major Changes

Please read this section carefully as it may be breaking changes.

New config format: `vcluster.yaml`

This release introduces the new vcluster.yaml file which centralizes all the configuration options for vCluster and serves as the Helm values at the same time. This new configuration features a completely revamped format designed to enhance the user experience:

Validation: We provide a JSON schema for vcluster.yaml, which is used by vCluster CLI and vCluster Platform UI now validate configurations before creating or upgrading virtual clusters. This schema has also been published to SchemaStore, so that most IDEs will recognize the vcluster.yaml file and provide autocomplete and validation directly in the IDE editor.
Consolidated configuration: All configurations are centralized in the vcluster.yaml file, eliminating confusion previously caused by the mix of CLI flags, annotations, environment variables, and Helm values.
Consistent grouping and naming: Fields in vcluster.yaml are logically grouped under topical categories, simplifying navigation and enhancing discoverability of related features.
Docs alignment: Our documentation now mirrors the structure of vcluster.yaml, making it easier to cross-reference settings within the file and corresponding sections in the docs.

Migrating to `vcluster.yaml`

In order to make it easy to convert your old values.yaml (v0.19 and below) to the new vcluster.yaml format, you can run the new vcluster convert config command. For example, let's take these pre-v0.20 configuration values:

# values.yaml
sync:
  ingresses:
    enabled: true
  nodes:
    enabled: true
  fake-nodes:
    enabled: false
syncer:
  replicas: 3
  extraArgs:
  - --tls-san=my-vcluster.example.com

Running vcluster convert config --distro k3s < /path/to/values.yaml will generate the following vcluster.yaml:

# vcluster.yaml
sync:
  toHost:
    ingresses:
      enabled: true
  fromHost:
    ingressClasses:
      enabled: true
    nodes:
      enabled: true
controlPlane:
  distro:
    k3s:
      enabled: true
  proxy:
    extraSANs:
    - my-vcluster.example.com
  statefulSet:
    highAvailability:
      replicas: 3
    scheduling:
      podManagementPolicy: OrderedReady

For more details on upgrading from older versions to v0.20, please read our configuration conversion guide.

Unified Helm chart for simplified deployment

We consolidated the distro-specific vCluster Helm charts (vcluster (k3s), vcluster-k8s, vcluster-k0s, and vcluster-eks) into a single, unified chart. This change is designed to simplify management and upgrading of virtual clusters:

Single source: No more juggling multiple charts. The vcluster.yaml serves as the single source for all configuration in a unified Helm chart for all distros.
Enhanced validation: We've introduced a JSON schema for the Helm values, ensuring that upgrades will only proceed if your configuration matches the expected format to reduce deployment errors.
Customizable distributions: The new unified chart structure enables easier customization of Kubernetes distributions directly via the Helm chart values:

# vcluster.yaml
controlPlane:
  distro:
    k8s:
      enabled: true

K8s distro now supports SQLite & external databases

So far, virtual clusters running the vanilla k8s distro only supported etcd as storage backend which made this distro comparatively harder to operate than k3s. With vCluster v0.20, we’re introducing two new backing store options for vanilla k8s besides etcd:

SQLite offers a more lightweight solution for data storage without the overhead associated with more complex choices like etcd or external databases. It is the new default for virtual clusters running the vanilla k8s distro.
External Databases allow users to use any MySQL or Postgres compatible databases as backing stores for virtual clusters running the vanilla k8s distro. This especially useful for users who plan to outsource the backing store operations to managed database offerings such as AWS RDS or Azure Database.

Note: Switching backing stores is currently not supported. In order to use this new backing store, you will need to deploy net new virtual clusters and migrate the data manually with backup and restore tooling such as Velero. Upgrading your configuration via vcluster convert config will explicitly write the previously used data store into your configuration to make sure upgrading an existing virtual cluster does not require changing the backing store.

EKS distro has been discontinued

Previously, vCluster offered the option to use EKS as a distro to run vCluster. However, this lead many users to believe they had to use the EKS distro to run vCluster on an EKS host cluster, which is not correct because any vCluster distro is able to run on an EKS host cluster. Given that the EKS distro did not provide any benefits beyond the vanilla k8s distro and introduced unnecessary confusion and maintenance effort, we decided to discontinue this distro. If you want to deploy virtual clusters on an EKS host cluster, we recommend using the k8s distro for vCluster going forward. If you plan on upgrading a virtual cluster that used EKS as a distro, please carefully read and follow this upgrade guide in the docs.

Changes in defaults for vCluster

There are several changes in the default configuration of a vCluster that are important for any users upgrading to v0.20+ or deploying net new clusters.

Default distro changed from k3s to vanilla k8s

We changed the default distribution for the vCluster control plane from K3s to K8s. This is the least opinionated option, offering greater flexibility and compatibility:

Flexibility: More customization and scalability options, catering to a broader range of deployment needs.
Compatibility: In addition to embedded and external etcd, you can now use various storage backends including SQLite, Postgres, and MySQL. This addition addresses previous challenges with using K8s for smaller virtual clusters.

Upgrade Notes: Switching distributions is not supported, so in order to use this new default, you will need to deploy net new virtual clusters.

Default image vcluster-pro

We've updated the default image repository for vCluster to ghcr.io/loft-sh/vcluster-pro. This change allows users to seamlessly test and adopt vCluster Pro features without having to switch images from OSS to Pro. The Pro features are integrated into the Pro image but remain inactive by default to ensure that your experience remains exactly the same as with the OSS image.

Upgrade Notes: When upgrading from previous versions, the image will automatically be updated to start to pull from the new repository. For users who prefer to continue using the open-source image, simply adjust your vcluster.yaml configuration to set the repository to loft-sh/vcluster-oss. See the docs for details.

New Default Scheduling of Control Plane Pod: Parallel

We’ve updated the scheduling rule of the control plane from OrderedReady to Parallel. Since vCluster typically runs as a StatefulSet, this setting cannot be changed after the virtual cluster been deployed.

Increased Resource Requests

We increased the default resource requests for vCluster including increasing:

Ephemeral storage from 200Mi to 400Mi (to ensure that SQLite powered virtual clusters have enough space to store data without running out of storage space when they are used over a prolonged period of time)
CPU from 3m to 20m
Memory from 16Mi to 64Mi

These changes are minimal and won’t have any significant impact on the footprint of a virtual cluster.

Disabled Node Syncing for Kind Clusters

When deploying virtual clusters with vCluster CLI, there is no automatic enabling of syncing real nodes for Kind clusters anymore.

Upgrade Notes: If you want to continue to enable this syncing, then you will need to this configuration to your vcluster.yaml :

sync:
  fromHost:
    nodes:
      enabled: true
controlPlane:
  service:
    spec:
      type: NodePort

Behavior Changes

CLI Updates

There have been significant CLI changes as the above changes have required refactoring how the CLI worked in some areas. Besides the above changes, we merged the overlapping commands found in loft and vcluster pro. The full summary of CLI changes can be found in our docs at the following sites:

General List of CLI Changes - Listing out what’s new, what’s been renamed or dropped.
Guide using vcluster convert to convert values.yaml files for pre-v0.20 virtual clusters to the updated vcluster.yaml to be used in upgrading to a v0.20+ vCluster
Reference guide of loft CLI commands to new vcluster commands

Ingress syncing behavior has changed

Prior to v0.20, when you enabled syncing Ingresses from the virtual to the host cluster, it would also automatically sync all IngressClasses from the host cluster. However, this required a cluster role which some vCluster users don’t have. We’ve now decoupled these syncing behaviors so you can individually enable syncing Ingresses as well as IngressClasses separately.

sync:
  toHost:
    ingresses:
      enabled: true
  fromHost:
    ingressClasses:
      enabled: true

Updated CAPI Provider Support

Our [Cluster API (CAPI) provider](https://github.com/loft-sh/cluster-api-provider-vcluster) has been updated with a new version (v0.2.0) that supports the new vcluster.yaml format.

Change Log

Features

• merge vCluster charts & new values.yaml by @FabianKramm in #1583
• feat: allow external etcd for k3s & k0s by @FabianKramm in #1620
• feat: allow embedded sqlite for k8s & default k8s by @FabianKramm in #1626
• feat: add migration function by @FabianKramm in #1646
• feat: add "migrate values" command to migrate to new values format by @johannesfrey in #1671
• feat: show a message if user attempts to use pro / platform features by @lizardruss in #1677
• feat: add --insecure flag to connect cluster by @FabianKramm in #1757
• feat: delete vCluster instance on helm delete by @FabianKramm in #1750
• feat: Added get current-user sub-command by @ThomasK33 in #1807
• feat: Added Disconnect command for every vCluster-related context by @ThomasK33 in #1821
• feat: update IsProFeatureEnabled for external.platform features by @lizardruss in #1888
• feat: add vcluster activate & apply platform secret by @FabianKramm in #1728
• feat: add migration function by @FabianKramm in #1646
• feat: allow embedded sqlite for k8s & default k8s by @FabianKramm in #1626
• feat: allow external etcd for k3s & k0s by @FabianKramm in #1620
• added draft for interceptor plugin by @facchettos in #1612
• [v0.20] Merge pull request #2031 from hidalgopl/add-chart-value-for-setting-k8s-version by @loft-bot in #2038
• [v0.20] add --ca-cert flag to the vcluster platform add vcluster, so it can b… (#2046) by @loft-bot in #2060
• Credits cmd by @facchettos in #1797
• Platform clusters by @facchettos in #1779
• Platform secrets by @facchettos in #1790
• Cluster access key by @facchettos in #1804
• Management by @facchettos in #1810
• added draft for interceptor plugin by @facchettos in #1612
• [v0.20] [feature] add describe command (#2055) by @loft-bot in #2068
• [v0.20] [enhancement] now refuses to create multiple virtual clusters in same ns (#2052) by @loft-bot in #2056

Fixes

• fix: IPFamilyPolicy not synced for default vcluster service by @rohantmp in #1592
• fix: only apply deploy config once by @FabianKramm in #1600
• fix: delete unused nodes correctly by @FabianKramm in #1607
• fix: override endpoints if service selector is empty by @FabianKramm in #1608
• fix: affinity label selector overridden by namespace selector by @rohantmp in #1609
• fix: multi-namespace mode configmap sync by @FabianKramm in #1657
• fix: unhandled error in endpoints sync by @FabianKramm in #1681
• fix: migrate sync ingresses correctly by @FabianKramm in #1685
• fix: rename embed chart by @FabianKramm in #1691
• fix: let pro commands fail instead of succeed by @FabianKramm in #1692
• fix: Semvare compare missing a v character by @ThomasK33 in #1696
• fix: ignore stderr in helm version check by @rohantmp in #1699
• fix: omitted log messages by @FabianKramm in #1706
• fix: flaking k8sdefaultendpoint syncing by @FabianKramm in #1712
• fix: replace vcluster/config with vcluster-config/config when syncing by @johannesfrey in #1720
• fix: Updated ConnectToPlatform function signature by @ThomasK33 in #1740
• fix: set import-name correctly by @FabianKramm in #1741
• fix: handle helm created vclusters correctly by @FabianKramm in #1744
• fix: e2e flakes when waiting for vcluster by @rohantmp in #1762
• fix: proxy kubelets loadbalancer by @FabianKramm in #1767
• fix: set control plane svc as cert secret owner by @rohantmp in #1859
• fix: incorrect version check by @FabianKramm in #1871
• fix: removed default connect for headless vClusters by @ThomasK33 in #1874
• fix: parse driver error by @FabianKramm in #1875
• fix: background-proxy flag by @FabianKramm in #1876
• fix: strip non alphanumeric characters in background-proxy by @FabianKramm in #1879
• fix: allow updating the virtual cluster using the platform driver by @lizardruss in #1889
• fix: k8s startup issue by @FabianKramm in #2010
• fix: add exception for the platform in network policy by @facchettos in #2008
• fix: add coredns version map with key equales to 1.26 item by @LinPr in #1673
• fix: migrate maps by @FabianKramm in #1991
• fix: migrate command name has changed by @johannesfrey in #1690
• fix: vcluster create merge values & default values by @FabianKramm in #1753
• fix: use correct parameter name for etcd volumeclaim storage class by @neogopher in #1758
• fix: vcluster start --set by @FabianKramm in #1763
• fix: e2e flakes when waiting for vcluster to come up by @rohantmp in #1819
• fix: set owner reference on cert secrets by @rohantmp in #1796
• fix: credits command not working by @rohantmp in #1805
• fix: disable global owner for multins mode by @rohantmp in #1808
• fix: don't init selves before platform start by @rohantmp in #1812
• fix: use platform token in kube config/fix description by @rohantmp in #1817
• fix: e2e flakes when waiting for vcluster to come up by @rohantmp in #1819
• fix: set control plane svc as cert secret owner by @rohantmp in #1859
• fix: use agent pod label to get cluster name by @rohantmp in #1802
• fix: handle helm created vclusters correctly by @FabianKramm in #1744
• fix: Updated ConnectToPlatform function signature by @ThomasK33 in #1740
• fix: set import-name correctly by @FabianKramm in #1741
• fix: migrate sync ingresses correctly by @FabianKramm in #1685
• fix: unhandled error in endpoints sync by @FabianKramm in #1681
• fix: multi-namespace mode configmap sync by @FabianKramm in #1657
• fix: affinity label selector overridden by namespace selector by @rohantmp in #1609
• fix: override endpoints if service selector is empty by @FabianKramm in #1608
• fix: delete unused nodes correctly by @FabianKramm in #1607
• fix: only apply deploy config once by @FabianKramm in #1600
• fix: IPFamilyPolicy not synced for default vcluster service by @rohantmp in #1592
• Enable workflow_dispatch for e2e tests by @heiko-braun in #1911
• print error on cli connection problems by @eumel8 in #1594
• replaced handlername with interceptor wide name by @facchettos in #1662
• moved the interceptor to after authz so plugins don't have to do it themselves by @facchettos in
#1703
• fail early if kine can't start by @facchettos in #1771
• now sets the cache dir correctly by @facchettos in #1811
• removed get service cidr by @facchettos in #1806
• removed dep on loftctl for reset by @facchettos in #1809
• fixed a var shadowing issue that cause nil pointer when creating from helm by @facchettos in #1768
• now takes into account the parameters from the kubeconfig config part by @facchettos in #1672
• added check for k0s to make sure we don't use incompatible options by @facchettos in #1674
• now sets up the controller with the exportkubeconfig namespace if it is set by @facchettos in #1676
• moved withinterceptor so it can catch the info from the context by @facchettos in #1658
• print error on cli connection problems by @eumel8 in #1594
• [v0.20] Merge pull request #2058 from rmweir/reject-empty-datasource by @loft-bot in #2059
• [BACKPORT v0.20] fix: set controlPlane.distro.k8s.version by default by @FabianKramm in #2062
• [v0.20] Merge pull request #2064 from facchettos/multi-vc-ns-allow by @loft-bot in #2065
• [v0.20] Merge pull request #2061 from FabianKramm/main by @loft-bot in #2063
• [v0.20] Merge pull request #2066 from FabianKramm/main by @loft-bot in #2067
• [v0.20] Merge pull request #2054 from zerbitx/DOC-226 by @loft-bot in #2069
• [v0.20] Merge pull request #2047 from sowmyav27/conformance-tests by @loft-bot in #2057

Chores

• chore: bump values schema by @FabianKramm in #1597
• chore: move ipblock comment to field instead of type definition by @pascalbreuninger in #1639
• chore(deps): bump azure/setup-helm from 3 to 4 by @dependabot in #1637
• chore: prevent distro switch from k0s by @FabianKramm in #1659
• chore: fix function name in comment by @clonefetch in #1661
• chore: validate passed in values files before passing it to helm by @johannesfrey in #1675
• chore: adjust helm version check to be more explicit by @johannesfrey in #1680
• chore: perform early exit when no values file is given for vcluster < v0.20 by @johannesfrey in #1686
• chore: rename vcluster migrate values to vcluster convert config by @johannesfrey in #1688
• chore: upgrade loft api and cli import to v4, replace pro mentions with platform by @rohantmp in #1761
• chore(config): also enable hpm when using deprecated flag by @johannesfrey in https://github.com/loft=sh/vcluster/pull/1770
• chore(vclustectl): do not print deprecated for platform cmd by @johannesfrey in #1774
• chore(vclusterctl): rename pro to platform by @johannesfrey in #1775
• chore(cli/config): unify config and make location configurable by @johannesfrey in #1798
• chore(platform): remove path knowledge from platform client by @johannesfrey in #1800
• chore(vclusterctl): drop platform import space command by @johannesfrey in #1863
• chore(vclusterctl): rename space to namespace by @johannesfrey in #1864
• chore(deps): bump braces from 3.0.2 to 3.0.3 in /docs by @dependabot in #1862
• chore: bump loft-sh dependency by @FabianKramm in #1868
• chore(deps): bump ws from 7.4.4 to 7.5.10 in /docs by @dependabot in #1872
• chore: provide space aliases for platform namespace subcommands by @johannesfrey in #1877
• chore: rename activate flag to add and move to helm flags by @johannesfrey in #1881
• chore: rename some legacy commands in info logs by @johannesfrey in #1905
• chore: document how to obtain a vcluster kubeconfig in legacy docs by @johannesfrey in #1908
• chore: bump pipeline e2e to v1.30 by @FabianKramm in #1892
• chore: bump kine version by @FabianKramm in #1894
• chore: sanitize hooks & main by @FabianKramm in #1895
• chore: upgrade to goreleaser v2 by @johannesfrey in #1854
• chore(vclusterctl): check for pre 0.20 values by @johannesfrey in #1738
• chore: bump kubernetes versions by @FabianKramm in #1891
• chore: bump loft-sh dependency by @FabianKramm in #1868
• chore(deps): bump ws from 7.4.4 to 7.5.10 in /docs by @dependabot in #1872
• chore: provide space aliases for platform namespace subcommands by @johannesfrey in #1877
• chore: rename activate flag to add and move to helm flags by @johannesfrey in #1881
• chore: rename some legacy commands in info logs by @johannesfrey in #1905
• chore: document how to obtain a vcluster kubeconfig in legacy docs by @johannesfrey in #1908
• chore: bump pipeline e2e to v1.30 by @FabianKramm in #1892
• chore: bump kine version by @FabianKramm in #1894
• chore: sanitize hooks & main by @FabianKramm in #1895
• chore: upgrade to goreleaser v2 by @johannesfrey in #1854
• chore(vclusterctl): check for pre 0.20 values by @johannesfrey in #1738
• chore: bump kubernetes versions by @FabianKramm in #1891
• chore: bump loft-sh dependency by @FabianKramm in #1868
• chore(deps): bump ws from 7.4.4 to 7.5.10 in /docs by @dependabot in #1872
• chore: provide space aliases for platform namespace subcommands by @johannesfrey in #1877
• chore: rename activate flag to add and move to helm flags by @johannesfrey in #1881
• chore: rename some legacy commands in info logs by @johannesfrey in #1905
• chore: document how to obtain a vcluster kubeconfig in legacy docs by @johannesfrey in #1908
• chore: bump pipeline e2e to v1.30 by @FabianKramm in #1892
• chore: bump kine version by @FabianKramm in #1894
• chore: sanitize hooks & main

Documentation

• docs: fix helm install command by @pdbrito in #1614
• Fix storage config docs that referenced old yaml structure by @heiko-braun in #1624
• docs: suggest required helm values for scraping kubelet targets in Isolated mode by @neogopher in #1642
• Explicitly mention that Helm must be >= v3.10.0 by @johannesfrey in #1656
• Change chart links by @johannesfrey in #1648
• Fix another chart reference by @johannesfrey in #1649
• Fix comment about default distro by @johannesfrey in #1650
• Regenerate CLI docs by @heiko-braun in #1630
• Files missing from CLI docs regen by @heiko-braun in #1631
• Update docusaurus.config.js by @LukasGentele in #1695
• Update the alpine image used to creating the override hosts file by @heiko-braun in #1835
• Update bug-report.yaml by @FabianKramm in #1890
• Update platform version to v4.0.0-alpha.18 by @heiko-braun in #1899
• Update the backup command's default ns to vcluster-platform by @zerbitx in #1901
• [bug] Fix indentation of statefulset pod annotations by @mtougeron in #1906

Refactors

• refactor: disabled -> enabled: auto & telemetry refactor by @FabianKramm in #1625
• refactor: make function replaceable by @FabianKramm in #1628
• refactor: improvements for plugins by @FabianKramm in #1629
• refactor: rename embeddedSqlite to embedded by @FabianKramm in #1640
• refactor: add concrete webhook types by @FabianKramm in #1643
• refactor: changes for vcluster-sdk by @FabianKramm in #1645
• refactor: remove special pro functionality from cli by @FabianKramm in #1665
• refactored pluginconfig by @facchettos in #1664
• refactor: add syncLabels by @FabianKramm in #1783
• refactor: allow disable csinodes & show k3s, k0s & k8s logs by default by @FabianKramm in #1700
• refactor: allow specifying resources for init container by @FabianKramm in #1705
• refactor: introduce vCluster manager by @FabianKramm in #1710
• refactor: remove need to pass cli.Config to subcommands by @rohantmp in #1813
• refactor: Readded ManagementConfig to platform client by @ThomasK33 in #1846
• refactor: rename config fields by @FabianKramm in #1884
• refactor: move vcluster plugin up in the coredns chain by @FabianKramm in #1893
• refactor: sanitize hooks & main by @FabianKramm in #1895
• refactor: integrations & apiservice by @FabianKramm in #1903
• refactor: add image.registry to configure registry separately by @FabianKramm in #1747
• refactor: throw error if using pro feature but not logged in by @FabianKramm in #1752
• refactor: add concrete webhook types by @FabianKramm in #1643
• refactor: rename embeddedSqlite to embedded by @FabianKramm in #1640
• refactor: improvements for plugins by @FabianKramm in #1629
• refactor: make function replaceable by @FabianKramm in #1628
• refactor: disabled -> enabled: auto & telemetry refactor by @FabianKramm in #1625
• refactored pluginconfig by @facchettos in #1664
• [v0.20] Merge pull request #2034 from facchettos/login-deprecate by @loft-bot in #2041

Builds

• build: add sync config workflow by @FabianKramm in #1603
• build: update sync-config.yaml by @FabianKramm in #1604
• build: update sync-config.yaml by @FabianKramm in #1605
• build: update sync-config.yaml by @FabianKramm in #1606
• build: release ghcr.io/loft-sh/vcluster-oss by @FabianKramm in #1663
• build(deps): bump golang.org/x/net from 0.20.0 to 0.23.0 by @dependabot in #1707
• build(deps): bump anchore/sbom-action from 0.15.1 to 0.15.11 by @dependabot in #1735
• build(deps): bump golangci/golangci-lint-action from 4 to 5 by @dependabot in #1734
• build(deps): bump alpine from 3.19 to 3.20 by @dependabot in #1826
• build(deps): bump anchore/sbom-action from 0.15.11 to 0.16.0 by @dependabot in #1827
• build(deps): bump golangci/golangci-lint-action from 5 to 6 by @dependabot in #1828
• build(deps): bump actions/github-script from 6 to 7 by @dependabot in #1897

License

• license(vCluster): Updated OSS licenses by @loft-bot in #1596
• license(vCluster): Updated OSS licenses by @loft-bot in #1869

New Contributors

• @eumel8 made their first contribution in #1594
• @pdbrito made their first contribution in #1614
• @joebowbeer made their first contribution in #1652
• @clonefetch made their first contribution in #1661
• @LinPr made their first contribution in #1673
• @andyluak made their first contribution in #1748
• @narcis96 made their first contribution in #1769
• @tamasborsstriva made their first contribution in #1880

loft-sh/vcluster v0.20.0 on GitHub