openebs/openebs v2.6.0

on GitHub

Release Summary

OpenEBS v2.6 contains some key enhancements and several fixes for the issues reported by the user community across all 9 types of OpenEBS volumes.

• 3 replicated types - Mayastor, cStor and Jiva.
• 5 types of Local PV backed by host path, device, ZFS, LVM and Rawfile, and
• RWX volumes using NFS.

Here are some of the key highlights in this release.

New capabilities

• OpenEBS is introducing a new CSI driver for dynamic provisioning of Jiva volumes. This driver is released as alpha and currently supports the following additional features compared to the non-CSI jiva volumes.

  • Jiva Replicas are backed by OpenEBS host path volumes
  • Auto-remount of volumes that are marked read-only by iSCSI client due to intermittent network issues
  • Handle the case of multi-attach error sometimes seen on on-premise clusters
  • A custom resource for Jiva volumes to help with easy access to the volume status

  For instructions on how to set up and use the Jiva CSI driver, please see. https://github.com/openebs/jiva-operator.

Key Improvements

• Several bug fixes to the Mayastor volumes along with improvements to the API documentation. See Mayastor release notes.
• Enhanced the NFS Dynamic Provisioner to support using Cluster IP for dynamically provisioned NFS server. It was observed that on some of the Kubernetes clusters the kubelet or the node trying to mount the NFS volume was unable to resolve the cluster local service.
• ZFS Local PV added support for resizing of the raw block volumes.
• LVM Local PV is enhanced with additional features and some key bug fixes like:
  • Raw block volume support
  • Snapshot support
  • Ability to schedule based on the capacity of the volumes provisioned
  • Ensure that LVM volume creation and deletion functions are idempotent
• NDM partition discovery was updated to fetch the device details from its parent block device.

Key Bug Fixes

• Fixed an issue with cStor helm chart w.r.t to CVC operator service that was causing velero restores to fail. (openebs/cstor-operators#239)
• Fixed an issue with cStor CSPC validation causing failures in creating pools with ZFS supported compression algorithms. (openebs/cstor-operators#237)
• Fixed an issue with the cStor validation webhook not getting deleted in K8s 1.20+ when the OpenEBS namespace is deleted. #3338)
• Fixed an issue with Jiva replicas attempting duplicate registration if the Pod IP of the replica changes. (#3323)
• Fixed an issue where Jiva cleanup jobs can't be executed when cluster PSP is restrictive. (#3331)
• Fixed an issue where cStor pool migration from SPC to CSPC could result in pool failure when migration takes time and the user tries to restart the SPC pool. (openebs/upgrade#82)
• Fixed an issue with NDM partition discovery causing errors when partitions were present on top of filtered devices like OS disk. (#3321)

Backward Incompatibilities

• Kubernetes 1.17 or higher release is recommended as this release contains the following updates that will not be compatible with older Kubernetes releases.

  • The CSI components have been upgraded to:
    • k8s.gcr.io/sig-storage/csi-attacher:v3.1.0
    • k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0
    • k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0
    • k8s.gcr.io/sig-storage/csi-resizer:v1.1.0
    • k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    • k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
    • k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes)
    • k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes)

• If you are upgrading from an older version of cStor operators to this version, you will need to manually delete the cStor CSI driver object prior to upgrading. kubectl delete csidriver cstor.csi.openebs.io. For complete details on how to upgrade your cStor operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools.

• The CRD API version has been updated for the cStor custom resources to v1. If you are upgrading via the helm chart, you might have to make sure that the new CRDs are updated. https://github.com/openebs/cstor-operators/tree/master/deploy/helm/charts/crds

• The e2e pipelines include upgrade testing only from 1.5 and higher releases to 2.6. If you are running on release older than 1.5, OpenEBS recommends you upgrade to the latest version as soon as possible.

Other notable updates

• OpenEBS has applied for becoming a CNCF incubation project and is currently undergoing a Storage SIG review of the project and addressing the review comment provided. One of the significant efforts we are taking in this direction is to upstream the changes done in uZFS to OpenZFS.
• Automation of further Day 2 operations like - automatically detecting a node deletion from the cluster, and re-balancing the volume replicas onto the next available node.
• Migrating the CI pipelines from Travis to GitHub actions.
• Several enhancements to the cStor Operators documentation with a lot of help from @survivant.
• PSP support has been added to ZFS Local PV and cStor helm charts.
• Improving the OpenEBS Rawfile Local PV in preparation for its beta release. In the current release, fixed some issues as well as added support for setting resource limits on the sidecar and few other optimizations.
• Sample Grafana dashboards for managing OpenEBS are being developed here: https://github.com/openebs/charts/tree/gh-pages/grafana-charts

Show your Support

Thank you @coboluxx (IDNT) for becoming a public reference and supporter of OpenEBS by sharing your use case on ADOPTERS.md

Are you using or evaluating OpenEBS? You can help OpenEBS in its journey towards becoming a CNCF Incubation project by sharing your OpenEBS story and join the league of OpenEBS Adopters.

Shout outs!

MANY THANKS to our existing contributors and for everyone helping OpenEBS Community going.

A very special thanks to our first-time contributors to code, tests, and docs: @luizcarlosfaria, @z0marlin, @iyashu, @dyasny, @hanieh-m, @si458, @Ab-hishek

Getting Started

Prerequisite to install

• Kubernetes 1.17+ or newer release is installed.
• Make sure that you run the below installation steps with the cluster-admin context. The installation will involve creating a new Service Account and assigning it to OpenEBS components.
• Make sure iSCSI Initiator is installed on the Kubernetes nodes.
• Node-Disk-Manager (NDM) helps in discovering the devices attached to Kubernetes nodes, which can be used to create storage pools. If you like to exclude some of the disks from getting discovered, update the filters in NDM Config Map to exclude paths before installing OpenEBS.
• NDM runs as a privileged pod since it needs to access the device information. Please make the necessary changes to grant access to run in privileged mode. For example, when running in RHEL/CentOS, you may need to set the security context appropriately. Refer Configuring OpenEBS with selinux=on

Install using kubectl

kubectl apply -f https://openebs.github.io/charts/2.6.0/openebs-operator.yaml

Install using Helm stable charts

helm repo add openebs https://openebs.github.io/charts
helm repo update
helm install --namespace openebs --name openebs openebs/openebs --version 2.6.0

For more details refer to the documentation at https://docs.openebs.io/

Upgrade

Upgrade to 2.6 is supported only from 1.0 or higher and follows a similar process as earlier releases. Detailed steps are provided here.

• Upgrade OpenEBS Control Plane components.
• Upgrade Jiva PVs to 2.6, either one at a time or multiple volumes.
• Upgrade CStor Pools to 2.6 and its associated Volumes, either one at a time or multiple volumes.

For upgrading from releases prior to 1.0, please refer to the respective release upgrade here.

Support

If you are having issues in setting up or upgrade, you can contact:

• OpenEBS Community for Support on Kubernetes Slack
  • Already signed up? Head to our user discussions at #openebs
  • Would you like to contribute to the OpenEBS project? Head to our contributor discussions at #openebs-dev
• Raise an issue
• Subscribe and reach out on our OpenEBS CNCF Mailing lists
  • For OpenEBS project updates, subscribe to OpenEBS Announcements
  • For interacting with other OpenEBS users, subscribe to OpenEBS Users
• Join Community Meetings

Major Limitations and Notes

For a more comprehensive list of open issues uncovered through e2e and community testing, please refer to open issues. If you are using the cStor Storage Engine, please review the following before upgrading to this release.

• The recommended approach for deploying cStor Pools is using the new custom resource called cStorPoolCluster (CSPC). Even though the provisioning of cStor Pools using StoragePoolClaim(SPC) is supported, it will be deprecated in future releases. The pools provisioned using SPC can be easily migrated to CSPC.
• When using cStor Pools, make sure that raw block devices are available on the nodes. If the block devices are formatted with a filesystem or mounted, then cStor Pool will not be created on the block device. In the current release, there are manual steps that could be followed to clear the filesystem or use partitions for creating cStor Pools, please reach out to the community (#openebs) at https://slack.k8s.io.
• If you are using cStor pools with ephemeral devices, starting with 1.2 - upon node restart, cStor Pool will not be automatically re-created on the new devices. This check has been put to make sure that nodes are not accidentally restarted with new disks. The steps to recover from such a situation are provided here, which involves changing the status of the corresponding CSP to Init.
• Capacity over-provisioning is enabled by default on the cStor pools. If you don’t have alerts set up for monitoring the usage of the cStor pools, the pools can be fully utilized and the volumes can get into a read-only state. To avoid this, set up resource quotas as described in #2855.