longhorn/longhorn v1.3.0-preview1

Longhorn v1.3.0-preview1 release

on GitHub

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.0-preview1 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below including stability, performance, availability, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Longhorn supports 3 installation ways including Rancher catalog, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

The live upgrade from v1.2.* is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

• For existing features, there are no deprecations or incompatible changes introduced in this release.
• v1beta2 resource version is introduced with validating and mutating admission webhooks. v1beta2 will be the default resource version.
• This release has a built-in CRD conversion webhook for v1beta1 compatibility. After upgrading from a previously supported version, all v1beta1 resources will be converted to v1beta2 but v1beta1 resources are still accessible by the conversion webhook.

Highlights

• [FEATURE] Clean up orphaned unused volume replicas after node failure (685) - @yangchiu @derekbit
• [REFACTOR] Longhorn CRD structural schema (791) - @kaxing @jenting
• [FEATURE] Longhorn with managed Kubernetes cluster(EKS, GKE, AKS) (1800) - @kaxing @c3y1huang
• [FEATURE] Longhorn should support multi-network K8s clusters (storage network) (2285) - @c3y1huang
• [FEATURE] Extend CSI Snapshot support to Longhorn snapshot (2534) - @PhanLe1010 @cchien816
• [POC] Rebuild logical volumes/replicas with SPDK (3044) - @keithalucas
• [FEATURE] CRD for snapshots (3144) - @PhanLe1010
• [FEATURE] Allowing users to download backing images (3155) - @shuo-wu
• [FEATURE] CRD version webhook for longhorn.io/v1beta1 and longhorn.io/v1beta2 (3265) - @kaxing @derekbit @jenting
• [FEATURE] Builds for IBM Z (3425) - @Anarkis
• [FEATURE] Support operations on custom resources directly instead of Longhorn REST APIs only (3562) - @kaxing @derekbit
• [FEATURE] mTLS authentication for manager <-> instance-manager communication (3839) - @joshimoo

Enhancements

• [FEATURE] Include a NetworkPolicy to secure communication inside of Longhorn (1805) - @kaxing @PhanLe1010
• [BUG] Remove hard-coded namespace in uninstall controller (2034) - @c3y1huang
• [FEATURE] Allow use of existing private registry secret (2157) - @yangchiu @c3y1huang
• [FEATURE] instance-manager compatibility with Cluster Autoscaler (2203) - @kaxing @c3y1huang
• [FEATURE] Recurring snapshot/backup job supports imagePullSecrets (2432) - @kaxing @jenting
• [BUG] helm upgrade won't apply customize default setting (2570) - @kaxing @shuo-wu
• [IMPROVEMENT] Support Filesystem expansion for encrypted volumes (2868) - @joshimoo @khushboo-rancher
• [FEATURE] podAntiAffinity for csi-attacher/csi-provisioner/csi-resizer/csi-snapshotter (2894) - @kaxing @derekbit
• [FEATURE] Introduce SUSE BCI based component images (2939) - @meldafrawi
• [FEATURE] Prometheus metrics for backups (2940) - @meldafrawi
• [FEATURE] Match type of status.conditions of nodes.longhorn.io to other k8s objects (2955) - @derekbit @khushboo-rancher
• [TASK] Publish Longhorn stable releases (3014) - @PhanLe1010 @smallteeths @cchien816
• [FEATURE] Mutating/Validating admission webhook (3241) - @kaxing @jenting
• [FEATURE] Expect LH has a lightweight solution for volume failure like "insufficient storage" (3734) - @PhanLe1010
• [FEATURE] Longhorn UI supports orphan CRs management (3826) - @yangchiu @derekbit @smallteeths
• [FEATURE] Make Longhorn UI Listen Binding configurable (3829) - @yangchiu @smallteeths

Improvement

• [IMPROVEMENT] refactor engine-image-controller to use structured logging (1726) - @jenting
• [IMPROVEMENT] Change the lifecycle management of the instance-manager pods. (2650) - @kaxing @shuo-wu
• [IMPROVEMENT] Pre-condition check required for the snapshot purge (2777) - @derekbit
• [IMPROVEMENT] move filesystem expansion into the csi driver (2794) - @joshimoo @khushboo-rancher
• [IMPROVEMENT] Improve communication between components (GRPC) (2821) - @joshimoo
• [IMPROVEMENT] Simplify the way to record and fetch backup info after introducing Backup CRD (3035) - @jenting @cchien816
• [IMPROVEMENT] Upgrade path improvement (3087) - @kaxing @shuo-wu
• [IMPROVEMENT] UI should inform users that backups created by a activated DR volume may reside in the original backup volume (3121) - @kaxing @smallteeths
• [IMPROVEMENT] Have backup progress icon on Backup page (3168) - @smallteeths @khushboo-rancher
• [IMPROVEMENT] Make GI as default value in volume expand screen (3217) - @kaxing @smallteeths
• [IMPROVEMENT] Have UI support to identify non-default instance manager image of each volume in the volume list (3306) - @kaxing @smallteeths
• [IMPROVEMENT] longhorn protocol buffer namespace conflict (3343) - @kaxing @jenting
• [IMPROVEMENT] Add network policy for longhorn webhook server (3513) - @kaxing @jenting
• [IMPROVEMENT] The longhorn manager should wait the longhorn webhook server ready (3523) - @kaxing @jenting
• [IMPROVEMENT] Add default replica auto balance to disabled for the existing volume (3524) - @derekbit @khushboo-rancher
• [IMPROVEMENT] Loading http requests on demand (3528) - @kaxing @smallteeths
• [IMPROVEMENT] Prevent PVC provisioning if there is insufficent storage (3529) - @yangchiu @derekbit
• [IMPROVEMENT] Implement gRPC proxy in IM to replace MGR engine binary invocation (3546) - @c3y1huang
• [IMPROVEMENT] set default non-root user for all longhorn component images (3549) - @jenting
• [IMPROVEMENT] Improve the environment check script to check for dependences (3550) - @kaxing @derekbit @cchien816
• [IMPROVEMENT] Improve snapshot purge (3613) - @yangchiu @shuo-wu
• [IMPROVEMENT] Bump Longhorn k8s dependency library to a newer version (3720) - @PhanLe1010
• [IMPROVEMENT] Disable snapshot revert and Allow snapshot removal when the snapshot marked as Removed (3748) - @shuo-wu @cchien816
• [IMPROVEMENT] Adjust volume page UI to make better use of screen space (3832) - @smallteeths
• [IMPROVEMENT] Add doc for the Orphan replica directory and the following improvement (3853) - @derekbit
• [IMPROVEMENT] Make storage parameter "migratable" available in the helm chart (3880) - @ibrokethecloud @cchien816

Performance

• [BUG] Backup created in prior release takes considerable time to show up on the backup page first time. (2902) - @kaxing @jenting @khushboo-rancher
• [BUG] Scalability issue of volumes in Longhorn (3083) - @kaxing @PhanLe1010
• [BUG] Volume attachment take long time, which may be caused by the replica controller queue being flooded by the backing image events (3242) - @kaxing @shuo-wu
• [BUG] Instance Manager Memory Leak (3545) - @meldafrawi @jenting
• [BUG] High CPU usage by instance manager (3636) - @keithalucas @meldafrawi

Stability

• [BUG] Unlimited volume replica rebuilding causes system unstable like latency of etcd access (2460) - @shuo-wu @khushboo-rancher

Bugs

• [BUG] Validate S3 input to make sure they don't contain trailing newline or space (811) - @kaxing @jenting
• [BUG] Error message is not seen on the UI when the backup to minio fails (1268) - @kaxing @jenting
• [BUG] Failed backups should be available to the user with a failed status (1311) - @jenting @khushboo-rancher
• [BUG] Name of PVC overlaps on Firefox (1598) - @smallteeths @cchien816
• [BUG] Backup Target - NFS - the k8s node supports only 4.0 - an incorrect mount option was specified (2438) - @kaxing @derekbit
• [BUG] Not all informer registered cacheSync function (2671) - @jenting @khushboo-rancher
• [BUG] Volume operations take long time during automatic upgrading the engines in a big cluster (2697) - @kaxing @PhanLe1010
• [BUG] allow volume migration when volume is degraded (harvester vm) (2805) - @shuo-wu @khushboo-rancher
• [BUG] Instance managers and Pods with attached volumes restarted every hour (2820) - @meldafrawi @PhanLe1010
• [BUG] Track terminating pods in the volume workload status (2874) - @joshimoo @khushboo-rancher
• [BUG] NFS driver: creates multiple 1st volume backup and immediately list backup volume names sometimes fails (2926) - @kaxing @jenting
• [BUG] Deleting the only pod in a single replica ReplicaSet causes an attachment failure of the replacement pods when the volume is RWX. (2946) - @joshimoo @khushboo-rancher
• [BUG] Over-provisioning doesn't work properly with values smaller than 200% (2952) - @meldafrawi @PhanLe1010
• [BUG] Backups don't work after upgrading 1.1.2 to 1.2.0 (2960) - @jenting @cchien816
• [BUG] Error when upgrading from 1.1.2 to 1.2.0 - Operation cannot be fulfilled on volumes.longhorn.io "pvc-edf41777-589d-4806-baca-b91d0a6c0d3c": the object has been modified; please apply your changes to the latest version and try again (2961) - @kaxing @c3y1huang
• [BUG] Loghorn volumes can be resized to any size and does not respect the over-provisioning limit (2962) - @meldafrawi @PhanLe1010
• [BUG] NodePublishVolume RWX CSI realpath failed to resolve symbolic links on microk8s (2987) - @kaxing @joshimoo
• [BUG] RWX prior existing PV/PVC that were marked as XFS will now fail to mount, since we only supported EXT4 previously so we ignored users choice (2991) - @joshimoo @khushboo-rancher
• [BUG] Tool tip on Snapshots and Backups List on UI are displayed inconsistently (2994) - @smallteeths @cchien816
• [BUG] Disk eviction not doing anything (2995) - @kaxing @shuo-wu
• [BUG] Recurring jobs showing incorrectly (2996) - @kaxing @smallteeths
• [BUG] No any implication for backup being pulling from backupstore (3023) - @kaxing @smallteeths
• [BUG] Snapshots and Backups chain displays backup that was previously deleted on Backup screen (3033) - @jenting @cchien816
• [BUG] The backing image file reuse will fail forever if there is an invalid file in the work directory (3061) - @kaxing @shuo-wu
• [BUG] Longhorn 1.2.0 unable to handle instance-manager failure (3070) - @PhanLe1010 @cchien816
• [BUG] Cannot migrate replicas in an evicting disk to a new disk on the same node when the node level hard affinity is applied. (3076) - @meldafrawi @PhanLe1010
• [BUG] fatal error: concurrent map writes (3077) - @derekbit @cchien816
• [BUG] Update the default advertised CSI version to csi version 1.2 (3079) - @PhanLe1010 @khushboo-rancher
• [BUG] Restoring backup always point to the latest backup (3111) - @kaxing @joshimoo @PhanLe1010 @shuo-wu
• [BUG] fsfreeze race condition (3125) - @joshimoo @khushboo-rancher
• [BUG] Backing Image upload failed when access through Rancher UI (3129) - @kaxing @smallteeths
• [BUG] Volume creating with Backing Image contains invalid checksum string will result in attaching problem (3133) - @kaxing @derekbit
• [BUG] Support bundle missed newly added Longhorn CRDs (3142) - @jenting @khushboo-rancher
• [BUG] The size of an uploaded or downloaded sparse file is the apparent file size (3174) - @kaxing @shuo-wu
• [BUG] cron setting won't persist (3180) - @smallteeths @khushboo-rancher
• [BUG] Volume API doesn't have the workloadStatus updated (3182) - @jenting
• [BUG] Restored volume show 'scheduled' with 2 replicas running and 1 unscheduled replica (3183) - @c3y1huang @cchien816
• [BUG] Longhorn UI: XHR request URL sometimes wrong when UI is proxied (3191) - @kaxing @PhanLe1010 @smallteeths
• [BUG] rwx volume fail to attach (3193) - @shuo-wu @khushboo-rancher
• [BUG] Volume crash during migrating (3206) - @shuo-wu @khushboo-rancher
• [BUG] Restored volume wrongly restarts restoration during migration (3210) - @meldafrawi @shuo-wu
• [BUG] Tests related to backup failed on nightly test run (3216) - @jenting @khushboo-rancher
• [BUG] Tooltip of a backup taken shows 0 size if a snapshot is taken just before it (3228) - @kaxing @jenting
• [BUG] longhorn-ui has a warning message in the browser's console (3230) - @smallteeths @khushboo-rancher
• [BUG] Instance process is still running when the corresponding engine/replica CR is gone (3255) - @shuo-wu
• [BUG] YAML in Recurring Snapshots and Backups page is out of format (3256) - @c3y1huang
• [BUG] can't reset Backup Target Credential Secret (3261) - @meldafrawi @jenting
• [BUG] Volume with robustness is unknown is not reflect under Volume page (3291) - @kaxing @smallteeths
• [BUG] Revision Counter is false by default? (3308) - @smallteeths @khushboo-rancher
• [BUG] Test related to backup is failing due to the state changed to 'completed' from 'complete' (3309) - @jenting
• [BUG] Drain stuck on K3s setup on node where RWX volume is attached (3348) - @shuo-wu
• [BUG] Data loss on doing K3s upgrade with drain (3350) - @PhanLe1010
• [BUG] Potential bugs inside Longhorn CSI related to the CSI snapshot flow (3378) - @PhanLe1010 @khushboo-rancher
• [BUG] Longhorn CSI prints out many error logs during the CSI backup (3392) - @PhanLe1010 @khushboo-rancher
• [BUG] Show snapshot in different color when backup fails (3394) - @smallteeths
• [BUG] Error inspecting backup config (3402) - @jenting @khushboo-rancher
• [BUG] Can't delete engineimage after upgrade from v1.2.2 to master-head (Dec 15, 2021) (3407) - @derekbit @jenting
• [BUG] cache.DeletedFinalStateUnknown conversion uses pointer type (3417) - @joshimoo
• [BUG] Upgrade failed through Rancher (2.5.5) UI from 1.2.0 -> 1.2.2 (3424) -
• [BUG] Failed to upgrade from v1.2.3 with volumes uses backing image to master branch (3426) - @kaxing @derekbit
• [BUG] Failed to upgrade helm deployment due to hardcoded namespace (v1.2.3) (3433) - @kaxing @jenting
• [BUG] Instance manager pods are recreated immediately when the node draining evicts the old pods (3459) - @shuo-wu @khushboo-rancher
• [BUG] Instance manager crashes with invalid memory address or nil pointer dereference (3466) - @shuo-wu @khushboo-rancher
• [BUG] Recurring backup not work on detached volume (master-head) (3470) - @c3y1huang @khushboo-rancher @cchien816
• [BUG] Longhorn manager NPE due to the nil engine during the backup creation (3484) - @kaxing @shuo-wu
• [BUG] RecurringJob should validate retain field is <= 50 like the UI does (3493) - @kaxing @jenting
• Volume cloning feature is broken due to CRDs schema in master-head 01/06/2022 (3508) - @PhanLe1010 @khushboo-rancher
• [Doc] Update the doc with the info encrypted filesystem does NOT need manual expansion (3511) - @khushboo-rancher
• [BUG] Recurring Job delete not working correctly (3517) - @c3y1huang @khushboo-rancher
• [BUG] Error while saving Node Disk setting when there is no tag (3538) - @kaxing @derekbit
• [BUG] test_setting_priority_class fails on master-head with error (3548) - @derekbit
• [BUG] Capitalization on the website for the blog and KB should be consistent (3573) - @derekbit
• [BUG] Backup in error state can not be deleted (3620) - @derekbit @cchien816
• [BUG] Volume fails to scheduled if while creating it couldn't be scheduled. (3626) - @PhanLe1010
• [BUG] Unable to finish install or upgrade when Failed to list *v1beta2.Node: v1beta2.NodeList.Items (3631) - @kaxing @derekbit
• [BUG] Statefulset pod get data input/output error after delete instance manager (3648) -
• [BUG] Replica auto-balance tests fails in regression (3649) - @c3y1huang @cchien816
• [BUG] longhorn-ui crashlooping when running RKE2 with cis-1.5 profile (3658) - @smallteeths @khushboo-rancher
• [BUG] Upgrade from v1.2.x to master-head fails if backupTarget exists (3677) - @yangchiu @derekbit
• [BUG] Volume not become scheduling after change over provisioning form 0 to 100 (3678) - @PhanLe1010 @cchien816
• [BUG] Volume fails to schedule when set Eviction Requested from true to false on node which contains replica (3679) - @kaxing @PhanLe1010
• [BUG] When Harvester delete an uploading image, the corresponding LH temp file is not deleted (3682) - @shuo-wu
• [BUG] Longhorn webhooks fail to deploy on hardened cluster (3701) - @kaxing @derekbit
• [BUG] longhornio/longhorn-manager:master-head currently CrashLoopBackOff-ing (3716) -
• [BUG] Data migration sample YAML moves data to the subfolder (3787) - @yangchiu @c3y1huang
• [BUG] go.mod and go.sum out of date (3796) - @derekbit
• [BUG] 405 status code show up twice when Create PV/PVC for new volume via Rancher Proxy (3807) - @kaxing @smallteeths
• [BUG] Volume can't detach when "Concurrent Replica Rebuild Per Node Limit = 0" and set node evicted from True to False (3820) - @PhanLe1010
• [BUG] Longhorn default StorageClass is not always synced with the ConfigMap (3821) - @yangchiu @shuo-wu
• [BUG] longhorn-engine controller rebuilding crash (3822) - @keithalucas
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @shuo-wu
• [BUG] BackingImageManager version request leaked grpc connection (3838) - @kaxing @derekbit
• [BUG] PVC create from volumsanpshot stuck at pending status(type=snap) (3860) - @kaxing @derekbit
• [BUG] liblonghorn sends integer fields in longhorn protocol as host endianness and longhorn-engine always expects litte endian (3875) - @keithalucas
• [BUG] A snapshot has a child that is no-longer exist in the snapshot chain (3883) - @PhanLe1010 @cchien816
• [BUG] Volume created from cloned PVC will stuck at Not Ready status (3886) -
• [BUG] UI pop error if long volume name used (when > 40 characters) (3915) - @derekbit @cchien816
• [BUG] Instance manager keep restarting (3922) - @joshimoo @cchien816
• [BUG] Websocket's icon in footer is showing wrong (3935) - @smallteeths

Misc

• [BUG] After upgrade to 1.1.0 some old managers still running (2144) - @derekbit
• [DOC] Change VolumeSnapshotClass API version to v1 from v1beta1 in documentation (2214) -
• [REFACTOR] Remove the keyword register of deepcopy-gen (2637) - @jenting @khushboo-rancher
• [TASK] Deprecate volume spec recurringJobs and storageClass recurringJobs field (2865) - @c3y1huang
• [TASK] Update document to reflect on Volume Attachment Recovery Policy feature deprecation (2954) - @jenting
• [TEST] Add test case in e2e integration test for fsgroup support (2967) - @khushboo-rancher
• [DOC] snapshot-controller 4.0 is not able to read VolumeSnapshotClass (2973) - @jenting
• [UI] Notify users about newer stable versions (3032) - @smallteeths @cchien816
• [TASK] Extend OS support matrix (3085) - @meldafrawi
• [TASK] Remove the feature that skip taking backup/snapshot when the volume head is empty in recurring backup jobs (3128) - @derekbit @khushboo-rancher
• [TASK] Change the CSI sidecar images from upstream repo to Longhorn repo (3258) - @PhanLe1010 @khushboo-rancher
• [BUG] Test_settings fails in the integration test (3280) - @jenting
• [TASK] Generate multiple CRD versions (3295) - @kaxing @derekbit @jenting
• [TASK] The CSI resizer may not work when the Job pod status is Completed (3303) - @kaxing @shuo-wu
• Failed auto installation of iscsi on Debian 11 bullseye (3313) - @derekbit
• [TASK] Handle the CRD validation error (3352) - @kaxing @derekbit
• [TASK] Restore to a brand new cluster that does not contain anything (3367) - @shuo-wu @khushboo-rancher
• [DOC] Document method to easily update Longhorn settings in scripted fashion/across multiple clusters (3578) - @yangchiu @jenting
• [TASK] Create golang API for mounting NVMeoF targets (3602) - @keithalucas
• [TASK] Improve the doc for volume actual size and space usage (3627) - @derekbit @shuo-wu
• [DOC] Update RHEL instructions for open-iscsi (3630) - @keithalucas
• [TASK] Need warning message on UI when deleting the last recurring job of the group (3651) - @yangchiu @smallteeths
• [TASK] Create a migration path from deprecated Rancher catalog to Rancher marketplace (3714) - @shuo-wu
• [DOC] Alert LonghornNodeDown example in documentation doesn't work (3729) - @kaxing @jenting
• [TASK] Add readme document for the admission webhook (3752) - @derekbit @cchien816
• [TASK] Collect and analyze reasons for unstable k8s cluster when running longhorn-tests on Jenkins (3770) - @yangchiu
• [TASK] Refactor the Logic for Node Monitoring (3775) - @derekbit
• [TASK] Move secscan nightly tests to aws (3779) - @meldafrawi
• [TASK] Improve the doc for troubleshooting NFS backuptarget (3825) - @derekbit
• [DOC] Add cluster autoscaler support (3884) - @c3y1huang
• [TASK] Update the upgrade doc about expected laborious upgrade due to mass backup CRs (3890) - @PhanLe1010 @cchien816
• [TASK] Bump K8s min version support, CSI sidecar versions, dependent lib version (3891) - @joshimoo
• [TASK] Update the volume size related doc after introducing the snapshot prune feature (3894) - @shuo-wu
• [TASK] add finalizer to resource in mutating webhook (3918) - @derekbit
• [TASK] Release 1.3.0 (3938) - @innobead

Contributors

• @Anarkis
• @PhanLe1010
• @c3y1huang
• @cchien816
• @derekbit
• @ibrokethecloud
• @innobead
• @jenting
• @joshimoo
• @kaxing
• @keithalucas
• @khushboo-rancher
• @meldafrawi
• @shuo-wu
• @smallteeths
• @yangchiu