• Features
  • Introduce support for persistent volume claims for executions on Kubernetes

Features

Introduce support for persistent volume claims for executions on Kubernetes

With this release, Metaflow users can attach existing persistent volume claims to Metaflow tasks running on a Kubernetes cluster.

To use this functionality, simply list your persistent volume claim and mount point using the persistent_volume_claims arg in @kubernetes decorator - @kubernetes(persistent_volume_claims={"pvc-claim-name": "mount-point", "another-pvc-claim-name": "another-mount-point"}).

Here is an example:

from metaflow import FlowSpec, step, kubernetes, current
import os

class MountPVCFlow(FlowSpec):

    @kubernetes(persistent_volume_claims={"test-pvc-feature-claim": "/mnt/testvol"})
    @step
    def start(self):
        print('testing PVC')
        mount = "/mnt/testvol"
        file = f"zeros_run_{current.run_id}"
        with open(os.path.join(mount, file), "w+") as f:
            f.write("\0" * 50)
            f.flush()
        
        print(f"mount folder contents: {os.listdir(mount)}")
        self.next(self.end)

    @step
    def end(self):
        print("finished")

if __name__=="__main__":
    MountPVCFlow()

In case you need any assistance or have feedback for us, ping us at chat.metaflow.org or open a GitHub issue.

What's Changed

• handle bools properly for argo-workflows task runtime cli by @savingoyal in #1395
• fix: migrate R support to use importlib by @saikonen in #1396
• Add configuration of username from metaflow_config.py by @tfurmston in #1400
• feature: add Kubernetes support for PVC mounts by @saikonen in #1402
• Update version to 2.8.6 by @savingoyal in #1404

Full Changelog: 2.8.5...2.8.6