Features

Configurable default Kubernetes resources

This release introduces configuration options for setting default values for cpu / memory / disk when running on Kubernetes. These can be set either with environment variables

METAFLOW_KUBERNETES_CPU=
METAFLOW_KUBERNETES_MEMORY=
METAFLOW_KUBERNETES_DISK=

or in a Metaflow profile

{
  "KUBERNETES_CPU": "",
  "KUBERNETES_MEMORY": "",
  "KUBERNETES_DISK": "",
}

These values will be overruled by specifying a value through the @kubernetes or @resources decorators.

Improvements

Support for wider foreach flows with Argo Workflows

This release changes the way task ids are generated on Argo Workflows in order to solve an issue where extremely wide foreach splits could not execute correctly due to hard limits on input parameters size on Argo Workflows.

What's Changed

fix: Deterministic foreach task id's for Argo Workflows by @saikonen in #1704
Make Kubernetes default resources configurable by @martinhausio in #1800
Add METAFLOW_ESCAPE_HATCH_WARNING for s3op by default. by @romain-intel in #1825
bump version to 2.11.13 by @saikonen in #1826

New Contributors

@martinhausio made their first contribution in #1800

Full Changelog: 2.11.12...2.11.13

Netflix/metaflow 2.11.13 on GitHub