Metaflow 2.3.3 Release Notes

The Metaflow 2.3.3 release is a patch release.

• Features
  • Support resource tags for Metaflow's integration with AWS Batch
• Bug Fixes
  • Properly handle None as defaults for parameters for AWS Step Functions execution
  • Fix return value of IncludeFile artifacts

Features

Support resource tags for Metaflow's integration with AWS Batch

Metaflow now supports setting resource tags for AWS Batch jobs and propagating them to the underlying ECS tasks. The following tags are attached to the AWS Batch jobs now -

• metaflow.flow_name
• metaflow.run_id
• metaflow.step_name
• metaflow.user / metaflow.owner
• metaflow.version
• metaflow.production_token

To enable this feature, set the environment variable (or alternatively in the metaflow config) METAFLOW_BATCH_EMIT_TAGS to True. Keep in mind that the IAM role (MetaflowUserRole, StepFunctionsRole) submitting the jobs to AWS Batch will need to have the Batch:TagResource permission.

Bug Fixes

Properly handle None as defaults for parameters for AWS Step Functions execution

Prior to this release, a parameter specification like -

Parameter(name="test_param", type=int, default=None)

will result in an error even though the default has been specified

Flow failed:
    The value of parameter test_param is ambiguous. It does not have a default and it is not required.

This release fixes this behavior by allowing the flow to execute as it would locally.

Fix return value of IncludeFile artifacts

The IncludeFile parameter would return JSONified metadata about the file rather than the file contents when accessed through the Metaflow Client. This release fixes that behavior by returning instead the file contents, just like any other Metaflow data artifact.