Alpha release - v0.16.0a3

This release introduces distributed training support and new serialization changes along with clearer error messages and miscellaneous bug fixes. See the proposal doc, which will be updated again and finalized for the coming beta release, which we expect to make by the end of next week.

Distributed Training support

Flyte task executions have to produce outputs if the interface of the task declares so. But, in distributed training, we may run multiple instances of the same task and usually only one of them will produce an output. This needs an ability for the plugin to tell that platform that only one output is expected. This is further complicated by the fact that the primary execution that produces the output may change, for e.g., if using MPI, it may be rank0, rank1, … rankN.

Flytekit now supports an IgnoreOutputs(Exception), which is like StopIteration. If raised, it will force flytekit to exit gracefully, without writing outputs. If all nodes in the task execution raise this exception, eventually FlytePropeller will error out, as the post-condition of outputs is not met from at-least one node. For Sagemaker distributed training, the user can specify a function that indicates if output should be persisted. For all cases in which the function returns false, the outputs are ignored. The input to the function is DistributedContext. By default, for distributed training, rank0 instances are expected to produce the output.

Serialization changes

[Optional] You now no longer need to enter your flyte project container in order to serialize entities during the registration process.

pyflyte -c flyte.config  --pkgs recipes serialize --in-container-config-path /root/flyte.config  \
--local-source-root ${CURDIR}  --image somedocker.com/myimage:someversion123 workflows -f _pb_output/

After using out of container serialize, proceed to register-files like you previously have, using flyte-cli and also optionally outside of your container.

Fast serialization

You can use out of container fast serialize to skip having to build a docker container between code changes as you iterate on your workflows. As long as your changes do not alter dependencies or otherwise require rebuilding your docker container you can use the following:

pyflyte -c flyte.config  --pkgs recipes serialize --in-container-config-path /root/flyte.config  \
--local-source-root `pwd`  --image somedocker.com/myimage:someversion123 fast workflows -f _pb_output/

And fast-register using:

flyte-cli fast-register-files -p <project> -d <domain> -h flyte.example.com \
--additional-distribution-dir s3://my-s3-bucket/fast/ --dest-dir /root/recipes _pb_output/*