[Beta] - v0.16.0b0

This is the first beta release of the Python native-typing API. Please see the earlier alpha releases a0, a1, a2, and a3 as well as the proposal doc which has been updated to reflect the latest design choices made during implementation. It'll be finalized as we move towards a final non-beta release. This is a pretty small release in terms of features, as the team has been mostly working on adding examples and cleaning up documentation (please see the release to the cookbook repo we'll be making shortly).

• The main change this release was a new plugins structure. Previously, plugin-provided functionality like Spark and Sagemaker tasks and Jupyter notebook integration through papermill were all intermingled through the rest of the flytekit codebase (in the taskplugins folder for the new API, and in a variety places in the old API). With this release, plugin development has been moved to different top-level folder, plugins). Contributors should think about this as a separate GitHub repo as that will likely be the case in the not-too-distant future. Please see the PR implementing this for more background, discussion, and documentation. Basically each plugin will have its own Pypi package (Spark or papermill for example).
  • Users of the new (Python native-typing API) should update their requirements files to pick up these new dependencies via the explicit Pypi name, (e.g. pip install flytekitplugins-spark). The complete list of migrated plugins is currently
    • https://pypi.org/manage/project/flytekitplugins-spark
    • https://pypi.org/project/flytekitplugins-awssagemaker/0.1.0/
    • https://pypi.org/project/flytekitplugins-hive/0.1.0/
    • https://pypi.org/project/flytekitplugins-pod/0.1.0/
    • https://pypi.org/project/flytekitplugins-kfpytorch/0.1.0/
    • https://pypi.org/project/flytekitplugins-kftensorflow/0.1.0/
    • https://pypi.org/project/flytekitplugins-papermill/0.1.0/
  • If you pip install flytekit==0.16.0b0[spark] you will still get some Spark related dependencies but these will install the old-style dependencies. Please avoid this.
• Because FlyteSchema functionality is so core to the Flyte experience, we've added pandas and pyarrow as requirements to flytekit. We were debating making this an explicit flytekitplugins plugin but the ubiquity of FlyteSchemas felt incompatible with an explicit requirement.
• A small change was introduced that tells plugins where to find the flytekit entrypoint in a container. This is currently only used in the Spark plugin implementation but we thought other plugins may need it in the future as well.