prefect-aws 0.5.0

Open Source Launch!

on Python PyPI

0.5.0

Released March 24, 2019

Features

• Add checkpoint option for individual Tasks, as well as a global checkpoint config setting for storing the results of Tasks using their result handlers - #649
• Add defaults_from_attrs decorator to easily construct Tasks whose attributes serve as defaults for Task.run - #293
• Environments follow new hierarchy (PIN-3) - #670
• Add OneTimeSchedule for one-time execution at a specified time - #680
• flow.run is now a blocking call which will run the Flow, on its schedule, and execute full state-based execution (including retries) - #690
• Pre-populate prefect.context with various formatted date strings during execution - #704
• Add ability to overwrite task attributes such as "name" when calling tasks in the functional API - #717
• Release Prefect Core under the Apache 2.0 license - #762

Enhancements

• Refactor all State objects to store fully hydrated Result objects which track information about how results should be handled - #612, #616
• Add google.cloud.storage as an optional extra requirement so that the GCSResultHandler can be exposed better - #626
• Add a start_time check for Scheduled flow runs, similar to the one for Task runs - #605
• Project names can now be specified for deployments instead of IDs - #633
• Add a createProject mutation function to the client - #633
• Add timestamp to auto-generated API docs footer - #639
• Refactor Result interface into Result and SafeResult - #649
• The manual_only trigger will pass if resume=True is found in context, which indicates that a Resume state was passed - #664
• Added DockerOnKubernetes environment (PIN-3) - #670
• Added Prefect docker image (PIN-3) - #670
• defaults_from_attrs now accepts a splatted list of arguments - #676
• Add retry functionality to flow.run(on_schedule=True) for local execution - #680
• Add helper_fns keyword to ShellTask for pre-populating helper functions to commands - #681
• Convert a few DEBUG level logs to INFO level logs - #682
• Added DaskOnKubernetes environment (PIN-3) - #695
• Load context from Cloud when running flows - #699
• Add Queued state - #705
• flow.serialize() will always serialize its environment, regardless of build - #696
• flow.deploy() now raises an informative error if your container cannot deserialize the Flow - #711
• Add _MetaState as a parent class for states that modify other states - #726
• Add flow keyword argument to Task.set_upstream() and Task.set_downstream() - #749
• Add is_retrying() helper method to all State objects - #753
• Allow for state handlers which return None - #753
• Add daylight saving time support for CronSchedule - #729
• Add idempotency_key and context arguments to Client.create_flow_run - #757
• Make EmailTask more secure by pulling credentials from secrets - #706

Task Library

• Add GCSUpload and GCSDownload for uploading / retrieving string data to / from Google Cloud Storage - #673
• Add BigQueryTask and BigQueryInsertTask for executing queries against BigQuery tables and inserting data - #678, #685
• Add FilterTask for filtering out lists of results - #637
• Add S3Download and S3Upload for interacting with data stored on AWS S3 - #692
• Add AirflowTask and AirflowTriggerDAG tasks to the task library for running individual Airflow tasks / DAGs - #735
• Add OpenGitHubIssue and CreateGitHubPR tasks for interacting with GitHub repositories - #771
• Add Kubernetes tasks for deployments, jobs, pods, and services - #779
• Add Airtable tasks - #803
• Add Twitter tasks - #803
• Add GetRepoInfo for pulling GitHub repository information - #816

Fixes

• Fix edge case in doc generation in which some Exceptions' call signature could not be inspected - #513
• Fix bug in which exceptions raised within flow runner state handlers could not be sent to Cloud - #628
• Fix issue wherein heartbeats were not being called on a fixed interval - #669
• Fix issue wherein code blocks inside of method docs couldn't use **kwargs - #658
• Fix bug in which Prefect-generated Keys for S3 buckets were not properly converted to strings - #698
• Fix next line after Docker Environment push/pull from overwriting progress bar - #702
• Fix issue with JinjaTemplate not being pickleable - #710
• Fix issue with creating secrets from JSON documents using the Core Client - #715
• Fix issue with deserialization of JSON secrets unnecessarily calling json.loads - #716
• Fix issue where IntervalSchedules didn't respect daylight saving time after serialization - #729

Breaking Changes

• Remove the BokehRunner and associated webapp - #609
• Rename ResultHandler methods from serialize / deserialize to write / read - #612
• Refactor all State objects to store fully hydrated Result objects which track information about how results should be handled - #612, #616
• Client.create_flow_run now returns a string instead of a GraphQLResult object to match the API of deploy - #630
• flow.deploy and client.deploy require a project_name instead of an ID - #633
• Upstream state results now take precedence for task inputs over cached_inputs - #591
• Rename Match task (used inside control flow) to CompareValue - #638
• Client.graphql() now returns a response with up to two keys (data and errors). Previously the data key was automatically selected - #642
• ContainerEnvironment was changed to DockerEnvironment - #670
• The environment from_file was moved to utilities.environments - #670
• Removed start_tasks argument from FlowRunner.run() and check_upstream argument from TaskRunner.run() - #672
• Remove support for Python 3.4 - #671
• flow.run is now a blocking call which will run the Flow, on its schedule, and execute full state-based execution (including retries) - #690
• Remove make_return_failed_handler as flow.run now returns all task states - #693
• Refactor Airflow migration tools into a single AirflowTask in the task library for running individual Airflow tasks - #735
• name is now required on all Flow objects - #732
• Separate installation "extras" packages into multiple, smaller extras - #739
• Flow.parameters() always returns a set of parameters - #756