dagster-io/dagster 1.1.15

1.1.15 (core) / 0.17.15 (libraries)

on GitHub

New

• Definitions now accepts Executor instances in its executor argument, not just ExecutorDefinitions.
• @multi_asset_sensor now accepts a request_assets parameter, which allows it to directly request that assets be materialized, instead of requesting a run of a job.
• Improved the performance of instantiating a Definitions when using large numbers of assets or many asset jobs.
• The job passed to build_schedule_from_partitioned_job no longer needs to have a partitions_def directly assigned to it. Instead, Dagster will infer from the partitions from the assets it targets.
• OpExecutionContext.asset_partition_keys_for_output no longer requires an argument to specify the default output.
• The “Reload all” button on the Code Locations page in Dagit will now detect changes to a pyproject.toml file that were made while Dagit was running. Previously, Dagit needed to be restarted in order for such changes to be shown.
• get_run_record_by_id has been added to DagsterInstance to provide easier access to RunRecord objects which expose the start_time and end_time of the run.
• [dagit] In the “Materialize” modal, you can now choose to pass a range of asset partitions to a single run rather than launching a backfill.
• [dagster-docker] Added a docker_container_op op and execute_docker_container_op helper function for running ops that launch arbitrary Docker containers. See the docs for more information.
• [dagster-snowflake-pyspark] The Snowflake I/O manager now supports PySpark DataFrames.
• [dagster-k8s] The Docker images include in the Dagster Helm chart are now built on the most recently released python:3.x-slim base image.

Bugfixes

• Previously, the build_asset_reconciliation_sensor could time out when evaluating ticks over large selections of assets, or assets with many partitions. A series of performance improvements should make this much less likely.
• Fixed a bug that caused a failure when using run_request_for_partition in a sensor that targeted multiple jobs created via define_asset_job.
• The cost of importing dagster has been reduced.
• Issues preventing “re-execute from failure” from working correctly with dynamic graphs have been fixed.
• [dagit] In Firefox, Dagit no longer truncates text unnecessarily in some cases.
• [dagit] Dagit’s asset graph now allows you to click “Materialize” without rendering the graph if you have too many assets to display.
• [dagit] Fixed a bug that stopped the backfill page from loading when assets that had previously been backfilled no longer had a PartitionsDefinition.
• [dagster-k8s] Fixed an issue where k8s_job_op raised an Exception when running pods with multiple containers.
• [dagster-airbyte] Loosened credentials masking for Airbyte managed ingestion, fixing the Hubspot source, thanks @joel-olazagasti!
• [dagster-airbyte] When using managed ingestion, Airbyte now pulls all source types available to the instance rather than the workspace, thanks @emilija-omnisend!
• [dagster-airbyte] Fixed an issue which arose when attaching freshness policies to Airbyte assets and using the multiprocessing executor.
• [dagster-fivetran] Added the ability to force assets to be output for all specified Fivetran tables during a sync in the case that a sync’s API outputs are missing one or more tables.

Breaking Changes

• The asset_keys and asset_selection parameters of the experimental @multi_asset_sensor decorator have been replaced with a monitored_assets parameter. This helps disambiguate them from the new request_assets parameter.

Community Contributions

• A broken docs link in snowflake_quickstart has been fixed, thanks @clayheaton!
• Troubleshooting help added to helm deployment guide, thanks @adam-bloom!
• StaticPartitionMapping is now serializable, thanks @AlexanderVR!
• [dagster-fivetran] build_fivetran_assets now supports group_name , thanks @toddy86!
• [dagster-azure] AzureBlobComputeManager now supports authentication via DefaultAzureCredential, thanks @mpicard!

Experimental

• [dagster-airflow] added a new api load_assets_from_airflow_dag that creates graph-backed, partitioned, assets based on the provided Airflow DAG.

All Changes

1.1.14...1.1.15