dagster-io/dagster 1.3.8

1.3.8 (core) / 0.19.8 (libraries)

on GitHub

New

• Multipartitioned assets with one time dimension can now depend on earlier partitions of themselves.
• define_asset_job now accepts a hooks argument.
• Added support for sqlalchemy==2.x
• [ui] The Runs page has been revamped with better filtering support
• [ui] The automaterialize policy page for SDA’s using the experimental AutomaterializePolicy feature now indicates time periods where no materializations happened due to no materialization conditions being met
• [dagster-k8s] The Dagster Helm chart now includes an additionalInstanceConfig key that allows you to supply additional configuration to the Dagster instance.
• [dagster-aws] The EcsRunLauncher now uses a different task definition family for each job, instead of registering a new task definition revision each time a different job is launched.
• [dagster-aws] The EcsRunLauncher now includes a run_ecs_tags config key that lets you configure tags on the launched ECS task for each run.

Bugfixes

• When a sensor had yield statement and also returned a SkipReason, the SkipReason would be ignored. This has been fixed.
• [dagster-cloud] Fixed a bug in the docker user code launcher that was preventing code location containers from being properly cleaned up.
• Fixed an issue where the Dagster UI would sometimes raise a ``RuntimeError: dictionary changed size during iteration` exception while code servers were being reloaded.
• Fixed an issue where the Dagster daemon reloaded your code server every 60 seconds when using the new experimental dagster code-server start CLI, instead of only reloading your code when you initiate a reload from the Dagster UI.
• Fixed a GraphQL error which would occur when loading the default config for jobs without config.
• [dagster-dbt] Fixed an error which would arise when trying to load assets from a DBT Cloud instance using the Pythonic-style resource.

Community Contributions

• Added the ability to specify metadata on asset jobs, by adding the metadata parameter to define_asset_job (Thanks Elliot2718!)
• [dagster-databricks] Connected databricks stdout to local stdout, to be handled by the compute log manager (Thanks loerinczy!)
• [dagster-census] Fixed poll_sync_run to handle the “preparing” status from the Census API (Thanks ldnicolasmay!)

Experimental

• @observable_source_asset-decorated functions can now return a DataVersionsByPartition to record versions for partitions.
• @dbt_assets
  • DbtCliTask's created by invoking DbtCli.cli(...) now have a method .is_successful(), which returns a boolean representing whether the underlying CLI process executed the dbt command successfully.
  • Descriptions of assets generated by @dbt_assets can now be customized by overriding DbtManifest.node_info_to_description.
  • IO Managers can now be configured on @dbt_assets.

Documentation

• New guide on using Dagster to manage machine learning pipelines

Dagster Cloud

• Added support for streaming upload of compute logs to Dagster Cloud
• The ECS agent now supports setting server_ecs_tags and run_ecs_tags that apply to each service or task created by the agent. See the docs for more information.
• Fixed run filtering for calls to instance.get_run_partition_data in Dagster Cloud.