dagster-io/dagster 1.6.3

1.6.3 (core) / 0.22.3 (libraries)

on GitHub

New

• Added support for the 3.0 release of the pendulum library, for Python versions 3.9 and higher.
• Performance improvements when starting run worker processes or step worker processes for runs in code locations with a large number of jobs.
• AllPartitionMapping now supports mapping to downstream partitions, enabling asset backfills with these dependencies. Thanks @craustin!
• [asset checks][experimental] @asset_check has new fields additional_deps and additional_ins to allow dependencies on assets other than the asset being checked.
• [ui] Asset graph group nodes now show status counts.
  • [dagster-snowflake] The Snowflake I/O Manager now has more specific error handling when a table doesn’t exist.
• [ui] [experimental] A new experimental UI for the auto-materialize history of a specific asset has been added. This view can be enabled under your user settings by setting “Use new asset auto-materialize history page”.
• [ui] Command clicking on an asset group will now select or deselect all assets in that group.
• [dagster-k8s] Added the ability to customize resource limits for initContainers used by Dagster system components in the Dagster Helm chart. Thanks @MattyKuzyk!
• [dagster-k8s] Added the ability to specify additional containers and initContainers in code locations in the Helm chart. Thanks @craustin!
• [dagster-k8s] Explicitly listed the set of RBAC permissions used by the agent Helm chart role instead of using a wildcard. Thanks @easontm!
• [dagster-dbt] Support for dbt-core==1.4.* is now removed because the version has reached end-of-life.

Bugfixes

• Previously, calling get_partition_keys_not_in_subset on a BaseTimeWindowPartitionsSubset that targeted a partitions definition with no partitions (e.g. a future start date) would raise an error. Now, it returns an empty list.
• Fixed issue which could cause invalid runs to be launched if a code location was updated during the course of an AMP evaluation.
• Previously, some asset backfills raised an error when targeting multi-assets with internal asset dependencies. This has been fixed.
• Previously, using the LocalComputeLogManager on Windows could result in errors relating to invalid paths. This has been resolved. Thanks @hainenber!
• An outdated path in the contribution guide has been updated. Thanks @hainenber!
• [ui] Previously an error was sometimes raised when attempting to create a dynamic partition within a multi-partitioned asset via the UI. This has been fixed.
• [ui] The “Upstream materializations are missing” warning when launching a run has been expanded to warn about failed upstream materializations as well.
• [ui] The community welcome modal now renders properly in dark mode and some elements of Asset and Op graphs have higher contrast in both themes.
• [ui] Fixed dark mode colors for datepicker, error message, and op definition elements.
• [ui] Pressing the arrow keys to navigate op/asset graphs while the layout is loading no longer causes errors.
• [ui] Exporting asset and op graphs to SVG no longer fails when chrome extensions inject additional stylesheets into Dagster’s UI.
• [ui] Dagster now defaults to UTC when the user’s default timezone cannot be identified, rather than crashing with a date formatting error.
• [ui] Fixed an issue in the asset graph sidebar that caused groups to only list their first asset.
• [ui] Fixed an issue where sensors runs would undercount the number of dynamic partition requests added or deleted if there were multiple requests for additions/deletions.
• [docs] Fixed a typo in the “Using Dagster with Delta Lake” guide. Thanks @avriiil!
• [asset checks] Fixed an issue which could cause errors when using asset checks with step launchers.
• [dagster-webserver] A bug preventing WebSocket connections from establishing on python 3.11+ has been fixed.
• [dagster-databricks] DatabricksJobRunner now ensures the correctdatabricks-sdk is installed. Thanks @zyd14!
• [dagster-dbt] On run termination, an interrupt signal is now correctly forwarded to any in-progress dbt subprocesses.
• [dagster-dbt] Descriptions for dbt tests ingested as asset checks can now be populated using the config.meta.description. Thanks @CapitanHeMo!
• [dagster-dbt] Previously, the error message displayed when no dbt profiles information was found would display an incorrect path. This has been fixed. Thanks @zoltanctoth!
• [dagster-k8s] PipesK8sClient can now correctly handle load_incluster_config . Thanks @aignas!

Documentation

• Added a new category to Concepts: Automation. This page provides a high-level overview of the various ways Dagster allows you run data pipelines without manual intervention.
• Moved several concept pages under Concepts > Automation: Schedules, Sensors, Asset Sensors, and Auto-materialize Policies.

Dagster Cloud

• Fixed an issue where configuring the agent_queue key in a dagster_cloud.yaml file incorrectly failed to validate when using the dagster-cloud ci init or dagster-cloud ci check commands during CI/CD.