dagster-io/dagster 1.5.4

1.5.4 / 0.21.4 (libraries)

on GitHub

New

• Added a report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well.
• The config argument is now supported on @graph_multi_asset
• [ui] Improved performance for global search UI, especially for deployments with very large numbers of jobs or assets.
• [dagster-pipes] Add S3 context injector/reader.
• [dagster-dbt] When an exception when running a dbt command, error messages from the underlying dbt invocation are now properly surfaced to the Dagster exception.
• [dagster-dbt] The path to the dbt executable is now configurable in DbtCliResource.

Bugfixes

• Fixed a bug introduced in 1.5.3 that caused errors when launching specific Ops in a Job.
• Fixed a bug introduced in 1.5.0 that prevented the AssetExecutionContext type annotation for the context parameter in @asset_check functions.
• Fixed an issue where the Dagster scheduler would sometimes fail to retry a tick if there was an error reloading a code location in the middle of the tick.
• [dagster-dbt] Fixed an issue where explicitly passing in profiles_dir=None into DbtCliResource would cause incorrect validation.
• [dagster-dbt] Fixed an issue where partial parsing was not working when reusing existing target paths in subsequent dbt invocations.
• [ui] Fixed an issue where the job partitions UI would show “0 total partitions” if the job consisted of more than 100 assets

Community Contributions

• [dagster-duckdb] The DuckDBResource and DuckDBIOManager accept a connection_config configuration that will be passed as config to the DuckDB connection. Thanks @xjhc!

Experimental

• Added events in the run log when a step is blocked by a global op concurrency limit.
• Added a backoff for steps querying for open concurrency slots.
• Auto-materialize logic to skip materializing when (1) a backfill is in progress or (2) parent partitions are required but nonexistent are now refactored to be skip rules.
• [ui] Added 2 new asset graph layout algorithms under user settings that are significantly faster for large graphs (1000+ assets).

Documentation

• Added several pieces of documentation for Dagster Pipes, including:
  • A high-level explanation of Pipes
  • A tutorial that demonstrates how to use Pipes with a local subprocess
  • A reference for using a local subprocess with Pipes
  • A detailed explanation of Pipes, including how to customize the process
  • API references for Pipes (orchestration-side) and dagster-pipes (external process)
• Added documentation for the new experimental External Assets feature

Dagster Cloud

• Running multiple agents is no longer considered experimental.
• When the agent spins up a new code server while updating a code location, it will now wait until the new code location uploads any changes to Dagster Cloud before allowing the new server to serve requests.