dagster-io/dagster 1.0.13

1.0.13 (core) / 0.16.13 (libraries) 2022-10-14

on GitHub

New

• AssetMaterialization now has a metadata property, which allows accessing the materialization’s metadata as a dictionary.
• DagsterInstance now has a get_latest_materialization_event method, which allows fetching the most recent materialization event for a particular asset key.
• RepositoryDefinition.load_asset_value and AssetValueLoader.load_asset_value now work with IO managers whose load_input implementation accesses the op_def and name attributes on the InputContext.
• RepositoryDefinition.load_asset_value and AssetValueLoader.load_asset_value now respect the DAGSTER_HOME environment variable.
• InMemoryIOManager, the IOManager that backs mem_io_manager, has been added to the public API.
• The multi_asset_sensor (experimental) now supports marking individual partitioned materializations as “consumed”. Unconsumed materializations will appear in future calls to partitioned context methods.
• The build_multi_asset_sensor_context testing method (experimental) now contains a flag to set the cursor to the newest events in the Dagster instance.
• TableSchema now has a static constructor that enables building it from a dictionary of column names to column types.
• Added a new CLI command dagster run migrate-repository which lets you migrate the run history for a given job from one repository to another. This is useful to preserve run history for a job when you have renamed a repository, for example.
• [dagit] The run timeline view now shows jobs grouped by repository, with each repository section collapsible. This feature was previously gated by a feature flag, and is now turned on for everyone.
• [dagster-airbyte] Added option to specify custom request params to the Airbyte resource, which can be used for auth purposes.
• [dagster-airbyte] When loading Airbyte assets from an instance or from YAML, a filter function can be specified to ignore certain connections.
• [dagster-airflow] DagsterCloudOperator and DagsterOperator now support Airflow 2. Previously, installing the library on Airflow 2 would break due to an import error.
• [dagster-duckdb] A new integration with DuckDB allows you to store op outputs and assets in an in-process database.

Bugfixes

• Previously, if retries were exceeded when running with execute_in_process, no error would be raised. Now, a DagsterMaxRetriesExceededError will be launched off.
• [dagster-airbyte] Fixed generating assets for Airbyte normalization tables corresponding with nested union types.
• [dagster-dbt] When running assets with load_assets_from_...(..., use_build=True), AssetObservation events would be emitted for each test. These events would have metadata fields which shared names with the fields added to the AssetMaterialization events, causing confusing historical graphs for fields such as Compilation Time. This has been fixed.
• [dagster-dbt] The name for the underlying op for load_assets_from_... was generated in a way which was non-deterministic for dbt projects which pulled in external packages, leading to errors when executing across multiple processes. This has been fixed.

Dependency changes

• [dagster-dbt] The package no longer depends on pandas and dagster-pandas.

Community Contributions

• [dagster-airbyte] Added possibility to change request timeout value when calling Airbyte. Thanks @FransDel!
• [dagster-airflow] Fixed an import error in dagster_airflow.hooks. Thanks @bollwyvl!
• [dagster-gcp] Unpin Google dependencies. dagster-gcp now supports google-api-python-client 2.x. Thanks @amarrella!
• [dagstermill] Fixed an issue where DagsterTranslator was missing an argument required by newer versions of papermill. Thanks @tizz98!

Documentation

• Added an example, underneath examples/assets_smoke_test, that shows how to write a smoke test that feeds empty data to all the transformations in a data pipeline.
• Added documentation for build_asset_reconciliation_sensor.
• Added documentation for monitoring partitioned materializations using the multi_asset_sensor and kicking off subsequent partitioned runs.
• [dagster-cloud] Added documentation for running the Dagster Cloud Docker agent with Docker credential helpers.
• [dagster-dbt] The class methods of the dbt_cli_resource are now visible in the API docs for the dagster-dbt library.
• [dagster-dbt] Added a step-by-step tutorial for using dbt models with Dagster software-defined assets

All Changes

1.0.12...1.0.13