New
- Added documentation and helm chart configuration for threaded sensor evaluations.
- Added documentation and helm chart configuration for tick retention policies.
- Added descriptions for default config schema. Fields like execution, loggers, ops, and resources are now documented.
- UnresolvedAssetJob objects can now be passed to run status sensors.
- [dagit] A new global asset lineage view, linked from the Asset Catalog and Asset Group pages, allows you to view a graph of assets in all loaded asset groups and filter by query selector and repo.
- [dagit] A new option on Asset Lineage pages allows you to choose how many layers of the upstream / downstream graph to display.
- [dagit] Dagit's DAG view now collapses large sets of edges between the same ops for improved readability and rendering performance.
Bugfixes
- Fixed a bug with
materialize
that would cause required resources to not be applied correctly. - Fixed issue that caused repositories to fail to load when
build_schedule_from_partitioned_job
anddefine_asset_job
were used together. - Fixed a bug that caused auto run retries to always use the
FROM_FAILURE
strategy - Previously, it was possible to construct Software-Defined Assets from graphs whose leaf ops were not mapped to assets. This is invalid, as these ops are not required for the production of any assets, and would cause confusing behavior or errors on execution. This will now result in an error at definition time, as intended.
- Fixed issue where the run monitoring daemon could mark completed runs as failed if they transitioned quickly between
STARTING
andSUCCESS
status. - Fixed stability issues with the sensor daemon introduced in 0.15.3 that caused the daemon to fail heartbeat checks if the sensor evaluation took too long.
- Fixed issues with the thread pool implementation of the sensor daemon where race conditions caused the sensor to fire more frequently than the minimum interval.
- Fixed an issue with storage implementations using MySQL server version 5.6 which caused SQL syntax exceptions to surface when rendering the Instance overview pages in Dagit.
- Fixed a bug with the
default_executor_def
argument on repository where asset jobs that defined executor config would result in errors. - Fixed a bug where an erroneous exception would be raised if an empty list was returned for a list output of an op.
- [dagit] Clicking the "Materialize" button for assets with configurable resources will now present the asset launchpad.
- [dagit] If you have an asset group and no jobs, Dagit will display it by default rather than directing you to the asset catalog.
- [dagit] DAG renderings of software-defined assets now display only the last component of the asset's key for improved readability.
- [dagit] Fixes a regression where clicking on a source asset would trigger a GraphQL error.
- [dagit] Fixed issue where the “Unloadable” section on the sensors / schedules pages in Dagit were populated erroneously with loadable sensors and schedules
- [dagster-dbt] Fixed an issue where an exception would be raised when using the dbt build command with Software-Defined Assets if a test was defined on a source.
Deprecations
- Removed the deprecated dagster-daemon health-check CLI command
Community Contributions
- TimeWindow is now exported from the dagster package (Thanks @nvinhphuc!)
- Added a fix to allow customization of slack messages (Thanks @solarisa21!)
- [dagster-databricks] The
databricks_pyspark_step_launcher
now allows you to configure the following (Thanks @Phazure!):- the
aws_attributes
of the cluster that will be spun up for the step. - arbitrary environment variables to be copied over to databricks from the host machine, rather than requiring these variables to be stored as secrets.
- job and cluster permissions, allowing users to view the completed runs through the databricks console, even if they’re kicked off by a service account.
- the
Experimental
-
[dagster-k8s] Added
k8s_job_op
to launch a Kubernetes Job with an arbitrary image and CLI command. This is in contrast with thek8s_job_executor
, which runs each Dagster op in a Dagster job in its own k8s job. This op may be useful when you need to orchestrate a command that isn't a Dagster op (or isn't written in Python). Usage:from dagster_k8s import k8s_job_op my_k8s_op = k8s_job_op.configured({ "image": "busybox", "command": ["/bin/sh", "-c"], "args": ["echo HELLO"], }, name="my_k8s_op", )
-
[dagster-dbt] The dbt asset-loading functions now support
partitions_def
andpartition_key_to_vars_fn
parameters, adding preliminary support for partitioned dbt assets. To learn more, check out the Github issue!
All Changes
0.15.4...0.15.5
See All Contributors
Remove unused check_heartbeats arg (#8673)
by @johannkm
docs: use dagster brand colors for README (#8660)
by @rexledesma
fix: use absolute url for README images (#8676)
by @rexledesma
Expose TimeWindow in dagster package (#8643)
by @nvinhphuc
hold shift to force asset config modal (#8668)
by @smackesey
add helm values for configuring instance sensor config (#8657)
by @prha
[dagit] If you have no jobs, prefer routing to asset group over asset catalog (#8613)
by @bengotow
[dagit] Add control for graph depth on the Asset Lineage page, default to 5 (#8531)
by @bengotow
[assets] Fix issue with graph backed assets + partitions (#8682)
by @OwenKephart
[dagster-dbt] rework dagster dbt logging, cleanup (#8681)
by @OwenKephart
Yield run requests from run status sensors (#8635)
by @jamiedemaria
fix docs build (#8688)
by @OwenKephart
apidoc for AssetSelection and AssetsDefinition (#8618)
by @sryza
Materialize has resources arg, materialize_to_memory sets mem_io_manager for all io managers (#8659)
by @dpeng817
Load asset launchpad for assets with configurable resources (#8677)
by @smackesey
docs: update external links for github issues (#8661)
by @rexledesma
docs: convert github issue templates to forms (#8663)
by @rexledesma
switch unloadable states to dedupe by selector id (#8656)
by @prha
0.15.3 changelog (#8690)
by @dpeng817
Automation: versioned docs for 0.15.3
by @elementl-devtools
docs: remove extra cruft from issue form (#8701)
by @rexledesma
docs: improve left nav for items with children DREL-359 (#8693)
by @yuhan
docs: fix prev/next pagination (#8697)
by @yuhan
docs: docs test should capture Next Image broken links and fix broken links (#8702)
by @yuhan
[dagit] Display last asset key component on DAG rather than truncated full path (#8692)
by @bengotow
[dagit] Collapsed DAG rendering of multiple edges between the same ops (#8479)
by @bengotow
[dagit] Add Analytics context (#8674)
by @hellendag
order the backfill partitions before creating (#8703)
by @prha
skip threadpool sensor daemon tests (#8717)
by @prha
skip threaded sensor tests in py36 (#8726)
by @prha
Allow run status sensors to support unresolved asset jobs (#8689)
by @smackesey
[dagster_databricks] - support configuration of job / cluster permissions (#8683)
by @Phazure
[dagster_databricks] support aws_attributes (#8684)
by @Phazure
[Job Log perf] Use rAF to call throttleSetNodes (#8735)
by @salazarm
[dagster_databricks] support arbitrary env variables (#8685)
by @Phazure
minor changes to make dev_install (#8745)
by @smackesey
Remove deprecated daemon health-check cli (#8751)
by @johannkm
Add reexecution strategy to auto run retries (#8718)
by @johannkm
Fix for blocks_fn option (#8448)
by @clairelin135
Automation: versioned docs for 0.15.4
by @elementl-devtools
Improve race condition in run monitor (#8729)
by @brad-alexander
Add get_daemon_statuses instance method (#8752)
by @johannkm
0.15.4 changelog here (#8766)
by @prha
add retention helm values (#8724)
by @prha
Op that runs a kubernetes job (#8161)
by @gibsondan
[op] fix empty list output bug (#8763)
by @alangenfeld
[dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725)
by @OwenKephart
add validation for graph backed asset graphs (#8754)
by @OwenKephart
docs: update config schema descriptions for default executors (#8757)
by @rexledesma
[dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775)
by @OwenKephart
Restore sensor daemon yielding when evaluating sensors synchronously (#8756)
by @prha
bypass bucketed queries for mysql versions that do not support it (#8753)
by @prha
prevent multiple sensor evaluations from multithreaded race conditions (#8720)
by @prha
Fold asset_defs submodule into main dagster structure (#8446)
by @smackesey
Fix config case for default executor (#8777)
by @dpeng817
Ensure graph inputs/outputs are included in all_dagster_types (#8736)
by @smackesey
[dagit] Bring back the global asset graph as an “all asset groups” view (#8709)
by @bengotow
fix source asset regression (#8784)
by @smackesey
improve error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776)
by @sryza
fix issue with repos and partitioned scheduled asset jobs (#8779)
by @sryza
0.15.5 Changelog (#8781)
by @prha
Fix bug with how resources are applied in materialize (#8790)
by @dpeng817
0.15.5
by @elementl-devtools