New
Definitions
is no longer marked as experimental and is the preferred API over@repository
for new users of Dagster. Examples, tutorials, and documentation have largely ported to this new API. No migration is needed. Please see GitHub discussion for more details.- The “Workspace” section of Dagit has been removed. All definitions for your code locations can be accessed via the “Deployment” section of the app. Just as in the old Workspace summary page, each code location will show counts of its available jobs, assets, schedules, and sensors. Additionally, the code locations page is now available at
/locations
. - Lagged / rolling window partition mappings:
TimeWindowPartitionMapping
now acceptsstart_offset
andend_offset
arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets. - Asset partitions can now depend on earlier time partitions of the same asset. The asset reconciliation sensor will respect these dependencies when requesting runs.
dagit
can now accept multiple arguments for the-m
and-f
flags. For each argument a new code location is loaded.- Schedules created by
build_schedule_from_partitioned_job
now execute more performantly - in constant time, rather than linear in the number of partitions. - The
QueuedRunCoordinator
now supports optionsdequeue_use_threads
anddequeue_num_workers
options to enable concurrent run dequeue operations for greater throughput. - [dagster-dbt]
load_assets_from_dbt_project
,load_assets_from_dbt_manifest
, andload_assets_from_dbt_cloud_job
now support applying freshness policies to loaded nodes. To do so, you can applydagster_freshness_policy
config directly in your dbt project, i.e.config(dagster_freshness_policy={"maximum_lag_minutes": 60})
would result in the corresponding asset being assigned aFreshnessPolicy(maximum_lag_minutes=60)
. - The
DAGSTER_RUN_JOB_NAME
environment variable is now set in containerized environments spun up by our run launchers and executor. - [dagster-airflow]
make_dagster_repo_from_airflow_dags_path
,make_dagster_job_from_airflow_dag
andmake_dagster_repo_from_airflow_dag_bag
have a newconnections
parameter which allows for configuring the airflow connections used by migrated dags.
Bugfixes
-
Fixed a bug where the
log
property was not available on theRunStatusSensorContext
context object provided for run status sensors for sensor logging. -
Fixed a bug where the re-execute button on runs of asset jobs would incorrectly show warning icon, indicating that the pipeline code may have changed since you last ran it.
-
Fixed an issue which would cause metadata supplied to graph-backed assets to not be viewable in the UI.
-
Fixed an issue where schedules often took up to 5 seconds to start after their tick time.
-
Fixed an issue where Dagster failed to load a dagster.yaml file that specified the folder to use for sqlite storage in the
dagster.yaml
file using an environment variable. -
Fixed an issue which would cause the k8s/docker executors to unnecessarily reload CacheableAssetsDefinitions (such as those created when using
load_assets_from_dbt_cloud_job
) on each step execution. -
[dagster-airbyte] Fixed an issue where Python-defined Airbyte sources and destinations were occasionally recreated unnecessarily.
-
Fixed an issue with
build_asset_reconciliation_sensor
that would cause it to ignore in-progress runs in some cases. -
Fixed a bug where GQL errors would be thrown in the asset explorer when a previously materialized asset had its dependencies changed.
-
[dagster-airbyte] Fixed an error when generating assets for normalization table for connections with non-object streams.
-
[dagster-dbt] Fixed an error where dbt Cloud jobs with
dbt run
anddbt run-operation
were incorrectly validated. -
[dagster-airflow]
use_ephemeral_airflow_db
now works when running within a PEX deployment artifact.
Documentation
- New documentation for Code locations and how to define one using
Definitions
- Lots of updates throughout the docs to reflect the recommended usage of
Definitions
. Any content not ported toDefinitions
in this release is in the process of being updated. - New documentation for dagster-airflow on how to start writing dagster code from an airflow background.
All Changes
1.1.6...1.1.7
See All Contributors
Non isolated runs docs (#10860)
by @johannkm
[dagit] Fix Gantt chart rendering of per-step resource init log messages (#10943)
by @bengotow
[dagit] Fix the “Assets” label on large asset runs (#10932)
by @bengotow
[dagit] Fix “Job In” label regression in Chrome v109 (#10934)
by @bengotow
[dagit] Pass repository tag when loading runs for Partitions page (#10948)
by @bengotow
[dagit] Updated asset DAG styles, added additional compute tags (#10931)
by @bengotow
[docs] - add a guide for scheduling assets (#10949)
by @slopp
cap packaging requirement at 22.0 (#10968)
by @smackesey
Execution result typing (#10919)
by @smackesey
solid -> node method renames (#10920)
by @smackesey
[dagster-airflow] re-enable airflow 2.5.0 tests (#10966)
by @Ramshackle-Jamathon
[dagster-airbyte][docs] Use dagster-airbyte CLI alias in docs (#10955)
by @benpankow
[dagster-slack] create slack_on_freshness_policy_sensor (#10960)
by @OwenKephart
[docs ] - fix image dimensions in hello-dagster materialize (#10977)
by @slopp
1.1.6 Changelog (#10978)
by @OwenKephart
Automation: versioned docs for 1.1.6
by @elementl-devtools
[convert-environment-variables-and-secrets-guide-stack-2] Convert env vars and secrets guide from repository to Definitions
by @schrockn
unexperimentalize PartitionMapping (#10980)
by @sryza
remove validation in AssetGraph.get_child_partition_keys_of_parent an… (#10981)
by @sryza
[convert-development-to-production-1] Move repository to __init__.py
by @schrockn
[convert-development-to-production-2] Changing development to production to use snowflake_pandas_io_manager
by @schrockn
[convert-deployment-to-production-3] Convert @repository to Definitions
by @schrockn
[convert-development-to-production-4] Use base object instead of resource
by @schrockn
[convert-development-to-production-6] Use pyproject.toml instead of workspace.yaml
by @schrockn
[convert-development-to-production-7] Convert guide to use Definitions
by @schrockn
[dagit] New Code Locations table (#10975)
by @hellendag
Pin graphene to <3.2
by @schrockn
fix missing metadata in dagit on graph-backed assets (#10988)
by @OwenKephart
[dagit] Export a few Code Location components for Cloud (#11008)
by @hellendag
Temporarily disable some Azure test suites (#11007)
by @jmsanders
[graphql] fix for graphene 3.2 (#11011)
by @alangenfeld
[code-location-selector-stack] Code location sensor tests 1/N. Rename workspace_load_target function to create_workspace_load_target
by @schrockn
[code-location-selector-stack] Code location sensor tests 2/N Make instance_with_multiple_repos_with_sensors workspace_load_target parameterizable
by @schrockn
[code-location-selector-stack] Code location sensor tests 3/N. Refactor instance_with_multiple_repos_with_sensors to handle multiple code locations
by @schrockn
[code-location-selector-stack] Code location sensor tests 4/N Actually add test to test cross code location selector
by @schrockn
[code-location-selector-stack Add CodeLocationSelector; Have run_status_sensor accept it
by @schrockn
Fix test_persistent
by @schrockn
chore: auto-assign dependabot pull requests (#10953)
by @rexledesma
fix(dbt-cloud): parse command string to find materialization commands (#10989)
by @rexledesma
[code-location-selector-stack] Change typehint on make_slack_on_run_failure_sensor to accept CodeLocationSelector
by @schrockn
[docs] re-org snowflake integration guide (#10984)
by @jamiedemaria
Check scheduler ticks right after each minute boundary instead of once every 5 seconds (#10886)
by @gibsondan
[docs] - [definitions] Update Configured API concept doc (#11020)
by @erinkcochran87
Fix bug with re-execution snapshot ids (#10967)
by @OwenKephart
[bugfix] UPathIOManger load_input type checking (#11022)
by @danielgafni
Pathspec typing fix (#11036)
by @smackesey
[dagit] /code-locations -> /locations (#11024)
by @hellendag
Replace partition ranges with subsets (#10909)
by @clairelin135
[dagit] With multiple assets selected, backfill “missing” should include partially materialized partitions (#11027)
by @bengotow
[dagit] Add empty value string for invalid tag input in Runs filter (#11044)
by @hellendag
[docs] - [definitions] Update Repository page for Definitions (#10986)
by @erinkcochran87
[definitions-accessors] Add get_job_def to
Definitions.
by @schrockn
1/ definitions in create new project: update dagster project CLI (#10829)
by @yuhan
2/ definitions in create new project: update create-new-project docs (#10830)
by @yuhan
[docs] - [definitions] - Update dbt tutorial to use Definitions (#10842)
by @erinkcochran87
2.1/ definitions in in create new project: remove unnecessary asset dir in scaffold + update docs (#10831)
by @yuhan
update isolated run docs to say that it's enabled by default (#11039)
by @gibsondan
[docs] - remove repository from hello dagster as its not needed (#11048)
by @slopp
[dagster-fivetran][ez] Add missing docstring entry for build_fivetran_assets (#11049)
by @benpankow
Add start_offset and end_offset to TimeWindowPartitionMapping (#10979)
by @sryza
fix subsettable multi asset case (#10878)
by @OwenKephart
move ecs task tagging to run_task call (#11037)
by @prha
Add custom resource key to
load_assets_from_dbt_project (#10827)
by @dpeng817
[dagit] Highlight Assets in top nav for global asset graph (#11052)
by @hellendag
[dagster-airbyte][docs][ez] Fix AirbyteConnection apidoc rendering (#11059)
by @benpankow
convert-examples 1.1/ assets_dbt_python defs + pyproject (#11060)
by @yuhan
convert-examples 2.1/ assets_modern_data_stack repo -> defs, pyproject (#11062)
by @yuhan
convert-examples 5/ assets_smoke_test repository -> definitions (#11071)
by @yuhan
convert-examples 6.1/ feature_graph_backed_assets pyproject, @repository -> Definitions (#11072)
by @yuhan
convert-examples 7.1/ quickstart_etl workspace->pyproject, repo->defs (#11074)
by @yuhan
convert-examples 8.1/ with_great_expectations workspace->pyproject, repo->defs (#11077)
by @yuhan
convert-examples 1.2/ assets_dbt_python file renames (#11061)
by @yuhan
convert-examples 6.2/ feature_graph_backed_assets file rename and reorder (#11073)
by @yuhan
convert-examples 2.2/ assets_modern_data_stack file renames (#11063)
by @yuhan
Removes codecov commands from tox (#10635)
by @dpeng817
[dagit] Disable Re-execute menu item in run action menu for perms (#11026)
by @hellendag
Support loading multiple modules in single command line invocation
by @schrockn
Enable multiple files in CLI tools
by @schrockn
Add get_sensor_def and get_schedule_def to Definitions
by @schrockn
convert-examples 10/ with_pyspark_emr repository.py -> defs.py, update [docs] (#11080)
by @yuhan
convert-examples 9/ with_pyspark repository.py -> defs.py, update [docs] (#11079)
by @yuhan
Add load_asset_value and get_asset_value_loader to Definitions
by @schrockn
[dagster-airflow] reorg dagster airflow api docs (#11009)
by @Ramshackle-Jamathon
convert-examples 3.1/ assets_pandas_pyspark definitons, pyproject (#11067)
by @yuhan
convert-examples 8.2/ with_great_expectations file rename (#11078)
by @yuhan
convert-examples 7.2/ quickstart_etl file rename + update README (#11075)
by @yuhan
Support module_name in tool.dagster section of pyproject.toml
by @schrockn
Move all examples to use module_name in pyproject.toml
by @schrockn
Pin tox on windows to < 4 (#11013)
by @jmsanders
[docs] - [definitions] Code location concept page (#10843)
by @erinkcochran87
Add instance for test to top level API (#10709)
by @dpeng817
[example] recomment code in dagstermill tutorial (#11098)
by @jamiedemaria
Make JobSelector work on cross code location
by @schrockn
[convert-branch-deployments-stack] Convert branch deployments guide to Definitions
by @schrockn
Eliminate package_name from pyproject.toml loading spec
by @schrockn
[docs] - [definitions] Update Enriching with SDAs guide (#10849)
by @erinkcochran87
[dagit] Don't show View link on code location toast if already there (#11095)
by @hellendag
set DAGSTER_RUN_JOB_NAME env var (#10888)
by @alangenfeld
remove parameter tag from dagstermill notebook (#11109)
by @jamiedemaria
sort task definition config secrets and env vars to ensure consistent ordering (#10982)
by @gibsondan
[dagster-airflow] from airflow to dagster guide (#10923)
by @Ramshackle-Jamathon
[dagit] Improvements to partition range selection interactions (#11017)
by @bengotow
add tests for legacy storage (#11047)
by @prha
keep track of max timestamps client side for code location updates (#11100)
by @prha
in default IO manager, handle PartitionMappings that return 0 partitions (#11065)
by @sryza
Fix bug with logical versions for assets with changed deps (#11083)
by @smackesey
Adding apidocs for Definitions
by @schrockn
convert-examples 3.2/ asset_pandas_pyspark re-arrange files + [docs] (#11068)
by @yuhan
[dagit] Persist AssetPartitions state to URL, open Materialize panel with same selection (#11042)
by @bengotow
[dagit] Render assets with self-dependencies in Dagit (#11043)
by @bengotow
run queue daemon refactor (#11090)
by @alangenfeld
Allow assets to depend on earlier partitions of themselves (#11066)
by @sryza
[docs] - [definitions] Update Schedules documentation (#11046)
by @erinkcochran87
Re-add section for customizing Docker images pre-fast-deploys (#11112)
by @shalabhc
Allow setting base_dir as a StringSource in sqlite storage (#11031)
by @gibsondan
handle self-dependencies in reconciliation sensor (#11085)
by @sryza
[docs] - [definitions] Update Workspaces concept page (#10944)
by @erinkcochran87
Coerce IOManager objects into IOManagerDefinitions in Definitions
by @schrockn
Revert "[docs] - [definitions] Update Workspaces concept page (#10944)"
by @alangenfeld
Allow overriding the default max_concurrent for the default executor via env var (#11116)
by @gibsondan
convert-examples 4.1/ assets_pandas_type_metadata definitions, pyproject (#11069)
by @yuhan
[docs] [definitions] convert snowflake guide to definitions (#11029)
by @jamiedemaria
[docs] - Fix workspace link (#11121)
by @erinkcochran87
convert-examples 4.2/ assets_pandas_type_metadata repository.py -> __init__.py (new) (#11127)
by @yuhan
Add executor and loggers to Definitions
by @schrockn
[docs] - [definitions] Update Sensors documentation (#11050)
by @erinkcochran87
threaded run queue daemon (#11113)
by @alangenfeld
convert-examples 11.1/ project_fully_featured defs, pyproject, no with_resources (#11081)
by @yuhan
[dagster-airbyte] Fix normalization table generation, materializations for non-object Streams (#10899)
by @benpankow
update assets_modern_data_stack example readme (#11122)
by @yuhan
[CacheableAssets] Fix bug that would cause definitions to be recomputed when using a StepDelegatingExecutor (#11086)
by @OwenKephart
convert-examples 11.2/ project_fully_featured repository.py -> __init__.py (#11082)
by @yuhan
update assets_dbt_python example readme (#11126)
by @yuhan
[dagster-dbt] Enable setting FreshnessPolicies on dbt assets (#11103)
by @OwenKephart
Mark Definitions as not experimental
by @schrockn
[declarative-scheduling] Fix issue where in-progress runs were not properly handled (#11118)
by @OwenKephart
[dagster-airbyte] Avoid unnecessarily recreating sources, destinations w/ managed ingestion (#11117)
by @benpankow
[dagster-airflow] pex compatibility for ephemeral db (#11115)
by @Ramshackle-Jamathon
[dagster-airflow] provide interface for passing connection models directly to dagster (#11006)
by @Ramshackle-Jamathon
[dagit] Left nav: show entire code location string for non-dunder repos (#11133)
by @hellendag
move quickstart_aws back to mono repo (#11130)
by @yuhan
move quickstart_gcp back to mono repo (#11131)
by @yuhan
move quickstart_snowflake back to mono repo (#11132)
by @yuhan
Update Definitions docs
by @schrockn
Use wrap_resources_for_execution in Definitions
by @schrockn
Use i/o manager coercion in assets_pandas_pyspark example
by @schrockn
convert-examples-repo-to-defs cloud nux quickstart_aws (#11135)
by @yuhan
convert-examples-repo-to-defs cloud nux quickstart_gcp (#11136)
by @yuhan
convert-examples-repo-to-defs cloud nux quickstart_snowflake (#11137)
by @yuhan
enable run status sensor logging (#11145)
by @prha
Move jobs concept guide to refer to Definitions
by @schrockn
Move I/O Manager Guide to be on Definitions
by @schrockn
add thread name prefix to run dequeue workers (#11155)
by @alangenfeld
Adjust resources guide to be in a Definitions world
by @schrockn
make schedules produced by build_schedule_from_partitioned_job more p… (#11147)
by @sryza
[docs] - [definitions] Update Dagit + tutorial screenshots (#11089)
by @erinkcochran87
[docs] - [definitions] Update Partitions concept docs (#11030)
by @erinkcochran87
Port asset sensor guide to Definitions
by @schrockn
Revert "keep track of max timestamps client side for code location up… (#11162)
by @prha
[docs] - Fix links (#11163)
by @erinkcochran87
Merge branch 'release-1.1.7' of github.com:dagster-io/dagster into release-1.1.7
by @jamiedemaria
1.1.7 Changelog (#11165)
by @jamiedemaria
1.1.7
by @elementl-devtools