- There is now a new top-level configuration section
storagewhich controls whether or not
execution should store intermediate values and the history of pipeline runs on the filesystem,
on S3, or in memory. The
dagsterCLI now includes options to list and wipe pipeline run
history. Facilities are provided for user-defined types to override the default serialization
used for storage.
- Similarily, there is a new configuration for
RunConfigwhere the user can specify
intermediate value storage via an API.
OutputDefinitionnow contains an explicit
is_optionalparameter and defaults to being
- New functionality in
- New functionality in
- Dagster default logging is now multiline for readability.
Nothingtype now allows dependencies to be constructed between solids that do not have
- Many error messages have been improved.
throw_on_user_errorhas been renamed to
raise_on_errorin all APIs, public and private
- The GraphQL layer has been extracted out of Dagit into a separate dagster-graphql package.
startSubplanExecutionhas been replaced by
startPipelineExecutionnow supports reexecution of pipeline subsets.
- It is now possible to reexecute subsets of a pipeline run from Dagit.
Executetab now opens runs in separate browser tabs and a new
Runstab allows you to
browse and view historical runs.
- Dagit no longer scaffolds configuration when creating new
Executetabs. This functionality will
be refined and revisited in the future.
Exploretab is more performant on large DAGs.
dagit -qcommand line flag has been deprecated in favor of a separate command-line
- The execute button is now greyed out when Dagit is offline.
- The Dagit UI now includes more contextual cues to make the solid in focus and its connections
- Dagit no longer offers to open materializations on your machine. Clicking an on-disk
materialization now copies the path to your clipboard.
- Pressing Ctrl-Enter now starts execution in Dagit's Execute tab.
- Dagit properly shows List and Nullable types in the DAG view.
- Dagster-Airflow includes functions to dynamically generate containerized (
and uncontainerized (
PythonOperator-based) Airflow DAGs from Dagster pipelines and config.
- Dagster integration code with AWS, Great Expectations, Pandas, Pyspark, Snowflake, and Spark
has been reorganized into a new top-level libraries directory. These modules are now
- Removed dagster-sqlalchemy and dagma
- Added the event-pipeline-demo, a realistic web event data pipeline using Spark and Scala.
- Added the Pyspark pagerank example, which demonstrates how to incrementally introduce dagster
into existing data processing workflows.
- Docs have been expanded, reorganized, and reformatted.