dagster-io/dagster 0.7.7

on GitHub

Breaking Changes

• The default sqlite and dagster-postgres implementations have been altered to extract the
  event step_key field as a column, to enable faster per-step queries. You will need to run
  dagster instance migrate to update the schema. You may optionally migrate your historical event
  log data to extract the step_key using the migrate_event_log_data function. This will ensure
  that your historical event log data will be captured in future step-key based views. This
  event_log data migration can be invoked as follows:

  from dagster.core.storage.event_log.migration import migrate_event_log_data
  from dagster import DagsterInstance
  
  migrate_event_log_data(instance=DagsterInstance.get())

• We have made pipeline metadata serializable and persist that along with run information.
  While there are no user-facing features to leverage this yet, it does require an instance migration.
  dagster instance migrate. If you have already run the migration for the event_log changes
  above, you do not need to run it again. Any unforeseen errors related the the new snapshot_id
  in the runs table or the new snapshots table are related to this migration.

• dagster-pandas ColumnTypeConstraint has been removed in favor of ColumnDTypeFnConstraint and
  ColumnDTypeInSetConstraint.

New

• You can now specify that dagstermill output notebooks be yielded as an output from dagstermill
  solids, in addition to being materialized.
• You may now set the extension on files created using the FileManager machinery.
• dagster-pandas typed PandasColumn constructors now support pandas 1.0 dtypes.
• The Dagit Playground has been restructured to make the relationship between Preset, Partition
  Sets, Modes, and subsets more clear. All of these buttons have be reconciled and moved to the
  left side of the Playground.
• Config sections that are required but not filled out in the Dagit playground are now detected
  and labeled in orange.
• dagster-celery config now support using env: to load from environment variables.

Bugfix

• Fixed a bug where selecting a preset in dagit would not populate tags specified on the pipeline
  definition.
• Fixed a bug where metadata attached to a raised Failure was not displayed in the error modal in
  dagit.
• Fixed an issue where reimporting dagstermill and calling dagstermill.get_context() outside of
  the parameters cell of a dagstermill notebook could lead to unexpected behavior.
• Fixed an issue with connection pooling in dagster-postgres, improving responsiveness when using
  the Postgres-backed storages.

Experimental

• Added a longitudinal view of runs for on the Schedule tab for scheduled, partitioned pipelines.
  Includes views of run status, execution time, and materializations across partitions. The UI is
  in flux and is currently optimized for daily schedules, but feedback is welcome.