github dagster-io/dagster 0.7.7

Breaking Changes

  • The default sqlite and dagster-postgres implementations have been altered to extract the
    event step_key field as a column, to enable faster per-step queries. You will need to run
    dagster instance migrate to update the schema. You may optionally migrate your historical event
    log data to extract the step_key using the migrate_event_log_data function. This will ensure
    that your historical event log data will be captured in future step-key based views. This
    event_log data migration can be invoked as follows:
  from dagster.core.storage.event_log.migration import migrate_event_log_data
  from dagster import DagsterInstance

  migrate_event_log_data(instance=DagsterInstance.get())
  • We have made pipeline metadata serializable and persist that along with run information.
    While there are no user-facing features to leverage this yet, it does require an instance migration.
    dagster instance migrate. If you have already run the migration for the event_log changes
    above, you do not need to run it again. Any unforeseen errors related the the new snapshot_id
    in the runs table or the new snapshots table are related to this migration.
  • dagster-pandas ColumnTypeConstraint has been removed in favor of ColumnDTypeFnConstraint and
    ColumnDTypeInSetConstraint.

New

  • You can now specify that dagstermill output notebooks be yielded as an output from dagstermill
    solids, in addition to being materialized.
  • You may now set the extension on files created using the FileManager machinery.
  • dagster-pandas typed PandasColumn constructors now support pandas 1.0 dtypes.
  • The Dagit Playground has been restructured to make the relationship between Preset, Partition
    Sets, Modes, and subsets more clear. All of these buttons have be reconciled and moved to the
    left side of the Playground.
  • Config sections that are required but not filled out in the Dagit playground are now detected
    and labeled in orange.
  • dagster-celery config now support using env: to load from environment variables.

Bugfix

  • Fixed a bug where selecting a preset in dagit would not populate tags specified on the pipeline
    definition.
  • Fixed a bug where metadata attached to a raised Failure was not displayed in the error modal in
    dagit.
  • Fixed an issue where reimporting dagstermill and calling dagstermill.get_context() outside of
    the parameters cell of a dagstermill notebook could lead to unexpected behavior.
  • Fixed an issue with connection pooling in dagster-postgres, improving responsiveness when using
    the Postgres-backed storages.

Experimental

  • Added a longitudinal view of runs for on the Schedule tab for scheduled, partitioned pipelines.
    Includes views of run status, execution time, and materializations across partitions. The UI is
    in flux and is currently optimized for daily schedules, but feedback is welcome.
latest releases: 0.10.1.pre0, 0.10.1, 0.10.0...
8 months ago