dagster-io/dagster 0.11.6

on GitHub

Breaking Changes

• DagsterInstance.get() no longer falls back to an ephemeral instance if DAGSTER_HOME is not set. We don’t expect this to break normal workflows. This change allows our tooling to be more consistent around it’s expectations. If you were relying on getting an ephemeral instance you can use DagsterInstance.ephemeral() directly.
• Undocumented attributes on HookContext have been removed. step_key and mode_def have been documented as attributes.

New

• Added a permanent, linkable panel in the Run view in Dagit to display the raw compute logs.
• Added more descriptive / actionable error messages throughout the config system.
• When viewing a partitioned asset in Dagit, display only the most recent materialization for a partition, with a link to view previous materializations in a dialog.
• When viewing a run in Dagit, individual log line timestams now have permalinks. When loading a timestamp permalink, the log table will highlight and scroll directly to that line.
• The default config_schema for all configurable objects - solids, resources, IO managers, composite solids, executors, loggers - is now Any. This means that you can now use configuration without explicitly providing a config_schema. Refer to the docs for more details: https://docs.dagster.io/concepts/configuration/config-schema.
• When launching an out of process run, resources are no longer initialized in the orchestrating process. This should give a performance boost for those using out of process execution with heavy resources (ie, spark context).
• input_defs and output_defs on @solid will now flexibly combine data that can be inferred from the function signature that is not declared explicitly via InputDefinition / OutputDefinition. This allows for more concise defining of solids with reduced repetition of information.
• [Helm] Postgres storage configuration now supports connection string parameter keywords.
• The Status page in Dagit will now display errors that were surfaced in the dagster-daemon process within the last 5 minutes. Previously, it would only display errors from the last 30 seconds.
• Hanging sensors and schedule functions will now raise a timeout exception after 60 seconds, instead of crashing the dagster-daemon process.
• The DockerRunLauncher now accepts a container_kwargs config parameter, allowing you to specify any argument to the run container that can be passed into the Docker containers.run method. See https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.ContainerCollection.run for the full list of available options.
• Added clearer error messages for when a Partition cannot be found in a Partition Set.
• The celery_k8s_job_executor now accepts a job_wait_timeout allowing you to override the default of 24 hours.

Bugfixes

• Fixed the raw compute logs in Dagit, which were not live updating as the selected step was executing.
• Fixed broken links in the Backfill table in Dagit when Dagit is started with a --prefix-path argument.
• Showed failed status of backfills in the Backfill table in Dagit, along with an error stack trace. Previously, the backfill jobs were stuck in a Requested state.
• Previously, if you passed a non-required Field to the output_config_schema or input_config_schema arguments of @io_manager, the config would still be required. Now, the config is not required.
• Fixed nested subdirectory views in the Assets catalog, where the view switcher would flip back from the directory view to the flat view when navigating into subdirectories.
• Fixed an issue where the dagster-daemon process would crash if it experienced a transient connection error while connecting to the Dagster database.
• Fixed an issue where the dagster-airflow scaffold command would raise an exception if a preset was specified.
• Fixed an issue where Dagit was not including the error stack trace in the Status page when a repository failed to load.