New
- In Dagit, the repository locations list has been moved from the Instance Status page to the Workspace page. When repository location errors are present, a warning icon will appear next to “Workspace” in the left navigation.
- Calls to
context.log.info()
and other similar functions now fully respect the python logging API. Concretely, log statements of the formcontext.log.error(“something %s happened!”, “bad”)
will now work as expected, and you are allowed to add things to the “extra” field to be consumed by downstream loggers:context.log.info("foo", extra={"some":"metadata"})
. - Utility functions
config_from_files
,config_from_pkg_resources
, andconfig_from_yaml_strings
have been added for constructing run config from yaml files and strings. DockerRunLauncher
can now be configured to launch runs that are connected to more than one network, by configuring thenetworks
key.
Bugfixes
- Fixed an issue with the pipeline and solid Kubernetes configuration tags.
env_from
andvolume_mounts
are now properly applied to the corresponding Kubernetes run worker and job pods. - Fixed an issue where Dagit sometimes couldn’t start up when using MySQL storage.
- [dagster-mlflow] The
end_mlflow_run_on_pipeline_finished
hook now no longer errors whenever invoked.
Breaking Changes
- Non-standard keyword arguments to
context.log
calls are now not allowed.context.log.info("msg", foo="hi")
should be rewritten ascontext.log.info("msg", extra={"foo":"hi"})
. - [dagstermill] When writing output notebook fails, e.g. no file manager provided, it won't yield
AssetMaterialization
. Previously, it would still yield anAssetMaterialization
where the path is a temp file path that won't exist after the notebook execution.
Experimental
- Previously, in order to use memoization, it was necessary to provide a resource version for every resource used in a pipeline. Now, resource versions are optional, and memoization can be used without providing them.
InputContext
andOutputContext
now each has anasset_key
that returns the asset key that was provided to the correspondingInputDefinition
orOutputDefinition
.
Documentation
- The Spark documentation now discusses all the ways of using Dagster with Spark, not just using PySpark