This is the first "major" release of dagster meant for consumption. The public APIs in this release will be supported for some time.
New things in this release:
- Solids do not specify their dependencies anymore. They are more easily reusable between pipelines. Dependencies now specified at the pipeline level.
- Solids support multiple outputs and branching
- Solids can take config, in addition to inputs and outputs.
- Sources and materializations have been eliminated as formal abstractions. Solids accepting configs enabled this.
- New configuration system with full type system instead of argument dictionary. Configs can be arbitrarily nested and support composite types.
- New result api
- New execution engine. this now does a compiler-esque pass where a new logical execution graph of nodes is generated from the logical definition files and config.
- Python 2.7, 3.5 and 3.6 now supported
- RepositoryDefinition has been added. pipelines.yml is gone
- Full documentation of all public APIs
- Multi-part tutorial that introduces all concepts.
- @solid now must take info object, which has config and context members. @lambda_solid is for simple cases that do not require config and context.
- ... Much more