github dagster-io/dagster v0.2.0

This is the first "major" release of dagster meant for consumption. The public APIs in this release will be supported for some time.

New things in this release:

  • Solids do not specify their dependencies anymore. They are more easily reusable between pipelines. Dependencies now specified at the pipeline level.
  • Solids support multiple outputs and branching
  • Solids can take config, in addition to inputs and outputs.
  • Sources and materializations have been eliminated as formal abstractions. Solids accepting configs enabled this.
  • New configuration system with full type system instead of argument dictionary. Configs can be arbitrarily nested and support composite types.
  • New result api
  • New execution engine. this now does a compiler-esque pass where a new logical execution graph of nodes is generated from the logical definition files and config.
  • Python 2.7, 3.5 and 3.6 now supported
  • RepositoryDefinition has been added. pipelines.yml is gone
  • Full documentation of all public APIs
  • Multi-part tutorial that introduces all concepts.
  • @solid now must take info object, which has config and context members. @lambda_solid is for simple cases that do not require config and context.
  • ... Much more
latest releases: 0.10.1.pre0, 0.10.1, 0.10.0...
2 years ago