🎆 🚢 🎆 Dagster 0.7.0: Waiting To Exhale 😤 😌 🍵
We are pleased to announce version 0.7.0 of Dagster, codenamed “Waiting To Exhale”. We set out to make Dagster a solution for production-grade pipelines on modern cloud infrastructure. In service of that goal, we needed to fill missing gaps and incorporate feedback from the community at large.
Our last release, 0.6.0, expanded Dagster from local developer experience to a hostable product, allowing for scheduling, execution, and monitoring of pipelines in the cloud.
This release goes further, supporting pipelines with 100s and 1000s of nodes, deployable to modern, scalable cloud infrastructure, with dramatically improved monitoring tools, as well as other features.
Given this, 0.7.0 introduces the following:
- Revamped, Scalable Dagit A completely redesigned Dagit with a more intuitive navigation structure, beautiful look-and-feel, and massive performance improvements to handle pipelines with hundreds or even thousands of nodes.
- Execution Viewer Executing and historical runs within Dagit uses a new live-updating, queryable waterfall viewer. See below for a preview of the new UI:
https://media.giphy.com/media/Rhx6ujovXlvuKaLCGY/giphy.gif
- A Dagster-K8s library which provides the ability to launch runs in ephemeral Kubernetes Pods, as well as an early helm chart for executing pipelines.
- A Dagster-Celery library designed to work with K8s that provides global resource management using dedicated queues, and distributed execution of dagster pipelines across a cluster.
- Streamlined scheduler configuration and new backfill APIs and tools to help manage your scheduled workflows in production.
- A Dagster-Pandas integration that provides useful APIs for dataframe validation, summary statistics emission, and auto-documentation in dagit so that you can better understand and control how data flows through your pipelines.
- Redesigned documentation, examples, and guides to help flesh out the core ideas behind the system.
Warning
There are a substantial number of breaking changes in the 0.7.0 release. These changes effect the scheduler system, config system, required resources, and the type system. We apologize for the thrash, and thank you for bearing with us!
For more info on changes check out the following resources:
Changelog: https://github.com/dagster-io/dagster/blob/master/CHANGES.md
0.7.0 migration guide: https://github.com/dagster-io/dagster/blob/master/070_MIGRATION.md