What's Changed
🎉 Exciting New Features
🌊 Streaming: Base Class Overhaul + 8 New Destinations
This. is. huge. With a complete base class re-write, every IO destination is now a streaming destination.
That means you can stream to:
And any future destinations added as an IO base. Huge shutout to @wangxiaoyou1993 on the herculean effort!
by @wangxiaoyou1993 in #3623
👀 Recently viewed pipelines
Some frontend polish now allows you to see your Recently Viewed pipelines from the overview page— a nice touch!
by @tommydangerous in #3633
👨👩👧👦 Community: GCS Sensors
@pammusankolli deliver's his first contribution by adding a Google Cloud Storage sensor to check if a file exists in a given bucket! Thanks for the add— this will be super useful to our Google platform users!
by @pammusankolli in #3651
🧩 Support syncing Git submodules
A solid improvement to our Git Sync functionality, you can now sync submodules, too! Just check the Git Sync settings to enable the feature.
🔂 Add always on schedule interval
On user request, we've added an @always_on interval for scheduled triggers. Always on schedules will trigger the new pipeline run as soon as the latest pipeline run is completed.
Once a pipeline run ends, regardless of whether or not it failed or succeeded, it will start a new run. Let us know if you find that valuable!
🐛 Bug Fixes
- Lazy import dbt block and library by @wangxiaoyou1993 in #3608
- Fix invalid keyword argument error when requesting a list from Stripe by @jdvermeire in #3479
- Fix snowflake output column names by @wangxiaoyou1993 in #3612
- Fix charts when trying to change name by @tommydangerous in #3614
- Fix export_batch_data not accepting tags as a kwarg by @tommydangerous in #3619
- Handle secret key and uuid whitespace by @dy46 in #3621
- Fix materializing upstream outputs for dbt by @christopherscholz in #3617
- Fix memory leak in data integration pipelines by @wangxiaoyou1993 in #3641
- Initialize db connection session by @dy46 in #3609
- Fix auto generating mage_sources.yml for dbt upstream blocks by @wangxiaoyou1993 in #3647
- Pipelines list fixes by @johnson-mage in #3648
💅 Enhancements & Polish
- Add sortable block run columns by @johnson-mage in #3613
- Clarify components based on UTC time in Overview page by @johnson-mage in #3615
- Support interpolating pipeline variables in streaming configs by @wangxiaoyou1993 in #3616
- Scheduler trigger interval as environment variable setting by @PopaRares in #3591
- Pipeline runs table (for individual pipelines) updates by @johnson-mage in #3629
- Reduce dynamic child block at any level by @tommydangerous in #3634
- Apache Kafka - SASL_PLAINTEXT by @Senpumaru in #3643
- Support Kafka SASL_SSL SCRAM-SHA-512 authentication by @wangxiaoyou1993 in #3649
- Update search_path command by @dy46 in #3625
New Contributors
- @jdvermeire made their first contribution in #3479
- @PopaRares made their first contribution in #3591
- @Senpumaru made their first contribution in #3643
- @pammusankolli made their first contribution in #3651
Full Changelog: 0.9.28...0.9.30