github mage-ai/mage-ai 0.9.48
0.9.48 | The Boy and the Heron

latest releases: 0.9.71, 0.9.70, 0.9.68...
6 months ago

What's Changed

🎉 Exciting New Features

Incremental data integration in batch pipelines

🥳 Data integrations in batch pipelines now support incremental replication! You can read more here to get started!

by @tommydangerous in #4068

[Streaming] RabbitMQ Destination


Another community PR from @shrutimantri adds support for RabbitMQ as a streaming data sink. 🔥

Check it out today with your favorite streaming sources! You can find the configuration reference here.

by @shrutimantri in #4041

Chroma integration


Mage now has a ChromaDB IO Class, meaning you can use data loaders and exporters in your batch pipelines to read/write from Chroma sources. You can read more about configuration here or visit Chroma's site to learn more about their vector database.

by @matrixstone in #4017

Bookmark overrides


🎊 If you're creating a trigger on a data integration, you can now override bookmarks with your own custom values!

by @tommydangerous in #4073

SQL Block environment variable interpolation

For our fans of SQL blocks, you can now interpolate environment variables directly in your queries!

SELECT 
    '{{ env_var("ENV") }}' AS test
    , '{{ variables("test") }}' AS test2
    , {{ test }} AS test3

This should allow for much greater flexibility in pipelines with SQL!

by @tommydangerous in #4076

Additional upstream dependencies for dynamic children


Love dynamic blocks? 🤔 They dynamic children can now have additional upstream dependencies!

by @tommydangerous in #4104

Support caching block output in memory

Previously, pipelines with large Spark DataFrames faced out of heap space errors when persisting block outputs to disk. This PR allows the user to disable persisting output. The feature is only supported in standard batch pipeline (without dynamic blocks) for now.

cache_block_output_in_memory: true
run_pipeline_in_one_process: true

by @wangxiaoyou1993 in #4127

🐛 Bug Fixes

💅 Enhancements & Polish

New Contributors

Full Changelog: 0.9.46...0.9.48

Don't miss a new mage-ai release

NewReleases is sending notifications on new releases.