What's Changed

🎉 Exciting New Features

✨ Global data products are now available in Mage! 🎉

A data product is any piece of data created by 1 or more blocks in a pipeline. For example, a block can create a data product that is an in-memory DataFrame, or a JSON serializable data structure, or a table in a database.

A global data product is a data product that can be referenced and used in any pipeline across the entire project. A global data product is entered into the global registry (global_data_products.yaml) under a unique ID (UUID) and it references an existing pipeline. Learn more here.

by @tommydangerous in #3206

✨ Add block templates for MSSQL 🤘

We now have some awesome block templates for our MySQL users out there!

Check them out:

by @wangxiaoyou1993 in #3294

✨ Support sharing memory objects across blocks

In a pipeline's metadata.yml, you can now configure running pipelines in a single process:

blocks:
...
run_pipeline_in_one_process: true
...

You may now also:

• Define object once and make it available in any block in the pipeline via keyword arguments kwargs['context']
• Pass variables between blocks in memory directly

by @wangxiaoyou1993 in #3280

🐛 Bug Fixes

• Added type check on unique_constraints by @Luishfs in #3282
• Proper filtering in multi-workspace pipeline runs & subproject role creation by @dy46 in #3283
• Add column identifier for Trino by @dy46 in #3311
• Fix mage run cli command with variables by @wangxiaoyou1993 in #3315
• Fix async_generate_block_with_description upstream_blocks param by @matrixstone in #3313
• Removed unused config in Salesforce destination by @Luishfs in #3287
• Fix BigQuery bookmark datetime comparison by @wangxiaoyou1993 in #3325
• Improve secrets handling by @dy46 in #3309
• Adding oracledb lib to mage-ai by @Luishfs in #3319

💅 Enhancements & Polish

• base.py: Add XML support for file read/write in S3, GCP, and other cloud storage providers by @adelcast in #3279
• Save logs scroll position for trigger logs by @johnson-mage in #3285
• Update error logging for block runs by @dy46 in #3234
• Browser-specific dependency graph improvements by @johnson-mage in #3300
• Updated Clickhouse destination by @Luishfs in #3286
• Make pipeline run count pagination consistent by @johnson-mage in #3310
• Add new base path environment variables by @dy46 in #3289
• Add pipeline created_at property by @johnson-mage in #3317
• Add pagination to pipeline run block runs page by @johnson-mage in #3316
• Added all columns as replication key by @Luishfs in #3301
• Improve streaming pipeline stability by @wangxiaoyou1993 in #3326
• Added test connection to google_ads source by @Luishfs in #3322
• Open and edit any file, including prefix . by @tommydangerous in #3255
• Add namespace to K8sBlockExecutor config by @mattcamp in #3246
• Add a community Code of Conduct by @mattppal in #3318
• Add documentation for adapting an existing tap by @mattppal in #3290

New Contributors

• @adelcast made their first contribution in #3279
• @mattcamp made their first contribution in #3246

Full Changelog: 0.9.14...0.9.16