mage-ai/mage-ai 0.9.10 on GitHub

What's Changed

🎉 Exciting New Features

🤖 Create blocks and documentation using LLMs

Block Creation

Document Generation

From the following PRs:

Generate block using AI by @tommydangerous in #3095
Generate documentation using AI by @tommydangerous in #3105
Generate block template based on block description by @matrixstone in #3029
Create model and API endpoint for global data products by @tommydangerous in #3135

❄️ Enable batch upload for Snowflake destination

Leveraging write_pandas in the snowflake-connector-python library, this feature enhances the speed of batch uploads using Snowflake destinations 🤯 by @csharplus in #2896

Auto-delete logs after retention period

Now, Mage can auto-remove logs after your retention period expires!
Configure retention_period in logging_config:

logging_config:
    retention_period: '15d'

Run command to delete old logs:

mage clean-old-logs k8s_project

by @wangxiaoyou1993 in #3139

MongoDB destination support (data integration)

MongoDB is now supported as a destination! 🎉 by @Luishfs in #3084

Pipeline-level concurrency

It's now possible to configure concurrency at the pipeline level:

concurrency_config:
  block_run_limit: 1
  pipeline_run_limit: 1

Doc: https://docs.mage.ai/design/data-pipeline-management#pipeline-level-concurrency
by @wangxiaoyou1993 in #3112

🧱 New add-block flow

Mage's UI has been improved to feature a new add-block flow! by @tommydangerous in #3094, #3074, & #3106

Custom k8s executors

Mage now support custom k8s executor configuration:

k8s_executor_config:
  service_account_name: mageai
  job_name_prefix: "{{ env_var('KUBE_NAMESPACE') }}"
  container_config:
    image: mageai/mageai:0.9.7
    env:
    - name: USER_CODE_PATH
      value: /home/src/k8s_project

by @wangxiaoyou1993 in #3127

Custom s3 `endpoint_url` in logger

You can now configure a custom endpoint_url in s3 loggers, allowing you to customize how messages are displayed!

logging_config:
  type: s3
  level: INFO
  destination_config:
    bucket: <bucket name>
    prefix: <prefix path>
    aws_access_key_id: <(optional) AWS access key ID>
    aws_secret_access_key: <(optional) AWS secret access key>
    endpoint_url: <(optional) custom endpoint url>

by @wangxiaoyou1993 in #3137

Render text/html from block output

Text and HTML from block output is now rendered!

by @dy46 in #3079

Clickhouse data integration support

Clickhouse is now supported as a integrations destination! by @Luishfs in #3005

Custom timeouts for ECS tasks

You can now set custom timeouts for all of your ECS tasks! by @wangxiaoyou1993 in #3144

Run multiple Mage instances with the same PostgreSQL databases

A single Postgres database can now support multiple Mage instances ✨ by @csharplus in #3070

🐛 Bug Fixes

Clear pipeline list filters when clicking Defaults by @johnson-mage in #3092
Fetch Snowflake role by @mattppal in #3100
Add service argument to OracleDB data loader by @mattppal in #3032
Fix pymssql dependency by @dy46 in #3114
Include runs without associated triggers in pipeline run count by @johnson-mage in #3118
Fix positioning of nested flyout menu 4 levels deep by @johnson-mage in #3125
Batch git sync in pipeline_scheduler by @dy46 in #3102
Only try to interpolate variables if they are in the query by @dy46 in #3142
Fix command for running streaming pipeline in k8s executor. by @wangxiaoyou1993 in #3093
Fix block flow bugs by @tommydangerous in #3094
Catch timeout exception for test_connection by @dy46 in #3143
Fix showing duplicate templates for v1 by @tommydangerous in #3128
check_status method bug fix to look at pipelines ran between specific time period by @sumanshusamarora in #3115
Fix db init on start by @tommydangerous in #3080
Fix logic for making API requests for block outputs or analyses that were throwing errors due to invalid pipeline_uuids by @johnson-mage in #3090

💅 Enhancements & Polish

Decrease variables response size by @johnson-mage in #3097
Cancel block runs when pipeline run fails by @dy46 in #3096
Clean up code in VariableResource method by @johnson-mage in #3099
Import block function from file for test execution by @dy46 in #3110
Misc UI improvements by @johnson-mage in #3133
Polish custom templates by @tommydangerous in #3120
Throw correct exception about io-config by @wangxiaoyou1993 in #3130
Clarify input for dbt profile target if a default target is not configured by @johnson-mage in #3077
Support nested sampling on output by @wangxiaoyou1993 in #3086
Add API endpoint for fetching OAuth access tokens by @tommydangerous in #3089

New Contributors

@sumanshusamarora made their first contribution in #3115

Full Changelog: 0.9.8...0.9.10

mage-ai/mage-ai 0.9.10 0.9.10 | Haunted Mansion on GitHub