What's Changed

🎉 Exciting New Features

💥 Add "Last run failed" filter to Pipeline Runs table

This feature adds a new filter option to the Pipeline Runs table, allowing users to easily view and filter pipeline runs that have failed in their most recent execution. This makes it simpler to identify and troubleshoot problematic pipelines.

by @johnson-mage in #4926

🛠️ Add workspace update features

This feature allows users to update the workspace settings (environment variables) and update the workspace main container config from the UI. Being able to update the container config will allow users to update the workspace image as well.

by @dy46 in #4951

🔄 Add upsert to MySQL, BQ, and Snowflake

Add ability to upsert data in MySQL, BigQuery, and Snowflake IO classes.
It can be achieved by specifying the following params in the export method:

            unique_conflict_method='UPDATE',
            unique_constraints=['col'],

by @dy46 in #4769

⚠️ Support alerts and limited retries in streaming pipeline

Streaming pipeline used to be retried automatically and infinitely.
With this feature, when retry_config is set in streaming pipeline's metadata.yaml at pipeline level, only retry the streaming pipeline with limited times and send alerts when pipeline fails.
Example retry config

retry_config:
  # Number of retry times
  retries: 3
  # Initial delay (in seconds) before retry. If exponential_backoff is true,
  # the delay time is multiplied by 2 for the next retry
  delay: 5
  # Maximum time between the first attempt and the last retry
  max_delay: 60
  # Whether to use exponential backoff retry
  exponential_backoff: true

by @wangxiaoyou1993 in #4902

🔑 Add setting to update user roles on login

This feature adds the UPDATE_ROLES_ON_LOGIN setting which will tell Mage to update the roles every time a user logs in using LDAP or OAuth (that supports roles mapping). If LDAP or OAuth returns no roles, the roles for the user will not be updated.

by @dy46 in #4905

🐛 Bug Fixes

• [dy] Catch JSON decode errors in mage targets by @dy46 in #4838
• [dy] Fix permission seed by @dy46 in #4846
• [jk] Replace Block reference with BlockFactory by @johnson-mage in #4850
• [jk] Fix block search add block error by @johnson-mage in #4855
• [xy] Fix spark dataframe read error. by @wangxiaoyou1993 in #4856
• [jk] Fix filters from being reset when items fetched by @johnson-mage in #4859
• [xy] Support cancelling jobs when using multiple replicas. by @wangxiaoyou1993 in #4861
• [xy] Fix updating pipeline content with unicode. by @wangxiaoyou1993 in #4864
• [xy] Fix dangling process when reloading webserver. by @wangxiaoyou1993 in #4866
• [jk] Multi-project platform project settings updates by @johnson-mage in #4878
• [xy] Refresh block run statuses before fetching crashed block runs. by @wangxiaoyou1993 in #4885
• Fix: add template for oracle export by @matrixstone in #4892
• [xy] Catch pipeline error in scheduler. by @wangxiaoyou1993 in #4911
• [dy] Create metadata.yaml file if it doesn't exist by @dy46 in #4884
• [xy] Fix dynamic block with great expectation block. by @wangxiaoyou1993 in #4913
• [dy] Fix duckdb sql by @dy46 in #4915
• [xy] Fix log based sync with uppercase columns. by @wangxiaoyou1993 in #4922
• [jk] Prevent renamed triggers in code from being duplicated by @johnson-mage in #4929
• [td] Lots of fixes and enhancements by @tommydangerous in #4937
• [xy] Skip checking memory in Windows. by @wangxiaoyou1993 in #4944
• [xy] Check type key in anyOf array objects. by @wangxiaoyou1993 in #4948
• [dy] Avoid creating duplicate admin user by @dy46 in #4949
• [xy] Fix downloading xlsx file. by @wangxiaoyou1993 in #4957
• [jk] Bugfix - Dependency tree layout by @johnson-mage in #4961
• [xy] Fix logging tags for streaming pipeline run. by @wangxiaoyou1993 in #4963
• [xy] Catch notification send exceptions. by @wangxiaoyou1993 in #4964
• [dy] Remove git dependency by @dy46 in #4959
• [jk] Cancel unsaved modified file dialogue by @johnson-mage in #4975
• [xy] Always set logger in different blocks separately. by @wangxiaoyou1993 in #4974
• [dy] Fix more git issues by @dy46 in #4980
• [xy] Fix k8s affinity parsing. by @wangxiaoyou1993 in #4985
• [xy] Fix encoding for block action object. by @wangxiaoyou1993 in #4987
• [xy] Fix None job manager. by @wangxiaoyou1993 in #4995
• [dy] Remove unused submodule by @dy46 in #4984
• [dy] Fix some git bugs by @dy46 in #4973
• Update jupyter-server-proxy to fix CVE-2024-28179 by @lqc in #4967
• Trino iceberg and delta lake array issues fix along with issue 4895 and other improvements by @sumanshusamarora in #4952
• [edw] Fix typo in Kafka streaming source by @edmondwinston in #4918
• [dy] Cast ambiguous columns to strings by @dy46 in #4857
• Bugfix/okta authorization url discovery by @moseswynn in #4810
• Update ads_insights_age_and_gender.json by @sumanshusamarora in #4891
• [dy] Use project uuid for workspace default permissions by @dy46 in #4843
• [dy] Update oauth token column type by @dy46 in #4933

💅 Enhancements & Polish

• [xy] Use executor type from pipeline level when block level executor is not set by @wangxiaoyou1993 in #4968
• [jk] Replace blue font with cyan in terminal by @johnson-mage in #4848
• [xy] Kill all jobs on scheduler crash. by @wangxiaoyou1993 in #4851
• [td] Update GX and other libraries to support higher versions of Pydantic by @tommydangerous in #4860
• [xy] Update snowflake destination datetime to use TIMESTAMP type. by @wangxiaoyou1993 in #4875
• [jk] Block output view option by @johnson-mage in #4883
• [jk] Only make specific files request with ArcaneLibrary by @johnson-mage in #4888
• Caching pipeline_project outside of loop to avoid multple lookups by @hugabora in #4887
• [jk] Remove pipeline_runs request appearing on different pages by @johnson-mage in #4890
• [xy] Add measurement to influxdb source. by @wangxiaoyou1993 in #4893
• [jk] Add visibility into project config loading errors by @johnson-mage in #4894
• [jk] Hide header actions on sign-in page by @johnson-mage in #4904
• [jk] Disallow numbers as first character of block names by @johnson-mage in #4914
• [jk] Triggers table UI fixes by @johnson-mage in #4934
• [xy] Support interpolating trigger name in k8s job name. by @wangxiaoyou1993 in #4947
• [xy] Only send failure notification when pipeline run status is failed. by @wangxiaoyou1993 in #4950
• [jk] Backfills table improvements by @johnson-mage in #4942
• [xy] Improve the performance of Postgres source metadata query. by @wangxiaoyou1993 in #4962
• [jk] Allow disabling pipeline autosave by @johnson-mage in #4988
• [xy] Support query pipelines with repo_path. by @wangxiaoyou1993 in #4993
• [Enhancement] Optionally Disable Wait Timeout in ECS Config by @moseswynn in #4970
• Modifying parser to handle pandas._libs.missing.NAType by @ashutosh-npsx in #4729
• [jk] Usage stats modal by @johnson-mage in #4863
• [edw] e2e: ensure example pipeline runs smoothly by @edmondwinston in #4771
• [jk] Autocomplete search results logging by @johnson-mage in #4858

✨ New Contributors

• @Wesseldr made their first contribution in #4927
• @ashutosh-npsx made their first contribution in #4729
• @lqc made their first contribution in #4967

Full Changelog: 0.9.68...0.9.70