astronomer/astronomer-cosmos astronomer-cosmos-v1.16.0a3

v1.16.0a3

on GitHub

Pre-release alpha of Cosmos 1.16.0.

Refresh of a2 with 15 additional milestone PRs merged since the second alpha. The head branch was synced with main again so this cut actually contains all of them. See the release PR for full provenance, the complete 135-PR changelog, and the test plan: #3073.

Behaviour Changes

• RenderConfig.exclude is now resolved against the full project graph instead of against the subset RenderConfig.select already narrowed. Graph operators traverse outwards from the nodes their selector matches, so when select had already dropped those root nodes the traversal had nothing to start from and shared ancestors or descendants were silently kept. For example, given a parent tagged test and its child, select=["+child"], exclude=["+tag:test"] previously rendered both parent and child, because +tag:test was evaluated after +child had already been applied; it now renders only child. This matches dbt, which evaluates both filters against the whole graph and subtracts afterwards. See #2976.
• ProjectConfig now keeps the models/seeds/snapshots relative paths it was configured with even when it has no dbt_project_path of its own, and LoadMode.CUSTOM falls back to them when the project root is supplied through RenderConfig/ExecutionConfig instead. Previously those relative paths were discarded in that split configuration and LegacyDbtProject silently crawled its defaults, so a project declaring e.g. models_relative_paths="custom/models" rendered no models at all; it now crawls custom/models as configured. Projects that set dbt_project_path on ProjectConfig are unaffected. See #2977.
• Under LoadMode.DBT_MANIFEST, RenderConfig.exclude and a YAML selector's exclude now apply to test nodes: tests matched by the exclude (for example resource_type:test or tag:<x>) are no longer rendered, where previously they were added back next to their model. With TestBehavior.AFTER_EACH, a model whose tests are all excluded is rendered as a plain task instead of a run and test group, so its task ids change (for example model_a.run becomes model_a_run). The tests of excluded models also leave DbtGraph.filtered_nodes. Test nodes also keep their own tags in addition to the tags of every parent, so select tag:<x> matches a test tagged <x> whose parent is not. An exclude now also removes every test that depends on an excluded node, as dbt's default indirect selection does, so a multi-parent test goes with any excluded parent while the other parents stay. Selecting by a test's own tag now renders that test where the test behavior allows it: a detached task for a multi-parent test when should_detach_multiple_parents_tests=True, or the single test task with TestBehavior.AFTER_ALL; previously nothing was rendered. A source whose tests are all excluded no longer counts as having tests, so with source_rendering_behavior=WITH_TESTS_OR_FRESHNESS it is rendered only if it has a freshness check. See #2981 and #3080.
• ExecutionMode.WATCHER consumers of a node that dbt skipped for an upstream failure now run the node on attempt 1 once the producer has finished, instead of failing and waiting one retry_delay for attempt 2. Previously every level of the graph below a failed model added one retry_delay. Test consumers whose model did not succeed do the same. This applies only when the task has retries set, and either watcher_dbt_retry_queue is unset or the consumer already runs on it. Such a node now gets retries + 1 real dbt runs instead of retries. See #3023 and #3047.
• On Airflow 3.3 and later, the ExecutionMode.WATCHER producer keeps its per-node XCom backup in the Airflow task store instead of an Airflow Variable when enable_watcher_reliable_retry is True, split across keys that roll over at 8 KB so each push rewrites only the open key. Variable writes look the key up in every configured secrets backend first, so with an external backend the producer task ran between about 3 and 6 times faster in our benchmark, depending on how many backends are configured. With False the Variable flush from the retry callback is unchanged. A producer that finds no backup on a retry logs a warning when the task store was in use. On Airflow 3.4 and later, clearing the producer from the UI or the REST API discards that backup unless "Keep task state" is ticked; see the watcher execution mode guide. See #2771 and #3076.
• Under TestBehavior.BUILD, a rendered dbt source that has tests is now a task group (<source>.source and <source>.test) instead of a single <source>_source task, so references to the old task id need updating. The source tests, which this mode never executed before, now run and can fail the DAG. See #1683 and #3037.

Features

• Add AIConfig (experimental, it may change without a deprecation period), an opt-in diagnosis of failed dbt tasks: on their last attempt, an LLM is asked for the root cause and a fix, which are added after the dbt error. It uses apache-airflow-providers-common-ai (pip install astronomer-cosmos[ai], for Python 3.11+ and Airflow 2.11+), works with ExecutionMode.LOCAL and ExecutionMode.VIRTUALENV, masks secrets, and lets you set what is sent and how much it can spend. See the AI config reference by @corsettigyg in #3018

Enhancements

• Improve ExecutionMode.WATCHER producer failure performance by keeping the per-node XCom backup in the Airflow task store on Airflow 3.3 and later by @tatiana in #3076

Bug Fixes

• Stop rendering tests removed by exclude under LoadMode.DBT_MANIFEST by @tatiana and @ege-arhan in #3080
• Resolve exclude graph selectors against the full project graph by @00yhj22-debug in #2976
• Keep ProjectConfig relative paths when the project root comes from RenderConfig by @00yhj22-debug in #2977
• Fix ExecutionMode.WATCHER consumers waiting one retry_delay per level below a failure by @biswasbiplob in #3047
• Surface the macro error on run-operation failure with dbt < 1.12 by @barking-code in #2986
• Skip dbt event callbacks for run-operation on dbt 2 by @pankajastro in #3089
• Fix a circular import when cosmos.constants is imported before Airflow by @tatiana in #3087
• Render source tests as a separate task under TestBehavior.BUILD by @developer-rpai in #3037
• Compare dbt node statuses case-insensitively in ExecutionMode.WATCHER and enable the watcher tests on dbt-oss, so a capitalised Warn or Error from dbt 2 is recognised by the source sensor and the default freshness callback by @pankajastro in #3088

Docs

• Document on_failure_callback access to dbt test failures by @pankajastro in #2846

Others

• Unskip dbt-oss integration tests for dbt ls and profile schema by @pankajastro in #3083
• Run the integration tests against dbt-oss on BigQuery by @pankajastro in #3046
• Run unit tests against dbt-oss in CI by @pankajastro in #3052

Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0a2...astronomer-cosmos-v1.16.0a3