astronomer/astronomer-cosmos astronomer-cosmos-v1.16.0rc1

v1.16.0rc1

on GitHub

Pre-release candidate of Cosmos 1.16.0.

First release candidate. It adds 29 milestone PRs merged since the third alpha, mostly dbt 2 (dbt-oss) support and the dbt-oss CI jobs. The head branch was synced with main so this cut contains all of them. Please try it and report any problems before the 1.16.0 GA. See the release PR for full provenance, the complete 164-PR changelog, and the test plan: #3073.

Behaviour Changes

• Cosmos now passes the --vars and --args dbt flags as one-line flow YAML (for example {var: '2'}) instead of block YAML, because dbt 2 rejects block YAML that contains quoted strings, for example var: '2'. dbt 1.x parses both forms to the same values, but the flag text in task logs and rendered commands differs. See #3099.
• With dbt 2 (dbt-oss) installed in the Airflow environment and openlineage-integration-common 1.43.0 or older, local operators now emit datasets (assets); before, they emitted none. emit_datasets is enabled by default, so DAGs scheduled on the assets of dbt models can be triggered by dbt 2 runs; set emit_datasets=False to keep the previous behaviour. With openlineage-integration-common 1.44.0 or later, dbt 2 already emitted datasets and this does not change. On Airflow 2.10.5, a version where registering new datasets is only partially supported, emitting datasets can fail with FlushError: Can't flush None value found in collection DatasetModel.aliases. See #3111.
• ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt-oss, which writes manifest.json when the run ends, and dbt Fusion. The producer builds the URI from the database, schema and alias in the relation of the node event, and the manifest still wins when it has one. With emit_datasets=False nothing changes, and neither do the Kubernetes and GKE watchers. See #3129.

Features

• Add a dbt-oss extra (pip install astronomer-cosmos[dbt-oss]) by @pankajastro in #3130
• DbtDocsS3Operator, DbtDocsAzureStorageOperator and DbtDocsGCSOperator now upload the docs site written by dbt-oss, which has no catalog.json, with the content type of each file. The files uploaded for dbt 1 are unchanged by @pankajastro in #3119

Bug Fixes

• Serialize the --vars and --args dbt flags as one-line flow YAML, because dbt 2 rejects block YAML that contains quoted strings by @pankajkoti in #3099
• Local operators now emit inlets and outlets (datasets) when dbt 2 (dbt-oss) is installed in the Airflow environment and openlineage-integration-common is 1.43.0 or older (Airflow's constraints pin 1.27.0 with Airflow 2.10.5, 1.37.0 with 3.1.0 and 1.43.0 with 3.1.7). dbt 2 leaves args.profiles_dir out of run_results.json. Those library versions read it without a fallback, so parsing the OpenLineage events raised a KeyError and the task got no inlets or outlets (1.44.0 and later use the --profiles-dir lookup, and Cosmos does not change their run results). Cosmos now adds the profiles directory (--profiles-dir or DBT_PROFILES_DIR) to the run results of the task's temporary project before parsing, only when dbt did not write it. A dbt 2 executable used with a dbt 1.x environment is not covered. dbt Fusion used with a dbt 2 environment is covered by @pankajastro in #3111
• In InvocationMode.SUBPROCESS, Cosmos now reads the text output of dbt 2 (dbt-oss) for dbt tests. DbtTestLocalOperator now calls on_warning_callback with the warned tests in test_names and test_results, and the exception for a failing dbt test, from any local operator, names the failing tests, with the number of failed rows when dbt 2 reports it. Previously the parsers only knew the dbt 1.x lines (Done. PASS=.. WARN=N .. TOTAL=N, Warning in test <name>, Failure in test <name>), so on dbt 2 the warning count was 0, the callback never fired, and the exception had no test details. The Kubernetes-based test operators (Kubernetes, GKE and EKS) now also read the dbt 2 Summary: .. | N warn line, so their on_warning_callback fires by @pankajastro in #3108
• ExecutionMode.WATCHER now recognises the nodes that dbt 2 (dbt-oss) skips because an upstream node failed. dbt 2 does not emit SkippingDetails or LogSkipBecauseError for them, only MarkSkippedChildren, so their consumers ended SKIPPED and were never built after the upstream recovered on a retry. Cosmos now reads the MarkSkippedChildren event that dbt 2 emits for each node it skips and treats the node as it does for dbt 1, including the nodes below a failed test and below an excluded node by @pankajastro in #3124, see #2698 and #3023
• InvocationMode.DBT_RUNNER now matches dbt node statuses case-insensitively when it collects warnings. dbt 2 (dbt-oss) reports a source freshness warning as Warn, which did not match warn, so DbtSourceLocalOperator called on_warning_callback with empty test_names and test_results. dbt 2 source freshness results have no message, so their status is reported in test_results instead. dbt 1 is unchanged by @pankajastro in #3123
• ExecutionMode.WATCHER now emits the datasets of models, seeds and snapshots when the manifest has no URI for them, as with dbt-oss and dbt Fusion by @pankajastro in #3129

Others

• Enable and unskip more integration tests in the dbt-oss CI jobs by @pankajastro and @pankajkoti in #3090, #3093, #3097, #3098, #3100, #3102, #3106, #3107, #3110, #3114, #3115, #3116, #3118, #3120, #3121, #3122, #3125, #3126, #3127, #3128, and #3129
• Run the test workflows on ubuntu-24.04 by @pankajastro in #3134
• Dependency updates by @dependabot and @pre-commit-ci in #3085, #3086, #3094, and #3131

Milestone: Cosmos 1.16.0
Full changelog: astronomer-cosmos-v1.16.0a3...astronomer-cosmos-v1.16.0rc1