- Airflow: add some basic stats to the Airflow integration
Uses the statsd component that already exists in the Airflow codebase and wraps the section that emits to event with a timer, as well as emitting a counter for exceptions in sending the event.
- Airflow: add columns as schema facet for
Adds columns (if set) from
airflow.lineage.Tableinlets/outlets to the OpenLineage Dataset.
- DBT: add SQLSERVER to supported dbt profile types
Adds support for dbt-sqlserver, solving #2129.
- Spark: support for latest 3.5
Integration tests are now run on Spark 3.5. Also upgrades 3.3 branch to 3.3.3. Please note that
icebergare not supported for Spark
3.5at this time.
- Airflow: fix find-links path in tox
Fixes a broken link.
- Airflow: add more graceful logging when no OpenLineage provider installed
Recognizes a failed import of
airflow.providers.openlineageand adds more graceful logging to fix a corner case.
- Spark: fix bug in PathUtils' prepareDatasetIdentifierFromDefaultTablePath(CatalogTable) to correctly preserve scheme from CatalogTable's location
prepareDatasetIdentifierFromDefaultTablePathmethod would override the scheme with the value of "file" when constructing a dataset identifier. It now uses the scheme of the
CatalogTable's URI for this. Thank you @pawel-big-lebowski for the quick triage and suggested fix.