- Support custom transport types
Adds a new interface,
TransportBuilder, for creating custom transport types without having to modify core components of OpenLineage.
- Airflow: dbt Cloud integration
Adds a new OpenLineage extractor for dbt Cloud that uses the dbt Cloud hook provided by Airflow to communicate with dbt Cloud via its API.
- Spark: support dataset name modification using regex
It is a common scenario to write Spark output datasets with a location path ending with
/year=2023/month=04. The Spark parameter
spark.openlineage.dataset.removePath.patternintroduced here allows for removing certain elements from a path with a regex pattern.
- Spark: catch exception when trying to obtain details of non-existing table.
This mostly happens when getting table details on START event while the table is still not created.
- Spark: LogicalPlanSerializer
LogicalPlanSerializerto make use of non-shaded Jackson classes in order to serialize
LogicalPlans. Note: class names are no longer serialized.
- Flink: fix Flink CI
Specifies an older image version that succeeds on CI in order to fix the Flink integration.