OpenLineage/OpenLineage 0.26.0

OpenLineage 0.26.0

on GitHub

Added

• Proxy: Fluentd proxy support (experimental) #1757 @pawel-big-lebowski
  Adds a Fluentd data collector as a proxy to buffer Openlineage events and send them to multiple backends (among many other purposes). Also implements a Fluentd Openlineage parser to validate incoming HTTP events at the beginning of the pipeline. See the readme file for more details.

Changed

• Python client: use Hatchling over setuptools to orchestrate Python env setup #1856 @gaborbernat
  Replaces setuptools with Hatchling for building the backend. Also includes a number of fixes, including to type definitions in transport.py and elsewhere.

Fixed

• Spark: support single file datasets #1855 @pawel-big-lebowski
  Fixes the naming of single file datasets so they are no longer named using the parent directory's path: spark.read.csv('file.csv').
• Spark: fix logicalPlan serialization issue on Databricks #1858 @pawel-big-lebowski
  Disables the spark_unknown facet by default to turn off serialization of logicalPlan.