OpenLineage/OpenLineage 1.51.0

OpenLineage 1.51.0

on GitHub

Added

• Dbt: Capture dbt model meta/config values #4653 @mobuchowski
  Capture per-model config values (e.g. materialized, access, owner, group) and the user-defined meta map from the dbt manifest into a new dbt_model dataset facet, attached in both the legacy/local and structured-logs dbt processors.
• Spark: Add support for open source Unity Catalog (Spark 4.0) #3674 @arturowczarek
  Add a catalog handler for the open source Unity Catalog's UCSingleCatalog Spark catalog so output dataset facets are produced when using OSS Unity Catalog, which cannot reuse the existing DeltaHandler implementation.
• Spec: Extend jobTypeJobFacet to capture event emission lifecycle #4592 @jakub-moravec
  Extend the Job Type facet so producers, particularly streaming producers that emit a single event covering a time period rather than START/RUNNING/COMPLETE events, can document the lifecycle of the OpenLineage events consumers should expect.

Fixed

• Spark: Fix missing output lineage for insertInto with append mode #4676 @mishrasangeeta87
  Fix a bug where df.write.mode("append").insertInto(table) did not emit an output dataset, by extracting the target table directly from the AppendData logical plan, aligning append-mode handling with overwrite-mode.
• Spark: Fix Unity Catalog symlink metadata on Databricks #4660 @mishrasangeeta87
  Build Databricks Unity Catalog table symlinks using the fully qualified table name and a unity-catalog namespace instead of one derived from the underlying storage path, giving lineage consumers stable, catalog-qualified identifiers.
• Client/Java: Change shaded paths to avoid conflicts when using composite transport #4652 @tnazarew
  Relocate shaded classes per-transport (e.g. io.openlineage.client.transports.gcs.shaded.x) instead of to a single shared shaded package, fixing Google library version conflicts when combining gcs and gcplineage transports in a composite transport.
• Client/Java: Normalize abfs/wasb to abfss/wasbs in ObjectStorageDatasetExtractor #4661 @arturowczarek
  Treat abfs:// and wasb:// URIs the same as their TLS variants (abfss/wasbs) when deriving the object-storage namespace, since they point at the same underlying files and only differ in transport.
• Client/Java: Stop swallowing actual cause when using CompositeTransport #4666 @mobuchowski
  Preserve the underlying exception instead of only its message when a CompositeTransport emit fails, so the real cause is visible instead of being lost.
• Common: Handle trailing command options #4624 @hcthakur2004
  Avoid indexing past the end of command-line arguments when a trailing option has no value.
• Common: Preserve nullable chain input #4627 @hcthakur2004
  Avoid mutating the caller-provided chain in get_from_nullable_chain, keeping it reusable after lookup.
• GreatExpectations: Preserve datasource URL ports #4679 @hcthakur2004
  Preserve the port on a datasource URL when Great Expectations strips credentials from it.
• Python: Isolate Datadog config defaults #4586 @hcthakur2004
  Replace shared, mutable Datadog config dictionary defaults with per-instance factories so retry and async transport rule defaults aren't accidentally shared across instances.
• Python: Support file URI transport paths #4656 @hcthakur2004
  Normalize local file:// URIs before opening file transport paths so events can be appended through a file:// log path.
• Spark: Fix Iceberg handler detection to probe SparkCatalog instead of Catalog #4665 @arturowczarek
  Detect Iceberg by the presence of the iceberg-spark SparkCatalog class rather than iceberg-core's Catalog, so the Iceberg handler no longer activates in environments where iceberg-core is on the classpath without iceberg-spark.