OpenLineage/OpenLineage 1.13.1

OpenLineage 1.13.1

on GitHub

Added

• Java: allow timeout for circuit breakers #2609 @pawel-big-lebowski
  Extends the circuit breaker mechanism to contain a global timeout that stops running OpenLineage integration code when a specified amount of time has elapsed.
• Java: handle DataSetEvent and JobEvent in Transport.emit #2611 @dolfinus
  Adds overloads Transport.emit(OpenLineage.DatasetEvent) and Transport.emit(OpenLineage.JobEvent), reusing the implementation of Transport.emit(OpenLineage.RunEvent). Please note: Transport.emit(String) is now deprecated and will be removed in 1.16.0.
• Java/Python: add GZIP compression to HttpTransport #2603 #2604 @dolfinus
  Adds a compression option to HttpTransport config in the Java and Python clients, with gzip implementation.
• Java/Python/Proxy: properly set Kafka message key #2571 #2597 #2598 @dolfinus
  Adds a new messageKey option to KafkaTransport config in the Python and Java clients, as well as the Proxy. This option replaces the localServerId option, which is now deprecated. Default value is generated using the run id (for RunEvent), job name (for JobEvent) or dataset name (for DatasetEvent). This value is used by the Kafka producer to distribute messages along topic partitions, instead of sending all the events to the same partition. This allows for full utilization of Kafka performance advantages.
• Flink: add support for Micrometer metrics #2633 @mobuchowski
  Adds a mechanism for forwarding metrics to any Micrometer-compatible implementation for Flink as has been implemented for Spark. Included: MeterRegistry, CompositeMeterRegistry, SimpleMeterRegistry, and MicrometerProvider.
• Python: generate Python facets from JSON schemas #2520 @JDarDagran
  Objects specified with JSON Schema needed to be manually developed and checked in Python, leading to many discrepancies, including wrong schema URLs. This adds a datamodel-code-generator for parsing JSON Schema and generating Pydantic or dataclasses classes, etc. In order to use attrs (a more modern version of dataclasses) and overcome some limitations of the tool, a number of steps have been added in order to customize code to meet OpenLineage requirements. Included: updated references to the latest base JSON Schema spec for all child facets. Please note: newly generated code creates a v2 interface that will be implemented in existing integrations in a future release. The v2 interface introduces some breaking changes: facets are put into separate modules per JSON Schema spec file, some names are changed, and several classes are now kw_only.
• Spark/Flink/Java: support YAML config files together with SparkConf/FlinkConf #2583 @pawel-big-lebowski
  Creates a SparkOpenlineageConfig and FlinkOpenlineageConfig for a more uniform configuration experience for the user. Renames OpenLineageYaml to OpenLineageConfig and modifies the code to use only OpenLineageConfig classes. Includes a doc update to mention that both ways can be used interchangeably and final documentation will merge all values provided.
• Spark: add custom token provider support #2613 @tnazarew
  Adds support for FQCN as spark.openlineage.transport.auth.type.
• Spark: enable timeout for circuit breakers #2609 @pawel-big-lebowski
  Implements within the circuit breaker an optional timeout that switches off the OpenLineage integration code.
• Spark: add custom token provider support #2613 @tnazarew
  Adds a TokenProviderTypeIdResolver to handle both FQCN and (for backward compatibility) api_key types in spark.openlineage.transport.auth.type.
• Spark/Flink: support YAML config files together with SparkConf & FlinkConf approaches #2583 @pawel-big-lebowski
  Adds support for config entries being provided by both YAML file and integration-specific configuration (SparkConf/FlinkConf). Allows each integration to have its own config entries.
• Spark/Flink: job ownership facet #2533 @pawel-big-lebowski
  Enables configuration entries specifying ownership of the job that will result in an OwnershipJobFacet being attached to job facets.

Changed

• Java: sync Kinesis partitionKey format with Kafka implementation #2620 @dolfinus
  Changes the format of Kinesis partitionKey from {jobNamespace}:{jobName} to run:{jobNamespace}/{jobName} to match the Kafka transport implementation.

Fixed

• Python: make load_config return an empty dict instead of None when file empty #2596 @kacpermuda
  utils.load_config() now returns an empty dict instead of None in the case of an empty file to prevent an OpenLineageClient crash.
• Java: render lombok-generated methods in javadoc #2614 @dolfinus
  Fixes rendering of javadoc for methods generated by lombok annotations by adding a delombok step.
• Spark/Snowflake: parse NPE when query option is used and table is empty #2599 @mobuchowski
  Fixes NPE when using query option when reading from Snowflake.