OpenMetadata 0.13.0 Release Highlights

Data Insights and KPIs

Data Insights, a game-changing feature, has been introduced that transforms the passive approach to data into a collaborative project toward improved data culture. Data Insights aims to provide a single-pane view of all the key metrics to best reflect the state of your data. OpenMetadata gathers all the metrics related to the metadata that you are extracting. The entities created the types of entities and the data evolution over a period of time. Based on these metrics, we provide analytics to assess the gathered data.

Admins can define the Key Performance Indicators (KPIs) and set goals within OpenMetadata to work towards better documentation, ownership, and tiering.
These goals are based on entities and driven to achieve targets within a specified time. For example, Admins can set goals to have at least 60% of the entities documented, owned and tiered by the end of Q1 2023.

The data insights dashboard provides a quick glance at aspects like data ownership, description coverage, data tiering, and so on. Teams can view a timeseries report to track progress and monitor the health of your data. In addition to the metrics on data, Admins can view the aggregated user activity and get insights into user engagement and user growth. Admins can check for Daily active users and know how the tool is being used.

The Data Insights Report is emailed weekly, so that teams can assess their performance relative to the KPIs set at an organizational level to improve data culture on an ongoing basis.

Lineage

The lineage UI has been transformed to enhance user experience. Users can get a holistic view of an entity from the Lineage tab. When an entity is selected, the UI displays end-to-end lineage traceability for the table and column levels. Just search for an entity and expand the graph to unfold lineage. It’ll display the upstream and downstream for each node.

The Lineage Tab UI supports two-finger scrolling to zoom in or zoom out.

Data Quality

With the OpenMetadata UI, users can now create and deploy profiling workflows for the Datalake connector, which supports AWS S3 and GCS. In the next release, we’ll add the support to run tests as well as covering Azure ADLS.
Security
With the addition of the LDAP SSO in the current release, OpenMetadata supports nine SSOs, which includes Google, Azure, Okta, OneLogin, Auth0, Amazon Cognito, Keycloak, and custom OIDC. In the 0.12.1 release, support was added for basic authentication to sign up using a Username/Password.

OpenMetadata Roles and Policies treat Bots as a special user with access to all the APIs and entities, just like an Admin. Bots have been in use for ingestion to extract metadata, as well as for data profiler and so on. In the 0.13 release, we’ve created multiple bots to serve different scenarios. For example, Ingestion Bot, Lineage Bot, Data Quality and Profiler Bot.

Given the varying roles for specific bots, the policies and access control for bots has been redefined. Now, Bots can have their own policies. For example, the Ingestion Bot can create and update entities. The Profiler Bot can only update the profile of a table, and not have policies for any other entities or access to update table description, etc.

Advanced Search

OpenMetadata already supports advanced search syntax. Since it is syntax-driven, it’s not easy to use for all except advanced users. In the current release, a Syntax Editor has been introduced for advanced search with And/Or conditions that help discover assets quickly. A huge thank you to Cristian Osiac from Bloomberg for helping with this feature.

New Connectors

In the 0.13 release, we have introduced four new connectors:

Domo, a cloud-based dashboard service. The Domo Business Cloud is a low-code data app platform that takes the power of BI to the next level by combining all your data and putting it to work across any business process or workflow. OpenMetadata supports Domo as a Database, Dashboard, as well as a Pipeline service.

Hacktoberfest has been a complete success at the OpenMetadata community, with three connectors being developed as part of the event:

AWS SageMaker, a fully managed machine learning service, where data scientists and developers can quickly and easily build and train machine learning models, and then directly deploy them into a production-ready hosted environment.
AWS Kinesis, a cloud-based messaging service that allows real-time processing of streaming large amounts of data per second.
AWS QuickSight, a cloud-scale business intelligence (BI) service that allows everyone in the organization to understand the data by asking questions in natural language, exploring through interactive dashboards, or automatically looking for patterns and outliers powered by machine learning.

Big thanks and congratulations to Michael Zhou for developing AWS QuickSight and to Tushar Mittal for adding both AWS SageMaker and AWS Kinesis.
Several improvements have been made to the ingestion framework. In the 0.12.1 release, we shipped the ability to add a custom service type. Users can now develop their own connector and ingest it as with any other supported service! If you’d like to learn more about that, you can check out the demo!

Messaging Service Schemas

Major enhancements have been made to how data is extracted from Kafka and Redpanda Messaging services. Previously, OpenMetadata extracted all the Topics in the messaging queue and also connected to the Schema Registry to get the Schemas. These schemas were taken as one payload and published to OpenMetadata. We now parse Avro and Protobuf Schemas to extract the fields. Now, users can document each of these fields within a schema by adding descriptions and tags. Users can search based on the fields in the Schema of a Topic.

Other Changes

Soft deleted entities can be restored. Currently, only the ML Models are not supported.
Soft deleted teams can be restored. When restoring a soft deleted parent team, the child teams will not be restored by default.