Release Highlights

We’re excited to announce the release of DataHub v0.9.0!

This minor release includes an upgrade to Java 11 and surfacing Column-Level Lineage support within the DataHub UI.

Here are some additional highlights:

User Experience

• Column-Level Lineage is now surfaced within the DataHub UI!
• Advanced Search now supports searching by Column-level details (i.e. name, description, tag, etc.), as well as complex AND/OR statements. For example:
  • Show results that match any filters
  • Show results that match all filters
  • Owner is either of Shannon or Mark
  • Oner is not Shannon nor Mark
  • Try it in demo here
• You can now add invite users and assign them to a default DataHub Role
• Improvements to site performance during the Browse experience

Developer Experience

• DataHub has been upgraded to Java 11!
• Improved tracking of GraphQL errors for bug resolution
• CorpUser and CorpGroup are now available via the Python SDK

Metadata Ingestion

• Automatically extract Column-Level Lineage from Snowflake & Looker sources
• dbt Meta Mapping is now supported at the Column Level - this means you can automatically extract Tags and Glossary Terms from your dbt model and surface them in DataHub

What's Changed

• fix(ingest): bigquery-beta - Getting datasets with biquery client by @treff7es in #6039
• feat(roles): add ability to invite users into a role by @aditya-radhakrishnan in #6015
• refactor(java11) - convert most modules to java 11 by @leifker in #5836
• docs(readme): Fixing broken article link by @davrax in #6042
• refactor(ingest): streamline pydantic configs by @hsheth2 in #6011
• docs(ingest): add example of dbt column_meta_mapping by @hsheth2 in #6038
• refactor(ingest): use aspect map in transformers by @hsheth2 in #6040
• feat(ui): Adding placeholder entity for DataPlatform by @jjoyce0510 in #6045
• feat(ingest): implement compression for CheckpointState by @alexey-kravtsov in #6007
• feat(advanced-search): adding select value modal by @gabe-lyons in #6026
• fix(ingest): bigquery-beta - Additional fixes for Bigquery beta by @treff7es in #6051
• feat(advanced search): adding advanced search filter component & prereqs for it by @gabe-lyons in #6055
• docs(ingest): add path spec examples for s3 by @mayurinehate in #6050
• fix(deps): metadata-io - remove parquet dependency by @shirshanka in #6046
• fix(ingestion): Tableau test case execution fix by @mohdsiddique in #6005
• feat(ingest): list referenced env variables in recipe by @hsheth2 in #6043
• fix(ingest): compat with mypy 0.981 by @hsheth2 in #6056
• fix(elasticsearch_index): create datahub_usage_event index where datahub_analytics_enabled set to false by @GyuhoonK in #5974
• docs(approval workflows): adding approval workflow docs by @gabe-lyons in #5896
• feat(retention): disable applying retention on bootstrap by @anshbansal in #6066
• fix(ingest): correct tableau browse paths by @hsheth2 in #6064
• fix(ingest): bigquery-beta - handling complex types properly by @treff7es in #6062
• docs: create SECURITY.md by @laulpogan in #6069
• fix(containers): show soft deleted status of containers by @gabe-lyons in #6072
• docs(ingest): clarify bigquery-beta multiproject setup by @hsheth2 in #6071
• chore(setup): change defaults for partitions by @anshbansal in #6074
• refactor(browse): Improving Browse Feature Performance by @jjoyce0510 in #6073
• feat(ingest): add column-level lineage support for snowflake by @mayurinehate in #6034
• feat(ingest): looker - support for simple column level lineage by @shirshanka in #6084
• fix(elastic-setup) Fixing env var logic by @pedro93 in #6079
• Revert "chore(setup): change defaults for partitions (#6074)" by @pedro93 in #6086
• fix(mae-consumer): fix regression on base64 encoding by @codesorcery in #6061
• fix(elasticsearch) Analytics indices creation on AWS ES by @tomas-kubin in #5502
• docs(ingest): note that Athena doesn't support lineage by @hsheth2 in #6081
• fix(ingest): alias for mssql-odbc source by @hsheth2 in #6080
• fix(ingest): presto-on-hive - Setting display name properly by @treff7es in #6065
• fix(schema filter): fix schema infinite rerender by @gabe-lyons in #6082
• feat(monitoring): track graphql errors in metrics by @szalai1 in #6087
• feat(advanced search): Add component to show all advanced search filters & add new filter by @gabe-lyons in #6058
• fix(ingest): bump lkml version by @hsheth2 in #6091
• fix(ingest): lookml - extract column correctly by @shirshanka in #6093
• feat(retention): change default policy, add API to apply retention by @anshbansal in #6088
• fix(lineage): fix missed casing in lineage registry by @gabe-lyons in #6078
• fix(ingest): bigquery-beta - Lowering a bit memory footprint of bigquery usage by @treff7es in #6095
• feat(ingest): remove hardcoded env variable default for cli version by @shirshanka in #6075
• docs: add information about mapping ports for datahub-gms by @shirshanka in #6092
• chore(deps): upgrade graphql-java deps to 19.0 by @shirshanka in #6099
• chore(deps): upgrade neo4j to 4.4.x by @shirshanka in #6101
• feat(docs): Improve documentation about Search by @szalai1 in #5889
• feat(ingest): add async option to ingest proposal endpoint by @RyanHolstien in #6097
• chore(deps): upgrade opentelemetry dependencies by @shirshanka in #6100
• refactor(recommendations): Bump default max recommendations count for Platforms by @jjoyce0510 in #6113
• feat(ingest): add Sandbox support by @rgudic in #6105
• fix(mae): use JAVA_TOOL_OPTIONS instead of JDK_JAVA_OPTIONS by @szalai1 in #6114
• feat(advanced-search): Complete Advanced Search: backend changes & tying UI together by @gabe-lyons in #6068
• feat(search): improved search snippet FE logic by @gabe-lyons in #6109
• feat(ingest): add CorpUser and CorpGroup to the Python SDK by @ttaubermarshall-stripe in #5930
• fix(ingest): hide deprecated path_spec option from config by @hsheth2 in #5944
• feat(posts): add posts feature to DataHub by @aditya-radhakrishnan in #6110
• fix(ingest): remove unused mysql golden file by @hsheth2 in #6106
• fix(ingestion): fix percent change computation in stale_entity_removal by @rslanka in #6121
• refactor(ingest): use pydantic utilities for NamingPattern by @hsheth2 in #6013
• fix(ingest): presto-on-hive - not failing on Hive type parsing error by @treff7es in #6118
• fix(ingest): ignore usage and operation for snowflake datasets withou… by @mayurinehate in #6112
• refactor(ingest): remove typing workarounds by @hsheth2 in #6108
• Added information about AUTH_OIDC_EXTRACT_GROUPS_ENABLED by @PrashantKhadke in #6120
• feat(lineage): show fully qualified task name in lineage UI by @gabe-lyons in #6126
• docs(tableau): adding a ingestion video by @shirshanka in #6124
• Sending "getting started" direct to quickstart by @laulpogan in #6125
• build: Update JNA for M1 Mac by @david-leifker in #6116
• fix(ingest): bigquery-beta - fix for missing key error if dataset list was empty by @treff7es in #6133
• fix(ingest): file - add configurability for counting all elements bef… by @shirshanka in #6136
• Worked on the feature to update group title by @Ankit-Keshari-Vituity in #6047
• fix(ingest): add trino package max version restriction by @hsheth2 in #6137
• test(KafkaEmitter): Enable ability to run test locally by @david-leifker in #6123
• fix(ingest): add column name quoting for approximate count distinct by @hsheth2 in #6107
• fix(ingestion): add fallback to trino by @IceS2 in #6044
• perf(search): temporarily disable fetching input fields for search results by @gabe-lyons in #6139
• feat(lineage) Add Column-Level to Lineage Visualization by @chriscollins3456 in #6138
• feat(tracking): add telemetry for frontend events by @aditya-radhakrishnan in #6129
• docs(approvals): update approval permission docs by @gabe-lyons in #6143
• fix(ingest): fetch workbook tags in workbooks graphql query by @mayurinehate in #6102
• fix(lineage) Fix possible null pointer exception in UpstreamLineagesMapper by @chriscollins3456 in #6147
• fix(ingest): bigquery-beta - Eliminate the need for data.read permission for table schema by @treff7es in #6146
• fix(lineage) Fix batching to ES for impact analysis by @chriscollins3456 in #6149
• feat(ingest/lookml): add support for local/remote dependencies by @hsheth2 in #6150
• fix(auth): fix login endpoint to respect session expiration env var by @aditya-radhakrishnan in #6151
• fix(impact analysis): fixing filtering on impact analysis + cypress tests by @gabe-lyons in #6152
• docs(favicon): add docs for customizing favicon by @gabe-lyons in #6155
• fix(ingest): bigquery-beta - ensure that status aspect is emitted for… by @shirshanka in #6154
• fix(ingest): bigquery - Fix syntax error in get_all_schema_tables_query by @hieunt-itfoss in #6159
• fix(ingest): allow snowflake profiling to work with geography type by @mayurinehate in #6162
• feat(ingest): support enabled flag for airflow config by @hsheth2 in #6089
• refactor(ingest): Tableau cleanup by @hsheth2 in #6131
• fix(ingest): bigquery-beta - turning sql parsing off in lineage extraction by @treff7es in #6163
• fix(ingest): allow hiding some fields from the schema by @hsheth2 in #6077

New Contributors

• @davrax made their first contribution in #6042
• @laulpogan made their first contribution in #6069
• @tomas-kubin made their first contribution in #5502
• @rgudic made their first contribution in #6105
• @ttaubermarshall-stripe made their first contribution in #5930
• @PrashantKhadke made their first contribution in #6120
• @david-leifker made their first contribution in #6116
• @IceS2 made their first contribution in #6044

Full Changelog: v0.8.45...v0.9.0