Highlights
User Experience
- NEW: support for surfacing outcomes of dbt Tests in dataset entity pages (see it in action here)
- NEW: Improved navigation of dbt resources: dbt models and their associated warehouse tables are now merged into a unified entity (see it here). This will automatically be enabled for all newly ingested entities. To view this for entities you have already ingested, you will need to run a restore indices job.
- Improvement to Impact Analysis: When looking at the
Lineagetab, you can now easily toggle between “Upstream” and “Downstream” entities (try it out here)
Developer Experience
- NEW: Java Kafka Emitter – Use this when you want to decouple your metadata producer from the uptime of your datahub metadata server by utilizing Kafka as a highly available message bus
Metadata Ingestion
- NEW: Make bulk edits to your metadata via CSV (read more)
- Snowflake ingestion improvements: configure profiling to run only if they have been updated within the prior N days
- Managed ingestion update: removed need for sink block
What's Changed
- fix(ui-ingestion): update looker ingestion warning banner by @aditya-radhakrishnan in #5142
- chore: Bump Default UI Ingestion Version 0.8.38 by @jjoyce0510 in #5145
- feat(schema): support rendering schemas with
.in field names by @gabe-lyons in #5141 - feat(dbt): Platform instances for target platform by @skrydal in #5129
- feat(ingest): snowflake profile tables only if they have been updates… by @mayurinehate in #5132
- fix(airflow): fixes DeprecationWarning with hook-class-names by @sayakmaity in #5143
- feat(frontend): Parse JWT access token claims by @chen4119 in #5138
- fix(tokens): Using keyword search filters for ListAccessTokensResolver by @jjoyce0510 in #5154
- feat(ui) Update the max text length of Terms/Term Groups by @chriscollins3456 in #5162
- docs(policies): add info about Manage User Credentials by @aditya-radhakrishnan in #5157
- fix(restore-indices): Do not fail on MAE row count diff by @dexter-mh-lee in #5165
- fix(Kafka-setup): Make sure it doesn't fail when the new envs are not set by @dexter-mh-lee in #5168
- chore(deps): Bump Nimbus Jose JWT dependency by @pedro93 in #5158
- fix(recs): Verify that an entity exists before recommending by @jjoyce0510 in #5163
- fix(business glossary): setting properties to be empty if the node has no properties aspect by @gabe-lyons in #5166
- refactor(ui): Misc improvements to Dataset Assertions UI by @jjoyce0510 in #5155
- chore(guava): force version of guava in client jars per #5134 by @RyanHolstien in #5153
- feat(boot): Make Glossary Term Upgrade Async by @jjoyce0510 in #5164
- fix(frontend): Add iam auth jar to frontend by @dexter-mh-lee in #5171
- docs(features): update & clean up Features page by @maggiehays in #5175
- fix(glue): fix glue profiling config option by @kangseonghyun in #5178
- feat(upgrade) Check version when determining to run RestoreGlossaryIndices step by @chriscollins3456 in #5182
- fix(jaas): fixed auth.jaas.enabled option parsing by @alexey-kravtsov in #5179
- feat(ingestion): bigquery - Option to send usage queries as well as Operational metadata by @treff7es in #5151
- feat(build): changes to decrease build time, cancel runs in case of multiple commits by @anshbansal in #5187
- refactor(docs): Update Metadata Events Docs by @jjoyce0510 in #5173
- fix(ingest): If there is no manager for a LDAP user (example: system account) by @bda618 in #5180
- bug(ingest): correct case of sys views for mssql description populati… by @BALyons in #5186
- refactor(configs): Simplify Kafka Topic name configurations + docs by @jjoyce0510 in #5198
- feat(ingest): dbt - adding support for dbt tests by @shirshanka in #5201
- fix(cli): correct handling of env variables by @anshbansal in #5203
- feat(ci): split integration tests to reduce run time by @anshbansal in #5205
- feat(datahub-client): add java kafka emitter by @MugdhaHardikar-GSLab in #5074
- feat(graphql): add metrics capturing for graphql latency by @RyanHolstien in #5200
- test(ingestion): bigquery-usage - Adding tests for bigquery usage filters by @treff7es in #5195
- fix(ui): load monaco-editor as a dependency and not from a third party CDN by @Masterchen09 in #5189
- feat(cli): Add token parameter for sample ingestion by @pedro93 in #5160
- feat(lineage) Update Lineage tab and Impact Analysis feature by @chriscollins3456 in #5121
- fix(ingest): add missing ownership types by @afghori in #5209
- feat(ingestion) ldap: make ldap attrs keys configurable by @atulsaurav in #4682
- Remove unnecessary space from application.yml of GMS by @mmmeeedddsss in #5216
- fix(upgrade): fix upgrade when s3 path has = by @RyanHolstien in #5220
- feat(docs) Add and update docs for the new Glossary experience by @chriscollins3456 in #5211
- feat(glossary) Add empty state for the Business Glossary home page by @chriscollins3456 in #5217
- feat(bootstrap): add bootstrap step to clear out unknown aspect rows from the database by @RyanHolstien in #5148
- feat(ingest): adds csv enricher ingestion source by @aditya-radhakrishnan in #5221
- fix(build): pin confluent kafka dependency by @anshbansal in #5224
- fix(ingest): databricks - ingest structs correctly through hive by @shirshanka in #5223
- feat(dbt): add sibling association logic to associate dbt elements with their target systems by @gabe-lyons in #5190
- feat(tableau): use pagination for all connection queries by @mayurinehate in #5204
- Handling 404 page not found by @Ankit-Keshari-Vituity in #5227
- refactor(UI): Refactor Dataset Health Status by @jjoyce0510 in #5222
- fix(dbt-test): Inconsistency in assertions by @Santhin in #5214
- feat(ingest): remove need for sink block in UI based ingestion by @anshbansal in #5208
- fix(ingest): bigquery - Grouping date named tables at bigquery by @treff7es in #5230
- Add check for 0 rows when profiling datasets from s3 by @Jiafi in #5219
- [bug fix]: disabled create buttons by @xiphl in #5234
- fix(ingest): bigquery - Handling gracefully sql parser error in bq lineage by @treff7es in #5238
- fix(ingest): do not dump password by @anshbansal in #5235
- feat(ingest): dbt - improving dbt_meta mapping by @shirshanka in #5237
- fix(siblings): Force merged urn to viewed entity urn by @jjoyce0510 in #5247
- refactor(docs): Move CLI docs to root level by @jjoyce0510 in #5246
New Contributors
- @skrydal made their first contribution in #5129
- @sayakmaity made their first contribution in #5143
- @kangseonghyun made their first contribution in #5178
- @alexey-kravtsov made their first contribution in #5179
- @bda618 made their first contribution in #5180
- @BALyons made their first contribution in #5186
- @afghori made their first contribution in #5209
- @Santhin made their first contribution in #5214
Full Changelog: v0.8.38...v0.8.39