v0.8.42
Highlights
User Experience
- Improved Search Experience - preview cards now display usage and freshness information
- Update to Schema History - incorporated Community feedback to remove “Blame” terminology
- Improved UI-Based Ingestion - easily configure metadata ingestion from Snowflake, BigQuery, Looker, and Tableau with an easy-to-follow form; YAML is still supported!
Developer/Community Experience
- Python 3.6 is no longer supported for ingestion – we expect this to impact fewer than 1% of DataHub users (based on PyPi download stats). Please upgrade to Python 3.7 or newer
- Update to GitHub Issue management - issues will be marked as “Inactive” after 30 days of no activity and will be automatically closed following an additional 30 days of inactivity
- We’ve updated our Slack Guidelines! Read them here
Metadata Ingestion
- You can now test your Snowflake connection via the CLI and UI-based Ingestion to ensure you have proper access levels required for general ingestion, profiling, and usage. We will be expanding this functionality to other cloud-based ingestion sources in upcoming cycles.
- Hard delete will now discover and remove soft deleted entities
- Resolved issue of assertion error with dbt stateful ingestion
Full Commit Log
What's Changed
- feat(quickstart,docs): updates for v0.8.41 by @anshbansal in #5409
- fix(ingest): ensure upgrade checks run async by @shirshanka in #5383
- fix(ingest): pass transport options to usage history looker api calls by @mayurinehate in #5417
- feat(quickstart): moving to official confluent images for m1 by @shirshanka in #5416
- fix(documentation) Fix erratic cursor in documentation editor bug by @chriscollins3456 in #5411
- feat(ui): Supporting enriched search preview + misc improvements by @jjoyce0510 in #5419
- chore: remove unnecessary modules from codebase by @shirshanka in #5420
- fix(ingest): extract usage for dashboards allowed by pattern by @mayurinehate in #5424
- fix(docker): fix kafka-setup command to support same capabilities as … by @shirshanka in #5428
- fix(protobuf): ownership fixes by @leifker in #5425
- fix(ui): add dataset qualifiedName parameter to lineage query by @alexey-kravtsov in #5427
- fix(glossary) Fix dropdown where disabled buttons are still clickable by @chriscollins3456 in #5430
- docs(bigquery): add changelog and unittest for profiling limits by @MugdhaHardikar-GSLab in #5407
- fix(siblings): fixing lineage fetching for siblings & sources by @gabe-lyons in #5415
- fix(ui): Fixing unreleased search preview bugs by @jjoyce0510 in #5432
- feat(ui): Adding Statistics Summary to Dataset + Dashboard Profiles by @jjoyce0510 in #5440
- feat(ingest): add test source connection feature, structured report file by @shirshanka in #5442
- fix(ingest/glue): handle error when generating s3 tags for virtual view tables by @timcosta in #5398
- feat(ingest): model - adding a small extension to support communicati… by @shirshanka in #5429
- fix(bigquery-usage): fix dataset name for sharded table by @MugdhaHardikar-GSLab in #5412
- feat(ingestion) Add new endpoint to test an ingestion connection by @chriscollins3456 in #5438
- feat(cli,build): remove deprecated variables GMS_HOST/_PORT by @anshbansal in #5451
- fix(search): make filters by default an empty list if null by @aditya-radhakrishnan in #5454
- fix(hive): add column comment as a column description by @MugdhaHardikar-GSLab in #5449
- feat(groups): add native groups concept to DataHub by @aditya-radhakrishnan in #5443
- fix(ingest): fix serialization of report to handle nesting by @shirshanka in #5455
- fix(tableau): fix tableau db error, add more logs by @mayurinehate in #5423
- build(deps): bump terser from 5.9.0 to 5.14.2 in /docs-website by @dependabot in #5448
- feat(doc): spark-lineage - Adding spark lineage configuration doc for Amazon EMR by @treff7es in #5459
- feat(schema-history): remove blame language for the schema history feature by @aditya-radhakrishnan in #5457
- Search header: Menu icon alignment by @Ankit-Keshari-Vituity in #5458
- build(deps): bump terser from 4.8.0 to 4.8.1 in /datahub-web-react by @dependabot in #5446
- feat(ingest): snowflake - basic test connection capability by @shirshanka in #5464
- fix(ingest/trino): Avoid exception if $properties table empty or not readable by @glinmac in #5447
- feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed by @treff7es in #5435
- fix(build): add base image with gradle wrapper cached by @anshbansal in #5467
- doc(bigquery): groups grants by requirements by @sgomezvillamor in #5468
- fix(docs,build): remove base image not needed, cleanup docs by @anshbansal in #5469
- feat(ui): Partial support for Chart usage by @jjoyce0510 in #5473
- fix(ingest): bigquery: multiproject profiling fix by @treff7es in #5474
- fix(ingest): kafka - revert deps back to < 1.9.0 by @shirshanka in #5476
- feat(docker): support multiplatform image for datahub-upgrade by @shirshanka in #5477
- feat(quickstart): experimental support for backup restore for quickstart by @shirshanka in #5418
- feat(dbt): updating source lineage logic by @gabe-lyons in #5414
- Ingestion: Added form in Big Query type to edit the queries. by @Ankit-Keshari-Vituity in #5431
- docs: fix docsearch config by @hsheth2 in #5479
- Search Results: Added checkbox option to select multiple results at once. by @Ankit-Keshari-Vituity in #5422
- feat(delete): hard delete deletes soft deleted entities by @anshbansal in #5478
- fix(docs): add missing closing marker for note section by @shirshanka in #5480
- fix(build): intermittent failure in github actions by @anshbansal in #5452
- feat(model, ingest): add user email in dashboard user usage counts by @mayurinehate in #5471
- feat(ingest): add support for capability report in snowflake test connection by @mayurinehate in #5472
- feat(build): automatically mark issues as stale to close inactive issues by @anshbansal in #5482
- fix(ingest): loosen confluent-kafka dep requirement by @hsheth2 in #5489
- refactor(ingest): cleanup importlib.import_module calls by @hsheth2 in #5490
- build(ingest): make gradle build less chatty by @hsheth2 in #5491
- fix(ingest): Fixing dbt trino datatypes by @aezomz in #5379
- refactor(ci): use custom action for checking codegen status by @hsheth2 in #5493
- feat(spark-lineage): Support ssl cert disable functionality by @MugdhaHardikar-GSLab in #5488
- docs(auth): fix link to point to new doc by @anshbansal in #5501
- docs(updating-datahub): add note for breaking change in looker usage … by @mayurinehate in #5499
- fix(ingest): cleanup unused flake8 noqa statements by @hsheth2 in #5492
- refactor(ci): refactor Docker build-and-push workflows by @hsheth2 in #5494
- docs(slack) Update to Slack guidelines by @maggiehays in #5504
- feat(cli): delete - make delete easier to use by @anshbansal in #5485
- fix(ingest): use temp dir for file generated during test by @hsheth2 in #5505
- feat(ui) Show Glossary and Domains header links to everyone by @chriscollins3456 in #5506
- Home Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in #5503
- fix(ingest): respect rest emitter timeout setting by @hsheth2 in #5508
- Search Page: Flickering Issue on search input field by @Ankit-Keshari-Vituity in #5515
- Ingestion: Added form to Looker and Tableau by @Ankit-Keshari-Vituity in #5487
- feat(identity): update azure and okta connectors to emit Origin aspects by @aditya-radhakrishnan in #5495
- feat(ui): Adding Search Select feature(frontend only) by @jjoyce0510 in #5507
- test(ingest): limit GMS retries in test by @hsheth2 in #5509
- fix(ingest)- airflow: update subdag check for compatibility with older Airflow versions by @vcs9 in #5523
- fix(ingest): fix unbound variable bug in cli ingest list-runs by @hsheth2 in #5527
- fix(ui) Display Term Group name properly in Recently Viewed by @chriscollins3456 in #5528
- feat(ingestion) Add frontend connection test for Snowflake by @chriscollins3456 in #5520
- fix(glossary) Fix Glossary success messages and sort Glossary by @chriscollins3456 in #5533
- feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms by @mohdsiddique in #4999
- feat(ingest): add deprecation warning for Python 3.6 by @hsheth2 in #5519
- docs(townhall) add past townhall agendas by @maggiehays in #5536
- feat(ingestion): add groups to ldap users by @alexey-kravtsov in #5470
- chore(issues): reduce time for issues to be marked stale and then closed by @aditya-radhakrishnan in #5537
- fix(ingestion) Set pipeline_name on UI recipes with forms by @chriscollins3456 in #5535
- fix(ui): Correctly hit /logOut on logout by @jjoyce0510 in #5538
- [analytics-tab] fix analytics tab config variable for gms by @salihcaan in #5529
- feat(ui): Support batch adding / remove tags from search lists. (Batch Actions part 2/7) by @jjoyce0510 in #5534
- fix(ingestionSource): improve error experience when ingestion source is in an inconsistent state by @RyanHolstien in #5522
- docs(schema history): Correct typo in schema history documentation by @skylersinclair in #5545
- fix(docker): Fixing dev docker and quickstart by @jjoyce0510 in #5550
- feat(ui): Support Batch adding and removing Glossary Terms (Batch Actions 3/7) by @jjoyce0510 in #5544
- feat(ci): test quickstart works by @anshbansal in #5518
- test(ingest): mark trino/hana tests as xfail due to flakes by @hsheth2 in #5549
- feat(ingestion/superset): add display_uri to config by @milimetric in #5408
- fix(quickstart): failure on a path not being present by @anshbansal in #5554
- fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests by @mohdsiddique in #5540
New Contributors
- @timcosta made their first contribution in #5398
- @glinmac made their first contribution in #5447
- @skylersinclair made their first contribution in #5545
- @milimetric made their first contribution in #5408
Full Changelog: v0.8.41...v0.8.42