datahub-project/datahub v0.8.34 on GitHub

Release Highlights

Developer Experience

DataHub Actions Framework is LIVE! The Actions Framework makes responding to real-time changes in your Metadata Graph easy, enabling you to seamlessly integrate DataHub into a broader events-based architecture. Check out the repo here
This release also introduces OpenAPI endpoints to post, get, and delete entities. Check out the usage guide here
Metadata Ingestion Source docs have a new look! We now have code-generated documentation to apply consistency in format and contents

User Experience

New! The Dataset Schema page now supports a “Blame View” to quickly understand how a field has evolved over semantic schema versions. You can find more info about how we compute versions here.

Ingestion Improvements

New! Now incubating the Apache Pulsar source
Update to Feast connector to support v0.18
Ongoing improvements to Snowflake external table support
Improvements to handling BigQuery audit log SQL queries
Miscellaneous Tableau fixes for lineage, browse path, non-embedded datasets

What's Changed

fix(cypress) - enable retries for failed tests to minimize flaking by @aditya-radhakrishnan in #4680
Deprecate an entity by @Ankit-Keshari-Vituity in #4633
fix(timeline): enhance schema field name change and removal support by @RyanHolstien in #4603
fix(cli): rest emitter should override config and env variables by @anshbansal in #4622
fix(docs): elasticsearch secret reference by @felixb in #4314
fix(mcl-processor): Remove unnecessary log.info by @dexter-mh-lee in #4686
fix(datahub-client): avoid parallel execution of metadat-io:test by @MugdhaHardikar-GSLab in #4685
docs(metadata-models-custom): add example script to show producing cu… by @shirshanka in #4681
fix(gms): Ensure Ordering by version when fetching next version by @arunvasudevan in #4696
fix(docker): Fix issue #4683 by @jjoyce0510 in #4697
feat(vulnerability): Upgrade spring libraries to latest version by @dexter-mh-lee in #4698
refactor(gms): EbeanAspectDao - make the orderBy clause explicitly ascending in getNextVersions by @jjoyce0510 in #4699
feat(gms): Entity change events v1 (Platform Event) by @jjoyce0510 in #4687
Redesign the login page by @Ankit-Keshari-Vituity in #4684
fix(snowflake): remove extra lineage edges in reports, change badly named config variable by @anshbansal in #4595
fix(bigquery): error due to not handling data properly by @anshbansal in #4702
fix(looker): Fix for Pydantic validation error for Looker TransportOptions on python 3.8 by @treff7es in #4705
fix(ingest) bigquery: Moving bigquery temporary credential deletion to atexit by @treff7es in #4701
fix(lineage): Fix lineage entity drawer height UI bug by @chriscollins3456 in #4707
feat(ingest) - update identity sources to add flags for masking sensitive work units by @aditya-radhakrishnan in #4711
fix(snowflake): deprecate config, update examples by @anshbansal in #4644
fix(glue): delete CatalogId parameter from get_jobs api call by @BoyuanZhangDE in #4646
fix(ui): Show deprecate button only for specific entity pages. by @jjoyce0510 in #4712
feat(ml): show custom properties for MLFeatureTable in UI by @maaaikoool in #4706
fix(glue): fix error for custom connector if ignore_unsupported_conne… by @mayurinehate in #4667
feat(ingest): add decimal128 custom type for mysql by @kevinhu in #4624
fix(policy): Use search to fetch all policies by @dexter-mh-lee in #4713
fix(transformers): add snapshot aspects from dataset into base_transf… by @shirshanka in #4719
Revert "fix(policy): Use search to fetch all policies" by @dexter-mh-lee in #4725
minor fix(metadata-ingestion): Add new schemas to python codegen by @jjoyce0510 in #4726
fix(ui): Display warning in UI when metadata service auth is disabled. by @jjoyce0510 in #4728
fix(timelineCli): fix naming for timeline cli by @RyanHolstien in #4729
fix(entity header): Fixes two issues in the EntityHeader - update UI and remove link by @chriscollins3456 in #4720
Revert "fix(timelineCli): fix naming for timeline cli (#4729)" by @jjoyce0510 in #4731
feat(cli): suppress stacktrace printing on configuration errors by @shirshanka in #4718
fix(cli): align default sink env variables across ingest and other cl… by @shirshanka in #4739
feat(ingest) dbt: Dbt query tag mapping and match template by @treff7es in #4744
fix(cli): telemetry - make config file processing more robust by @shirshanka in #4738
feat(react theming): stop homepage flicker for env-var based logos by @gabe-lyons in #4730
feat(Cassandra): add Cassandra implementation of EntityService by @xdl in #3286
fix(policies): Re-revert the policies fix + ingest documents directly to search by @dexter-mh-lee in #4733
feat(cli): Eagerly load datahub actions CLI commands by @jjoyce0510 in #4748
fix(ingest) bigquery: Fix BigQuery Datetime/Timestamp type column partition table profile bug by @sebkim in #4658
docs: add missing PR numbers by @anshbansal in #4742
fix(azure_ad): silently discard other Azure AD object types (#4693) by @cccs-eric in #4704
fix(datahub-frontend): OIDC discovery URL will not have NONE as auth_methods_supported by @chen4119 in #4710
fix(docs): fix links by @daha in #4703
feat(ingest): add Feast repository source by @danilopeixoto in #4094
feat(soft deletes): rephrasing soft delete banner by @gabe-lyons in #4753
feat(ebeans): Add metrics to track connection pool by @dexter-mh-lee in #4755
fix(AWS) When using aws_profile, grab temporary credentials from the session. by @Jiafi in #4751
feat(metadata-ingestion): Custom endpoint url and proxies in S3. by @pawel3275 in #4708
fix(tableau): miscellaneous tableau fixes for lineage, browse path, non-embedded datasets by @mayurinehate in #4724
doc: add warning for JDK by @anshbansal in #4761
fix(ui): fix expandedName for dataset by @mayurinehate in #4762
fix(ui): Users and Groups UI bug fixes by @ShubhamThakre in #4746
fix(azure_ad): make redirect and graph_url optional parameters and update docs by @aditya-radhakrishnan in #4754
docs(glue): clarify that table regex patterns should be fully-qualified by @aditya-radhakrishnan in #4747
fix(ml models): fix features tab by @gabe-lyons in #4769
fix(lint): lib upgrade caused by @anshbansal in #4773
fix(lineage) Filter dataset -> dataset lineage edges if data is transformed by @chriscollins3456 in #4732
fix(build): Fix breaking changes from GE 0.15.3 that are affecting our Python3.6 smoke_tests by @rslanka in #4779
fix(ingestion): Fixing how we eagerly import DataHub actions by @jjoyce0510 in #4784
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … by @shirshanka in #4786
feat(ingestion) Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job by @Jiafi in #4689
fix(snowflake): improve debug log for external tables by @anshbansal in #4772
feat(snowflake): add option to disable checking role grants by @anshbansal in #4760
fix(m1): tweak m1 preflight by @anshbansal in #4771
feat(ingestion): add Pulsar source by @vanmeete in #4721
fix(mae consumer): Fixes delete logic in MAE consumer by @pedro93 in #4790
feat(analytics): display glossary term percentage coverage by @aditya-radhakrishnan in #4782
refactor(gms): Removing unused source field by @jjoyce0510 in #4781
feat(versionedDataset): adds a versionStamp to timeline response & adds versionStamp param to dataset graphql by @RyanHolstien in #4727
fix(s3): improved handling for corner cases by @mayurinehate in #4774
fix(ingest): databricks - hive ingestion should not fail on table com… by @shirshanka in #4787
fix(ui ingest): Unschedule all sources on ingestion source refresh, fix delete not being enforced by @jjoyce0510 in #4792
feat(tracking) Configure whether mixpanel is enabled with env variable by @chriscollins3456 in #4768
feat(ingest): docs - overhaul source connector docs to make it code driven by @shirshanka in #4798
fix(docs): Fixing outdated control-center doc on policies.md by @jjoyce0510 in #4799
fix(ui): update default preview component with new ui design by @ShubhamThakre in #4783
feat(operation): display the reported time for last updated in the UI by @aditya-radhakrishnan in #4800
feat(blame) - add schema history blame UI by @aditya-radhakrishnan in #4793
fix(ingestion): Fix schema field type for avro logical types by @rslanka in #4801
Create sample_pii_glossary.yml by @mitchelllovessoftware123 in #4795
fix(ingestion): Fix presto_on_hive tests. by @rslanka in #4802
fix(bigquery): improve handling of extracted audit log sql queries by @vgaidass in #4735
fix(snowflake): get external tables when there is default namespace by @anshbansal in #4803
fix(snowflake): passing connect args should not cause failures by @anshbansal in #4764
fix(scrolling) Fixes scrolling and weird heights for embeddedListSearch across entities by @chriscollins3456 in #4805
fix(ui): update default preview card description text by @ShubhamThakre in #4796
fix(ui): preview card UI design update by @ShubhamThakre in #4808
fix(blame): make view blame prior to button work properly by @aditya-radhakrishnan in #4810
fix(docgen): fix failure count incrementing during doc generation by @shirshanka in #4806
fix(search) Fixes a UI issue so results and filters are always separated by @chriscollins3456 in #4811
feat(openapi): initial post,get, and delete endpoints for entities by @RyanHolstien in #4775
feat(protobuf) Adding deprecation support for datasets and fields by @leifker in #4634

New Contributors

@felixb made their first contribution in #4314
@chriscollins3456 made their first contribution in #4707
@sebkim made their first contribution in #4658
@chen4119 made their first contribution in #4710
@Jiafi made their first contribution in #4751
@pawel3275 made their first contribution in #4708
@vanmeete made their first contribution in #4721
@mitchelllovessoftware123 made their first contribution in #4795
@vgaidass made their first contribution in #4735

Full Changelog: v0.8.33...v0.8.34