datahub-project/datahub v0.8.45 on GitHub

Release Highlights

User Experience

Allow Term Groups to be the target of permissions
Customize browser favicon via REACT_APP_FAVICON_URL param
Some UX improvements for charts & dashboards entity pages to reduce confusion
Performance improvements on the lineage visualization
Search bar for dataset schema tab

Developer Experience

Add rest endpoint for restoring indices of a single entity (/aspects?action=restoreIndices)
Create new platform instances via CLI
Improved impact analysis performance due to an added caching layer
Support for Patch as seen in August 2022 town hall.

Metadata Ingestion

Introduces bigquery-beta source
Looker source memory usage dramatically reduced
Report memory usage during ingestion
Improve Tableau lineage
Usage statistics for Tableau
LookML can automatically clone your Git repository. LookML is now supported in UI-based ingestion.
dbt supports column-level meta mappings
Support for deletion & rollback of time series data
Upgrade to browse path forms

[see next page for list of commits]

What's Changed

fix(privileges) Add Term Groups as targetable entities for privileges by @chriscollins3456 in #5806
fix(javadocs): remove ampersand from pdl causing issue in doc generation for openapi by @RyanHolstien in #5808
chore(ingest): remove archived docs by @hsheth2 in #5793
feat(ingest): add rewrite option for metadata file check by @hsheth2 in #5763
feat(cli): add support for sampled reporting to keep logs manageable by @shirshanka in #5800
docs(refactor): Refactor Tags Feature Guide by @maggiehays in #5781
docs(feature-guide) Impact Analysis by @maggiehays in #5765
feat(theming): set custom favicon via env var by @gabe-lyons in #5810
test(smoke-test): check debug arg in executor requests by @hsheth2 in #5811
fix(ingest): bigquery-beta - Fixing dependencies by @treff7es in #5814
feat(ingest): looker - reduce memory requirements by @shirshanka in #5815
feat(restore-indices): add endpoint for restore indices, add basic check for graph by @anshbansal in #5805
fix(frontend): download node only when USE_SYSTEM_NODE is set to false by @szalai1 in #5817
doc: Make Airflow link clickable by @daha in #5803
feat(ingest):looker - reduce mem usage, misc reporting improvements by @shirshanka in #5823
feat(model, ingest): populate sizeInBytes in snowflake, fall back to table level profiling for large tables by @mayurinehate in #5774
chore(docker): make curl/wget commands quiet in docker by @hsheth2 in #5819
chore: cleanup references to the old ember app by @hsheth2 in #5797
fix(ingest): spark-lineage: Adding additional debug logs to spark lineage by @treff7es in #5772
fix(docker): add missing port mappings for non-neo4j quickstart by @hsheth2 in #5799
fix(ingest): looker - report dashboard scanning correctly by @shirshanka in #5829
feat(cli): report memory usage during ingest by @shirshanka in #5828
fix(ingest): presto-on-hive - Fixing mysql filter by @treff7es in #5825
docs(big query): add needed delete permission to list by @maaaikoool in #5826
chore(ingest): set isort combine_as_imports by @hsheth2 in #5820
fix(ingest): use AwsConnectionConfig instead of AwsSourceConfig by @hsheth2 in #5813
feat(ingest): looker test connection by @hsheth2 in #5768
feat(ingest): improve tableau lineage, workbooks query, fix pagination by @mayurinehate in #5756
fix(ingest): profiling - memory usage reduction by @shirshanka in #5830
feat(monitoring): enable JMX and OTEL for frontend pods by @szalai1 in #5834
fix(standalone-consumers): Exclude Solr from spring boot application config & make them run on M1 by @pedro93 in #5827
feat(hooks): Add toggle for enabling/disabling platform event hook by @pedro93 in #5840
feat(transformers): Add semantics & transform_aspect support in transformers by @mohdsiddique in #5514
feat(ci): auto label PRs by @anshbansal in #5839
feat(inputs): improving clarity on inputs for dashboards by @gabe-lyons in #5841
feat(ingest): add utility for converting MCEs to MCPs by @hsheth2 in #5812
chore(smoke): add additional log in smoke test by @hsheth2 in #5842
fix(ingest): fix doc generation import ordering issue with postgres by @hsheth2 in #5846
feat(docker) Adds Sasl support to base ingestion image by @pedro93 in #5855
fix(graphql) Fix null pointer exception when fetching entity aspect via graphql by @chriscollins3456 in #5857
fix(ingest): reporting should work with timestamps by @shirshanka in #5860
fix(patch-entity-registry): Remove exception for entities with key aspects. by @pghazanfari in #5831
fix(browse): Fixing browse path to remove requirement for simple name suffix by @jjoyce0510 in #5634
fix(ingest): bigquery - Fixing sharded regexp pattern config by @treff7es in #5861
perf(elastic search graph service): improving perf of lineage query by @gabe-lyons in #5858
chore(ingest): remove outdated GE compatibility hack by @hsheth2 in #5862
ci(ingest): test with python 3.10 by @hsheth2 in #5863
docs: improve doc generation, add better docs for snowflake, looker by @shirshanka in #5867
feat(ci): tweak auto-label globs by @anshbansal in #5849
fix(m1): preflight works with brew postgres@14 by @shirshanka in #5868
feat(smoke-tests) Make smoke tests use standalone consumers by @pedro93 in #5856
fix(domains): adding 10,000+ text when domain list caps out elastic count capacity by @gabe-lyons in #5838
docs(notifications): slack notification docs by @anshbansal in #5871
feat(docker): Update Dockerfiles to use java 11 runtime by @pedro93 in #5853
Scroll issue on Glossary related entity page by @Ankit-Keshari-Vituity in #5804
fix(ingest): include urns in rest sink failure logs by @hsheth2 in #5848
fix(docker): Bumps JRE 11 to latest by @pedro93 in #5875
feat(ingest): support reading config file from stdin by @hsheth2 in #5847
fix(ingest): remove dbt delete_tests_as_datasets option by @hsheth2 in #5865
fix(ingest): avrogen handling for missing fields with default values by @hsheth2 in #5844
refactor(ingest): add ALL_ENV_TYPES constant by @hsheth2 in #5866
feat(cli) Make docker compose quiet by @pedro93 in #5869
feat(datahub-protobuf): add support for shadow jar, publish by @shirshanka in #5882
feat(jars): better jar versioning for datahub-client, spark-lineage and protobuf by @shirshanka in #5883
fix(dev-docker): set right context for frontend dev build by @szalai1 in #5885
fix(ci): fix jar release action dependencies by @shirshanka in #5884
feat(schema) Add search filter to Schema tab by @chriscollins3456 in #5845
feat(ui) Add entity search input to shared folder by @chriscollins3456 in #5876
fix(ingest): datahub-api - move instantiation to the right config class by @shirshanka in #5878
feat(ingest): looker - improve defaults for usage extraction now that… by @shirshanka in #5893
fix(ingest): add missing trino types by @ms32035 in #5870
fix(ingest): remove dbt disable_dbt_node_creation and load_schema options by @hsheth2 in #5877
fix(frontend): forward Host header as X-Forwarded-Host by @codesorcery in #5816
feat(ingestion-ui) Sort ingestion sources by last execution time by @chriscollins3456 in #5807
docs(schema-history): update schema history docs by @aditya-radhakrishnan in #5894
docs(feature-guide) Refactor Domains feature guide by @chriscollins3456 in #5859
feat(charts & dashboards): improving chart & dashboard entity page rendering by @gabe-lyons in #5864
test(ingest): use pytest parameterization for dbt integration tests by @hsheth2 in #5879
refactor(ingest): move aspect maps to dedicated file by @hsheth2 in #5821
feat(ingest): make sink use type annotations by @hsheth2 in #5899
feat(ingest): add entity type inference to mcpw by @hsheth2 in #5880
feat(deletion & rollback): Server & Client side changes to support timeseries aspect deletion & rollback. by @rslanka in #4756
fix(ci): fix globs for auto-labeling PRs by @anshbansal in #5903
feat(ingest): minor changes in snowflake-beta source, add basic tests by @mayurinehate in #5910
feat(ingestion): schema inference for jsonlines in S3 by @hieunt-itfoss in #5725
refactor(gms): Refactoring util + entity client class locations by @jjoyce0510 in #5902
fix(ingest): support relative paths in lookml base_folder by @hsheth2 in #5914
feat(graphql) Add new 'entities' endpoint to batch get entities by urn by @chriscollins3456 in #5915
fix(ingest): handle lookml directory being in a symlink'd folder by @hsheth2 in #5916
fix(doc) - fix boolean property in json schema by @treff7es in #5919
fix(ingest): hide internal profiler.allow_deny_patterns from config by @hsheth2 in #5619
feat(ingestion-ui) Display ingestion sources in UI more dynamically by @chriscollins3456 in #5789
fix(docs): Feast recipe documentation by @danilopeixoto in #5917
fix(docker): skip zoneinfo backport on newer python versions by @hsheth2 in #5912
fix(favicons): update multiple favicons if present by @gabe-lyons in #5927
feat(ingest): lookml - support for git checkout by @shirshanka in #5924
refactor(ingest): simplify DefaultSQLParser alias by @hsheth2 in #5935
feat(ingest): support version option in datahub client get_aspect_v2 by @hsheth2 in #5934
fix(ingest): add dbt redshift type mappings by @hsheth2 in #5933
fix(ci): process older issues first by @anshbansal in #5926
fix(ingest): move git requirement into lookml deps by @hsheth2 in #5932
feat(elastic-setup): more verbose logging by @anshbansal in #5937
fix(docker) Add platform to docker-compose command by @firasomrane in #5683
fix(airflow): prioritize conn_type over id-based type inference by @hsheth2 in #5911
feat(ingestion): Refactor standard state-handling tasks into a common handler that are common across all stateful ingestion sources. by @rslanka in #5766
fix(docs): lookml - clean up links by @shirshanka in #5938
feat(search): Add support for Elasticsearch object field type by @justinas-marozas in #5891
feat(ingest): add ConfigEnum type by @hsheth2 in #5734
fix(transformer): fix invalid lastModified.actor entry in transformers by @mayurinehate in #5906
refactor(gms): Adding Java Entity Services by @jjoyce0510 in #5931
fix(ingest): fix type annotations on some pydantic fields by @hsheth2 in #5795
feat(docs) Adds guide on how to use personal access tokens by @pedro93 in #5873
feat(ingestion) Add more info to glue entities by @skrydal in #5874
feat(ingestion-ui) Add LookML form to managed ingestion UI by @chriscollins3456 in #5939
fix(ingest): Fix snowflake usage stateful ingestion backward compatibility check. by @rslanka in #5943
feat(ingest): skip ssh known_hosts verification for git clone by @hsheth2 in #5945
fix(ui) Make LookML deploy key a textarea and format key by @chriscollins3456 in #5946
feat(docs): lookml - add video for ui ingestion, remove secret step u… by @shirshanka in #5948
fix(ingest): add trailing newline to ssh keys by @hsheth2 in #5947
docs(airflow): show that rest host points at gms by @hsheth2 in #5913
feat(delta-lake): support delta tables from minio by @MugdhaHardikar-GSLab in #5758
refractor(snowflake): move snowflake-beta to certified snowflake source by @mayurinehate in #5923
docs: datahub_conn_id => conn_id in Airflow Integration by @GyuhoonK in #5920
fix(docs): fix loom embed by @shirshanka in #5956
fix(build): fix preflight script for m1 for lib postgresql by @shirshanka in #5957
docs(airflow) add back auth token info for airflow by @hsheth2 in #5960
feat: qualifiedName support + populating glue ARN by @skrydal in #5952
feat(elastic-setup): add better error handling by @anshbansal in #5963
docs(ingest): add example of ssl with mysql by @hsheth2 in #5954
feat(ingestion-ui) Use CLI version defined in recipe for test connection by @chriscollins3456 in #5959
feat(ingest): add support for aliases in plugin registry by @hsheth2 in #5958
feat(elasticsearch-setup): Add insecure option for curl by @BogdanAntoniu78 in #5887
feat(ci): exempt from stale label by @anshbansal in #5964
fix(ingest): continue validation of s3 path_specs even if platform is set by @mayurinehate in #5951
fix(ui) Filter search through schemas based on non-editable tags and terms as well by @chriscollins3456 in #5942
refactor(ingest): use typed objects in dbt ingestion by @hsheth2 in #5929
feat(ingest): bigquery-v2 - Add profiling feature by @treff7es in #5953
fix(quickstart): elasticsearch-setup script fails on curl by @shirshanka in #5975
feat(protobuf): support for custom platform, subtypes, misc improvements by @shirshanka in #5973
feat(dashboard): add subTypes aspect to dashboard entity by @Masterchen09 in #5843
fix(patch-entity-registry): Use AspectSpec to retrieve aspect class in order to support custom entity ingestion via patch mechanism. by @pghazanfari in #5936
fix(ingest): handle uname missing on windows by @hsheth2 in #5981
ci: reduce pip backtracking in airflow plugin by @hsheth2 in #5982
fix(ingest): pin great-expectations dependency to fix profiling error by @mayurinehate in #5980
refactor(looker): Migrating to new browse path format by @jjoyce0510 in #5688
feat(ingest): dbt column-level meta mappings + add_terms operation by @hsheth2 in #5970
fix(datahub-ranger-plugin): add support to publish jars by @shirshanka in #5983
feat(docs): Adds docs on K8s scheduled ingestion by @pedro93 in #5984
feat(ingest): presto-on-hive stateful, catalog and pascalcase subtype config by @aezomz in #5890
fix(kafka-setup) Bump kafka version to existing version by @pedro93 in #5995
Small UI bugs by @Ankit-Keshari-Vituity in #5886
feat(data platform) adding data platform indexing & select platform modal in frontend by @gabe-lyons in #5988
Worked on the pop-over of matched entity by @Ankit-Keshari-Vituity in #5851
refactor(docs): Updating GraphQL documentation with fixes, and more examples by @jjoyce0510 in #5989
docs(upgrading-datahub): Add docs for upgrading-datahub v0.8.44 by @jjoyce0510 in #5998
chore(docs-website): bump docusaurus to v2.1.0 by @hsheth2 in #5792
docs: remove wip UI ingestion doc by @hsheth2 in #6000
feat(cli): extend put command to support platform creation by @shirshanka in #5990
feat(ingest): presto-on-hive - Description and hive view support by @treff7es in #5993
fix(ingest): encode reserved characters when creating dataset urn by @mayurinehate in #5977
feat(ingestion): Tableau dashboard & chart usage stats by @mohdsiddique in #5999
feat(ingestion): Documentation on adding stateful ingestion use-cases to new sources by @rslanka in #5985
refactor(ingest): streamline two-tier db config validation by @hsheth2 in #5986
feat(ui): Add Nullable label to Schema Render by @mkamalas in #5909
refactor(frontend): Misc frontend improvements by @jjoyce0510 in #6012
fix(docs-site): make banner a bit smaller by @hsheth2 in #6016
feat(ingest): add --no-pull-images option to docker quickstart by @hsheth2 in #6017
docs(users): move changing default user up by @anshbansal in #6020
Worked on the dynamic column width of stats table by @Ankit-Keshari-Vituity in #5996
Worked to hide the recent search from search bar by @Ankit-Keshari-Vituity in #6021
docs: fix a markdown typo by @ltxlouis in #6019
fix(ingest): lookml - make explore parsing more robust by @shirshanka in #6028
fix(ingest): better error handling in glue by @hsheth2 in #6030
fix(gms): Write back lineage search results to in-memory cache bound to feature flag by @RyanHolstien in #6006
fix(docs-site): style custom announcement bar by @jeffmerrick in #6022
fix(ownership-differ): standardise ownership change event payload by @ngamanda in #5967
feat(ingestion): Add fail-safe stale entity removal via configurable 'fail_safe_threshold' param. by @rslanka in #6027
refactor(ingest): move common host_port validation by @hsheth2 in #6009
feat(ingest): aws - support extra args to role config by @hsheth2 in #6031
fix(frontend): refactoring AuthServiceClient by @aditya-radhakrishnan in #6029
refactor(gms): Improving JWT parsing logic by @jjoyce0510 in #6025
fix(ingest): support snowflake-beta extra compatibility by @hsheth2 in #6032
fix(platform): Add aws-secretsmanager-jdbc driver in dependencies by @atul-chegg in #5968
feat(patch): initial support of json patch style semantics in MCPs by @RyanHolstien in #5901
fix(ingest): bigquery-beta - Project name quote and graceful lineage/usage failures by @treff7es in #6035

New Contributors

@pghazanfari made their first contribution in #5831
@codesorcery made their first contribution in #5816
@hieunt-itfoss made their first contribution in #5725
@firasomrane made their first contribution in #5683
@GyuhoonK made their first contribution in #5920
@BogdanAntoniu78 made their first contribution in #5887
@ltxlouis made their first contribution in #6019
@atul-chegg made their first contribution in #5968

Full Changelog: v0.8.44...v0.8.45

datahub-project/datahub v0.8.45 DataHub v0.8.45 on GitHub

Release Highlights

User Experience

Developer Experience

Metadata Ingestion

What's Changed

New Contributors

datahub-project/datahub v0.8.45
DataHub v0.8.45

on GitHub