datahub-project/datahub v0.9.4

[Known Issues] DataHub v0.9.4

on GitHub

Known Issues

In this release, the version of our OIDC SSO library was majorly upgraded. There is an issue with how the newer version of the library interacts with OIDC providers. We have addressed this issue in v0.9.5. We recommend avoiding upgrading to this version if your organization is actively using OIDC to manage user authentication.

What's Changed

• chore(): Updating default CLI version, update updating-datahub.md by @jjoyce0510 in #6590
• fix(ingest): profiling - Profiling failed if column cardinality threw an error by @treff7es in #6582
• fix(actions): add missing datahub-gms-protocol env var by @shirshanka in #6593
• fix(ingest): restrict snowflake-connector-python dependency by @mayurinehate in #6594
• feat(ingest/bigquery): avoid creating/deleting tables for profiling by @hsheth2 in #6578
• fix(ingest): unify emit interface by @hsheth2 in #6592
• fix(security): security version updates by @david-leifker in #6602
• docs: remove Kafka Streams from documentation by @maver1ck in #6596
• refactor(ui): Improving Kafka UI Ingestion Form, Create Domain, Create Secret Modals by @jjoyce0510 in #6588
• fix(ingest): clarify tableau auth error messages by @hsheth2 in #6600
• docs(graphql): fix deleteTest "Create"->"Delete" by @nickwu241 in #6574
• fix(gms/startup): remove set -x from start.sh by @timcosta in #6589
• feat(sql): Add SQL index on createdon field by @pedro93 in #6522
• feat(ml model): updating view of ml model feature list by @gabe-lyons in #6576
• fix(ingest/bigquery): ignore complex types from profiling by @treff7es in #6613
• feat(ingest): add external url for snowflake objects by @mayurinehate in #6580
• chore(ingest): bump and pin mypy by @hsheth2 in #6584
• fix(ingest): only require github_info for lookml and not looker by @hsheth2 in #6608
• docs(ingest): add airflow docs that use the PythonVirtualenvOperator by @hsheth2 in #6604
• fix(ui) Fix double scroll in embedded list search sections by @chriscollins3456 in #6618
• feat(ingest): print detailed GMS error messages by @djordje-mijatovic in #6519
• Townhall agenda wikimedia by @maggiehays in #6622
• fix(analytics): skip ListDomains if user cannot manage domains and have only one loading message by @aditya-radhakrishnan in #6624
• feat(quickstart): add support for passing thru env vars needed by Sla… by @shirshanka in #6591
• docs(actions): slack, teams by @shirshanka in #6632
• fix(logging): Remove lombok as source of slf4j-api by @david-leifker in #6616
• docs: add links from main README to slack, teams actions by @shirshanka in #6633
• feat(ingest): Support config variable for specifying a direct privat… by @mayurinehate in #6609
• Add AWS Postgres Iam Auth jar to GMS by @syedzoherer in #6371
• feat(ingest/snowflake): support filtering by fully qualified schema_pattern by @mayurinehate in #6611
• feat(ingest/kafka-connect): support MongoSourceConnector by @frsann in #6416
• feat(graph) Add createdOn, createdActor, updatedOn, updatedActor to graph edges by @chriscollins3456 in #6615
• refactor(ui): Making improvements to UI ingestion forms, adding MySQL, Trino, Presto, MSSQL, MariaDB forms by @jjoyce0510 in #6607
• perf(ui-ingestion): cache on creation or deletion of ingestion sources to reduce latency by @aditya-radhakrishnan in #6647
• feat(ingest): add dummy data source for automated testing by @anshbansal in #6550
• docs(managed datahub): adding release notes for v0.1.70 by @anshbansal in #6655
• feat(gms): Pluggable Authentication & Authorization Framework by @mohdsiddique in #6634
• docs: move rfcs to separate repo by @laulpogan in #6621
• fix(ingest): fix lingering demo-data source issues by @hsheth2 in #6659
• feat(ingest): bigquery - Running lineage extraction after metadata extraction by @treff7es in #6653
• fix(ingest): issue deprecation warning correctly by @hsheth2 in #6623
• chore(ingest): remove feast-legacy by @hsheth2 in #6661
• fix(ingest/snowflake): support domains for snowflake schema containers by @hsheth2 in #6662
• build(deps): bump decode-uri-component from 0.2.0 to 0.2.2 in /datahub-web-react by @dependabot in #6617
• feat(ingest/dbt): add support for latest DBT version 1.3 by @MatthieuBlais in #6651
• docs: add languages to code highlighting by @hsheth2 in #5576
• docs(typo) Correct typo in domains.md by @maggiehays in #6667
• feat(gms): Enable auth-api publishing to maven by @mohdsiddique in #6671
• fix(ingest/powerbi-report-server): deprecate unused graphql config by @daha in #6630
• fix(docker): Fix datahub-frontend dockerfile by @jjoyce0510 in #6670
• fix(ingest): profiling - Changing profiling defaults by @treff7es in #6640
• feat(ci): add smoke test for domain mutation by @anshbansal in #6641
• fix(datahub-protobuf): fix missing httpclient dependency by @shirshanka in #6672
• feat(ingest): update snowflake docs, add simple validations by @mayurinehate in #6636
• fix(gms): DataHub Auth API java doc fix by @mohdsiddique in #6674
• feat(ingest): run profiler in more cardinality cases by @hsheth2 in #6397
• docs(search) update broken youtube link by @maggiehays in #6678
• docs(protobuf): update examples for protobuf by @david-leifker in #6681
• feat(ingest): support knowledge links in business glossary by @mohdsiddique in #6375
• fix(ingestion/vertica): support columns with timestamp precision by @inancdokurel in #6295
• feat(ingest): add timestamps for snowflake objects by @mayurinehate in #6570
• feat(onboarding): adds framework and some steps for onboarding steps UI by @aditya-radhakrishnan in #6462
• feat(ingest): use entry point for registering transformers by @Masterchen09 in #6628
• chore(ci): update base ingestion image requirements file by @anshbansal in #6687
• fix(ci): reduce warnings due to deprecated action by @anshbansal in #6686
• refactor(ui): Adding caching for users, groups, and roles by @jjoyce0510 in #6673
• fix(ci): revert confluent kafka in base image by @anshbansal in #6690
• fix(security): version bump to latest minor python image by @david-leifker in #6694
• docs(ingest/salesforce): list required permissions by @orlandine in #6610
• feat(ingest): bigquery - option to set on behalf project by @treff7es in #6660
• ci: stop commenting test results on PR by @hsheth2 in #6700
• fix(auth-api): Attempting to fix publish for auth-api by @jjoyce0510 in #6695
• build(deps): bump qs from 6.5.2 to 6.5.3 in /smoke-test/tests/cypress by @dependabot in #6663
• build(deps): bump express from 4.17.1 to 4.18.2 in /datahub-web-react by @dependabot in #6665
• fix(ingest/tableau): support ssl_verify flag properly by @hsheth2 in #6682
• fix(config): unify the handling of boolean environment variables by @Masterchen09 in #6684
• fix(ui): fix search on policy builder by @aditya-radhakrishnan in #6703
• build(deps): bump qs from 6.5.2 to 6.5.3 in /datahub-web-react by @dependabot in #6664
• fix(ingest): cleanup config extra usage by @hsheth2 in #6699
• docs(logos): add Great Expectations logo by @maggiehays in #6698
• fix(security): play framework upgrade by @david-leifker in #6626
• fix(ingest/sagemaker): handle missing ProcessingInputs field by @hsheth2 in #6697
• build: add retries to gradle wrapper download in ingestion docker by @hsheth2 in #6704
• test(quickstart): add debugging to quickstart test by @david-leifker in #6718
• fix(setup): Bump setup images to alpine 3.14 with arch based on machine OS. by @pedro93 in #6612
• fix(ingest): fix bug in auto_status_aspect by @hsheth2 in #6705
• fix(security): commons-text, hadoop-commons versions by @david-leifker in #6723
• fix(build): rename conflicting module auth-api by @david-leifker in #6728
• docs(aws): edit markdown link by @jx2lee in #6706
• fix(ingest): fix mysql ingestion issue with non-lowercase database by @mayurinehate in #6713
• feat(ingest): redact configs reported in ingestion_run_summary by @hsheth2 in #6696
• fix(ingest): rectify filter for BigQuery external tables by @janhicken in #6691
• feat(ingest): add separate config for include_column_lineage in snowf… by @mayurinehate in #6712
• fix(ci): flakiness due to onboarding tour in add user test by @anshbansal in #6734
• feat(ui): Support DataBricks Unity Catalog Source in Ui Ingestion by @jjoyce0510 in #6707
• feat(ingest/iceberg): add stateful ingestion by @cccs-Dustin in #6344
• doc(restore): document restore indices API endpoint by @anshbansal in #6737
• feat(): Views Feature Milestone 1 by @jjoyce0510 in #6666
• feat(ingest): bigquery - external url support and a small profiling filter fix by @treff7es in #6714
• test(ingest): make hive/trino test more reliable by @hsheth2 in #6741
• Initial commit for bigquery ingestion guide by @treff7es in #6587
• fix(ci): remove warnings due to deprecated action by @anshbansal in #6735
• feat(ingest): add stateful ingestion to the ldap source by @bda618 in #6127
• fix(ingest): fix codegen from_obj for empty dicts in unions with null by @hsheth2 in #6745
• feat(ingest): start simplifying stateful ingestion state by @hsheth2 in #6740
• docs(gms): plugins# auth-api as compileOnly dependency by @mohdsiddique in #6747
• fix(elasticsearch): build in resilience against IO exceptions on httpclient by @RyanHolstien in #6680
• ci: fix ingestion gradle retry by @hsheth2 in #6752
• fix(ingest): support airflow mapped operators by @cccs-seb in #6738
• fix(actions): fix mistype slack/teams base url by @ssilb4 in #6754
• fix(smoke-test): fix stateful ingestion test regression by @hsheth2 in #6753
• fix(auth): Renames metadata-auth archive name to not conflict with other modules. by @pedro93 in #6749
• fix(ingest/lookml): fix directory handling and a config validation bug by @hsheth2 in #6751
• refactor(ingest): bigquery-lineage - allow tables and datasets in uppercase by @PatrickfBraz in #6739
• refactor(ux): Misc UX Improvements (tutorial copy, caching, filters) by @jjoyce0510 in #6743
• Added build failed yarn error by @jakobhanna in #6757
• feat(ingest): remove source config from DatahubIngestionCheckpoint by @hsheth2 in #6722
• fix(python-sdk): DataHubGraph get_aspect should accept empty responses by @shirshanka in #6760
• fix(datahub-web-react): Properly escape a quote in React by @jjoyce0510 in #6764
• docs(ingest/airflow): clarify Airflow 1.x docs for airflow plugin by @hsheth2 in #6761
• feat(ingest): simplify more stateful ingestion state by @hsheth2 in #6762
• fix(ingest): bigquery - handling custom sql errors as warning in profiling by @treff7es in #6777
• docs(docker): add section for adding community images by @anshbansal in #6770
• docs(ingest): fix error in custom tags transformer example by @hsheth2 in #6767
• feat(ingest): add datahub state inspect command by @hsheth2 in #6763
• refactor(ui): Caching Ingestion Secrets by @jjoyce0510 in #6772
• docs(snowflake) Snowflake quick ingestion guide by @maggiehays in #6750
• Optimize kafka setup by @david-leifker in #6778
• feat(ingest/lookml): add unreachable views to report by @hsheth2 in #6779
• feat(ci): adding github security reporting to trivy scans by @shirshanka in #6773
• fix(smoke-test): remove stateful ingestion config check by @hsheth2 in #6781
• fix(ingest): correct external url for account identifier with account name by @mayurinehate in #6715
• fix(tutorial): skip getting steps if there is no user by @aditya-radhakrishnan in #6786
• fix(kafka-setup): fix return code check by @david-leifker in #6782
• refactor(ui): Make include_tables and include_views default to True. Improve Tableau default recipe. by @jjoyce0510 in #6790
• fix(ingest): prevent NullPointerException when non-jdbc SaveIntoDataS… by @danielli-ziprecruiter in #6803
• docs(architecture): edit documents in architecture section by @jx2lee in #6798
• fix(ingest/dbt): remove unsupported usage indicator by @hsheth2 in #6805
• refactor(ui): Adding frontend caching + some misc. refactoring by @jjoyce0510 in #6796
• fix(ingest): bigquery - sharded table support improvements by @treff7es in #6789
• chore(ingest): pin black version by @hsheth2 in #6807
• refactor(ingest/stateful): remove most remaining state classes by @hsheth2 in #6791
• fix(profile): bigquery-legacy - Fix for TypeError-related failures in legacy plugin by @senapatim in #6806
• Update Grafana Dashboard by @NavinSharma13 in #6076
• refactor(ingest/stateful): remove IngestionJobStateProvider by @hsheth2 in #6792
• chore(ingest): bump python package dependencies to resolve vulns by @cyberay01 in #6384
• refactor(ingest/stateful): remove get_last_state method by @hsheth2 in #6794
• fix(ui): URL encode urns for ownership entity links by @aditya-radhakrishnan in #6814
• fix(posts): add deletePost GraphQL endpoint by @aditya-radhakrishnan in #6813
• fix(policies): resolve the associated domain for a domain as the domain itself by @aditya-radhakrishnan in #6812
• feat(lineage) Adds ability to edit lineage manually from the UI by @chriscollins3456 in #6816
• fix(ui): change caching to happen post server-response when creating a UI ingestion recipe by @aditya-radhakrishnan in #6815
• feat(ingest/stateful): remove platform_instance_id from state urn by @hsheth2 in #6795
• feat(ui): Adding DBT Cloud support for UI ingestion by @jjoyce0510 in #6804
• feat(kafka): expose default kafka producer mechanism by @djordje-mijatovic in #6381

New Contributors

• @maver1ck made their first contribution in #6596
• @MatthieuBlais made their first contribution in #6651
• @inancdokurel made their first contribution in #6295
• @orlandine made their first contribution in #6610
• @janhicken made their first contribution in #6691
• @cccs-Dustin made their first contribution in #6344
• @cccs-seb made their first contribution in #6738
• @ssilb4 made their first contribution in #6754
• @senapatim made their first contribution in #6806
• @cyberay01 made their first contribution in #6384

Full Changelog: v0.9.3...v0.9.4