This release includes the patch for CVE-2021-44228, pinning log4j to 0.2.17. Small bug fixes & improvements, otherwise.

Release Highlights

• Configurable aspect retention in application.yml (disabled by default)
• Metabase Ingestion Source connector
• Constrain log4j to version 0.2.17
• Upgrade logback to 1.2.9

What's Changed

• feat(spark-lineage): add ability to push data lineage from spark to d… by @MugdhaHardikar-GSLab in #3664
• feat(cli): allow to nuke without deleting data in quickstart by @anshbansal in #3655
• feat(Dgraph): Make Dgraph a proper Neo4j alternative by @EnricoMi in #3578
• feat(retention): Add retention to Local DB by @dexter-mh-lee in #3715
• feat(ingest): cleanup deprecated datahub.integrations.airflow.* imports by @hsheth2 in #3732
• feat(ingestion) : Add Metabase Source Connector by @jawadqu in #3602
• fix(ingest): count profiled tables separately in report by @hsheth2 in #3731
• feat(perf-test): changes for perf testing by @anshbansal in #3728
• ci(cypress): adding the foundation for cypress integration tests & some starter coverage for login, search & updates by @gabe-lyons in #3672
• (fix) Elastic search container log4j CVE-2021-44228 vulnerability by @nsbala-tw in #3733
• Revert "feat(Dgraph): Make Dgraph a proper Neo4j alternative" by @gabe-lyons in #3740
• fix(CI): Regenerate Docker Quickstart by @jjoyce0510 in #3741
• fix(DataHubGraph): changing datahub-graph to use underlying session connection. by @varunbharill in #3743
• fix(ingest): Remove unecessary isalpha check for data platforms + warnings by @jjoyce0510 in #3742
• feat(snowflake-usage): add knob for direct objects accesssed vs base objects accessed by @gabe-lyons in #3744
• fix(snowflake): support snowflake allow/deny pattern for lineage and usage by @varunbharill in #3748
• refactor(gms auth): Remove base64 decoding of token service signing key by @jjoyce0510 in #3747
• test(ingest): fix pytest warning for class starting with Test by @hsheth2 in #3745
• feat: enables dbt metadata files to be loaded from URIs by @sgomezvillamor in #3739
• fix(ingestion): Skipping duplicate tables from ingestion by @treff7es in #3753
• feat(Stateful Ingestion): 1/3 Stateful ingestion server changes by @rslanka in #3749
• Fix CVE-2021-44228 continued: log4j constraints to version 2.16.0 by @jjoyce0510 in #3755
• build(ingest): restrict latest mypy version by @hsheth2 in #3756
• doc: Add IOMED as a DataHub adopter by @merqurio in #3758
• docs(spark-lineage): update artifact name and version by @MugdhaHardikar-GSLab in #3760
• feat(profiler): add upper bound on combined query size by @hsheth2 in #3762
• feat(ingestion): Mode retry wait logic to avoid hitting Mode API rate limit by @jawadqu in #3761
• feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. by @rslanka in #3763
• refactor(test): replace CliRunner with run_datahub_cmd method by @hsheth2 in #3746
• feat(bigquery): add support for parsing exported bigquery audit logs by @hyunminch in #3680
• feat(ingest): Adding support for Elasticsearch and Clickhouse by @sudotty in #3227
• Upgrade to logback 1.2.9 to address CVE-2021-42550 by @jjoyce0510 in #3771
• fix(profiling): Disabling expensive profilers by default by @treff7es in #3759
• docs(ingestion): Add details of sensitive info handling by @anshbansal in #3767
• docs(snowflake): Adding documentation about required Snowflake Privileges by @jjoyce0510 in #3770
• Upgrade to 3rd Apache patch for log4j by @xiphl in #3772
• fix(ingestion): Fix for same schema foreign key reference by @treff7es in #3769
• fix(ingest): fix compatibility with google composer by @anshbansal in #3774

Known Issues

We've been made aware that in large deployments the re-indexing step required at boot-up time exceeds the 30 second timeout. We've since made changes to loosen this timeout limit, with these changes coming in 0.8.21.

New Contributors

• @MugdhaHardikar-GSLab made their first contribution in #3664
• @jawadqu made their first contribution in #3602
• @nsbala-tw made their first contribution in #3733
• @merqurio made their first contribution in #3758
• @hyunminch made their first contribution in #3680
• @sudotty made their first contribution in #3227
• @xiphl made their first contribution in #3772

Full Changelog: v0.8.19...v0.8.20