Release Stability
- There are a few bugs reported on this release that are fixed in 0.8.8. Users are highly recommended to skip past this release!
Release Highlights
- Dataset Profiling and support for time-series metadata
- UI for ML Models, Features; support for AWS SageMaker and Feast
- Cli: support for rollback operations after ingestion
- Integration fixes for Looker, dbt, and many more.
- Demos for all these features are available in our July Townhall video
ChangeLog
- #3021 @kevinhu feat(ingest): extract dbt versions into custom properties
- #3020 @gabe-lyons fix(caching): refetch query on update
- #3019 @kevinhu fix(ingest): don't assume Glue job description always exists
- #3000 @topwebtek7 fix(react): fix weird 0 rendering possible bugs
- #3018 @dexter-mh-lee feat(ingest): add kafka emitters for MetadataChangeProposal format
- #2999 @jjoyce0510 fix(gms): Adding Rest.li Write-Time Model Validation
- #3009 @jjoyce0510 fix(quickstart): Bumping Default Memory for GMS and Frontend
- #3007 @jjoyce0510 fix(gms): better logging on failed MCL / MAE
- #3008 @gabe-lyons fix(blank pages): removing apollo caching
- #3006 @jjoyce0510 fix(ci): using AspectExtractor instead of removed SnapshotToAspectMap
- #2998 @gabe-lyons fix(graphql): fetching data platforms using standard procedure
- #2944 @EnricoMi refactor(test): Refactor GraphService tests
- #2972 @jameslamb fix(ingest): map all LookML dimension types to corresponding avro types
- #3005 @dexter-mh-lee fix(ingestion): Safeguard against empty values for profile ingestion
- #3002 @dexter-mh-lee fix(datahub-upgrade) add config registry to datahub upgrade container
- #3003 @jjoyce0510 fix(dataset stats): Fix checks for existence of row and column counts
- #2997 @topwebtek7 feat(react): update dataset documents tab with a merged document column
- #2991 @topwebtek7 feat(react): update search result has result counts for each entities that has result
- #2983 @jjoyce0510 Introducing TimeSeries Aspects + Dataset Profile (Stats) Aspect
- #2984 @dexter-mh-lee fix(browse): Fix browse pagination and multi-browse path issue
- #2995 @aseembansal-gogo docs(ingest): Add instructions to install required dependency
- #2960 @gabe-lyons feat(deletes): add run commands (list, show, rollback) to datahub ingest
- #2994 @chinmay-bhat docs(ingest): fixed Snowflake recipe to escape dollar-sign
- #2981 @hsheth2 docs: remove a few outdated docs
- #2988 @jjoyce0510 docs: add docs on extracting container logs
- #2963 @hsheth2 test(ingestion): run full tests on both python versions
- #2967 @jameslamb fix(ingest): add more debug logging to LookML metadata ingestion
- #2966 @jameslamb fix(ingest): ensure that LookML files are always parsed in the same order
- #2965 @jameslamb fix(ingest): ensure workunits are created for all LookML views
- #2982 @gabe-lyons fix(tags): fixing tag applied to module for tags w/ colons in the name
- #2961 @gabe-lyons feat(ml-model): adding ml models and ml model groups
- #2975 @kevinhu feat(ingest): type stubs for boto3
- #2979 @jameslamb perf(ingest): remove unused variable in Looker ingestion
- #2980 @hsheth2 fix(ingest): infer bigquery project identifier
- #2978 @chinmay-bhat fix(ingest): fix hive ingestion to respect database configuration
- #2976 @hsheth2 feat(ingest): stricter deserialization for MCE JSONs
- #2959 @kevinhu feat(docs): tutorial for writing a custom transformer
- #2977 @hsheth2 fix(ingestion): isolate dependency requirements of airflow hooks
- #2962 @hsheth2 feat(ingest): add timezone validation to bigquery usage
- #2974 @dexter-mh-lee fix(elasticsearch-setup): fix elasticsearch setup for aws
- #2952 @hsheth2 text(ingestion): test multiple python versions in CI
- #2958 @hsheth2 feat(ingest): add Airflow TaskFlow example
- #2950 @kevinhu fix(ingest): patch lookml types and refactor ingestion sources layout
- #2957 @jameslamb fix(ingest): match nested LookML files mentioned in 'include' statements
- #2956 @gabe-lyons Revert "fix(gql): removing data platform caching in gql (#2947)"
- #2955 @kevinhu feat(ingest): ingest descriptions from dbt models
- #2948 @hsheth2 fix(ingestion): add more mypy annotations
- #2946 @hsheth2 feat(ingestion): test GMS connections before ingestion
- #2947 @gabe-lyons fix(gql): removing data platform caching in gql
- #2949 @hsheth2 test(ingestion): fix flaky package discovery test
- #2951 @kevinhu feat(docs): update videos and integration logos
- #2953 @hsheth2 fix(ingestion): resolve test bugs for 3.6
- #2943 @kevinhu feat(ingest): add logo and platform entry for Glue
- #2940 @hsheth2 fix(ingest): handle quotes in lookml properly
- #2938 @kevinhu feat(models): remove versions from metrics and hyperparams
- #2942 @hsheth2 fix(ingestion): make snowflake database names lowercase
- #2939 @hsheth2 feat(ingest): use urn builders in looker and validate data platforms
- #2941 @aseembansal-gogo refactor(ingest): make code pythonic
- #2937 @kevinhu fix(ingest): allow custom Glue scripts
- #2921 @kafkahw refactor(datahub-web): removing frontend Ember app (i.e. datahub-web folder)
- #2913 @hsheth2 fix(ingest): refactor + fix recursion in lookml file loading logic
- #2925 @hsheth2 feat(ingest): improve bigquery-usage robustness and docs
- #2931 @aseembansal-gogo fix(ingest): fix workunit name to be consistent with other sources
- #2935 @kevinhu fix(ingest): fix browsepaths and ownership urns
- #2930 @aseembansal-gogo fix(ingest): glue add support for mapping varchar, decimal types
- #2929 @kevinhu feat(ingest): refactor mlModel grouping and add browsepaths
- #2934 @hsheth2 docs(ingest): update looker + docker script docs
- #2926 @hsheth2 feat(ingest): add
make_data_platform_urnmethod to builder - #2932 @topwebtek7 feat(react): surface edited descriptions on search preview for dataset, datajob, dataflow, chart, dashboard
- #2911 @hsheth2 fix(ingest): add quotes to secured kafka yaml config example
- #2927 @kevinhu feat(ingest): dbt aliases
- #2806 @saxo-lalrishav fix(react): enable relation between glossary term and datasets searchable
- #2910 @kevinhu feat(ingest): extract SageMaker metrics, hyperparameters, and external URLs
- #2915 @aseembansal-gogo docs: update docs for consistency in naming
- #2922 @kevinhu feat(ingest): test dbt ingestion with and without schemas
- #2924 @hsheth2 fix(ingest): note that views are not supported for Athena
- #2920 @hsheth2 feat(ingestion): support multiple project IDs in bigquery usage stats
- #2923 @hsheth2 fix(ingest): pin snowflake sqlalchemy connector
- #2909 @hsheth2 feat(ingest): add support for Oracle spatial types
- #2917 @kevinhu docs(ingest): update sample recipe and test input for dbt
- #2887 @topwebtek7 feat(mlFeatureTable): add graphql, ui/ux for mlFeatureTable, mlFeature, mlPrimaryKey entities
- #2916 @kevinhu fix(ingest): stringify all dbt custom props
- #2898 @aseembansal-gogo feat(ingest): Add option to change name of database for postgres
- #2912 @hsheth2 fix(ingest): issue a warning if the column list is empty
- #2894 @kevinhu feat(ingest): lineage for SageMaker model endpoints and groups
- #2905 @hsheth2 feat(ingest): add
can_add_aspectmethod for MCEs - #2906 @hsheth2 test(ingest): update tox test configurations and test airflow 2.x by default
- #2904 @jjoyce0510 fix(frontend): Don't use Apollo Cache for IsAnalyticsEnabled query.
- #2877 @remisalmon feat(ingest): use node comment as description if existing else default to key
- #2889 @hsheth2 fix(react): avoid displaying "0" for ignored timestamps
- #2890 @gabe-lyons fix(search): fixing case where someone issues a null query
- #2893 @hsheth2 fix(ingest): use logger.warning instead of logger.warn
- #2888 @jameslamb fix(ingest): change LookMLSource._get_upsteam_lineage() to _get_upstream_lineage()
- #2901 @topwebtek7 feat(react): update schema history visualizing, truncate long type, original desc bug
- #2891 @hsheth2 fix(ingest): correct globs in lookml model discovery
- #2902 @kevinhu feat(ingest): add connectivity check for Looker
- #2597 @wan54 feat(react): configure Cypress + MirageJS + GraphQL mock for functional testing plus a couple of example tests
- #2903 @shirshanka docs: update docs for July townhall
- #2900 @kevinhu fix(ingest): string-ify dbt custom props
- #2899 @jjoyce0510 fix(docs): fixing miscellaneous docs
- #2788 @saxo-lalrishav fix(glossary):default browse path for glossary term
- #2868 @kevinhu feat(ingest): extract lineage between SageMaker jobs and models
- #2884 @dexter-mh-lee fix(search): Fix index builder
- #2883 @hsheth2 docs: revamp adoption section
- #2882 @hsheth2 fix(ingest): fix druid misconfiguration bug
- #2881 @hsheth2 fix(ingest): default to unlimited query log delay in bigquery-usage
- #2790 @saxo-lalrishav fix(search): enable search on business glossary terms
- #2872 @hsheth2 build(ingest): reduce dependencies for dev install
- #2874 @topwebtek7 fix(react): fix bug in description update modal
- #2866 @hsheth2 build(ingestion): add version prompt to release script
- #2869 @kevinhu feat(ingest): update golden files only when diff fails
- #2876 @kevinhu feat(ingest): extract dbt meta fields
- #2875 @hsheth2 docs(quickstart): add default password to quickstart
- #2873 @hsheth2 fix(quickstart): update compose spec version
- #2862 @hsheth2 build(ingest): separate metadata-ingestion build workflow fully
- #2867 @hsheth2 fix(build): increase retries for dependency fetches
- #2849 @kevinhu feat(ingest): add browse paths + dataplatform for Feast features
- #2859 @kevinhu feat(docs): swap Medium and videos sections
- #2858 @hsheth2 feat(ingest): support dynamic imports for transfomer methods