datahub-project/datahub v0.8.17 on GitHub

Notable Changes

Added Recommendations and redesigned the home page!
- Modular way to add recommendations throughout the application
- Recommendation modules for top platforms, recently viewed, popular entities, top tags/terms were added to home page
- Search page also has top tags/terms module on the bottom
Ingestion Sources
- DBT enhancements
  - Creating dbt platform entities to capture dbt node types such as models, tests, source, seed, etc. linking dbt entities with other dbt or underlying platform entities.
- OpenAPI specs
- Kafka Connect (Regex based transformers, BigQuery sink)
- Trino Usage (Starburst)
Improved lineage viz performance and lineage viz UX
- Improved layout logic
- Nodes can be dragged and dropped
Fixes for delete API not always deleting all of an entities data
Improved documentation for adding a custom Metadata Ingestion Source
- Fixes description rendering for Charts, Dashboards, Flows, Jobs
Add YAML configuration file for Metadata Service
Filter search results by Sub-Type (Looker Explore, View, etc)
Support proxying DataHub Frontend requests to Metadata Service at /api/gms
Multi-platform (x86, arm64) support for Docker images (Apple M1 support)
Graph Service: DGraph support (phase 1)

What's Changed

fix(docs): fix image paths and company logo link by @jeffmerrick in #3435
feat(docs-site): two small tweaks by @gabe-lyons in #3437
feat(ingestion): support custom properties to be ingested via business glossary yaml by @gabe-lyons in #3438
fix(restli entity client): fix case where sortCriterion is null by @gabe-lyons in #3436
feat(lineage): improved lineage performance + simplified layout logic + some easter eggs by @gabe-lyons in #3357
docs(metamodel): added DataHub's metadata model diagram by @swaroopjagadish in #3449
fix(tag+terms): improved error messaging & rules on tag + term mutations by @gabe-lyons in #3448
fix(browse): disable breadcrumb links on non-browsable entities by @gabe-lyons in #3447
fix(ingest): fix lookml derived tables parsing by @remisalmon in #3443
docs(docs-site): small nits for docs site homepage by @gabe-lyons in #3444
perf(ingest): lazy load ingestion plugins by @hsheth2 in #3430
Fix docs website by @jeffmerrick in #3446
fix(restore): Fix restore backup jobs by @dexter-mh-lee in #3445
fix(ingest): lineage for Airflow subdags by @kevinhu in #3351
docs: Update to Q3 2021 accomplishments by @maggiehays in #3420
fix(bigquery): Add gcp logging dependency for bigquery source. by @varunbharill in #3451
build(frontend): unzip depend on yarnBuild by @gabe-lyons in #3452
feat(react): add handy webpack analyze command by @gabe-lyons in #3454
test(CI): show test results on GitHub by @EnricoMi in #3362
docs(transformers): add exemple of custom tag function by @WaStCo in #3354
docs: add guide for using custom sources by @DSchmidtDev in #3324
feat(dbt-ingestion): added possibility to skip specific models by @AndreasTA-AW in #3340
fix(mongodb): Support filtering mongodb documents as per size. by @varunbharill in #3456
fix(mysql): Update default mysql collation to utf8mb4_bin by @jjoyce0510 in #3459
fix(ingestion): Workaround for Python 3.8/3.9 mypy invalid syntax issue with airflow 2.2.0 by @rslanka in #3460
fix(ui): Fixing UI User + Group display name by @jjoyce0510 in #3461
fix(react): fix up yarn test error reporting by @gabe-lyons in #3462
docs(frontend): remove confusing suggestion to manually create users by @gabe-lyons in #3465
docs: Overhaul of DataHub Features page by @maggiehays in #3439
docs: Update TownHall Agenda and TownHall History by @maggiehays in #3463
fix(tags): fix links to tags when there are special chars in the urls by @gabe-lyons in #3464
fix(CI): Stabalize gradle build by @EnricoMi in #3413
docs: update next Townhall date in README.md by @maggiehays in #3466
perf(react bundle): decrease bundle size by 15% by @gabe-lyons in #3468
fix(graphql): fixing Graphql engine factory when analytics are disabled by @gabe-lyons in #3467
feat(recommendations): Recommendations infra P1 by @jjoyce0510 in #3455
refactor(styling): Improving recommendation Tag / Search query list styling by @jjoyce0510 in #3472
fix(docs): fix transformer doc example by @aseembansal-gogo in #3469
fix(ingest): redshift source gets external table types properly by @treff7es in #3371
fix(recs): Remove removed entities from aggregation by @dexter-mh-lee in #3473
fix(ui): fix double formatting of entity count on home page by @jjoyce0510 in #3474
fix(subtypes): fix case where subtypes are not being fetched for leaf datasets by @gabe-lyons in #3476
feat(ingestion): User configurable dataset profiling. by @rslanka in #3453
styling(ui): improve tag list, glossary term list recommendation styling by @jjoyce0510 in #3475
feat(ui): Provide filtering capability for Sub Types inside the UI by @jjoyce0510 in #3479
fix(ingest): correctly support multiple snowflake databases by @hsheth2 in #3482
fix(datajobs): fetch dataflow properties from a relationship by @gabe-lyons in #3487
fix(fk): fix schemaField urn construction in foreign keys by @gabe-lyons in #3486
fix(fk): trim whitespace from fk constraints in the case the fieldspec has leading or trailing whitespace characters by @gabe-lyons in #3485
feat(dbt): add dbt logo and platform. by @varunbharill in #3483
feat(lineage): some ux improvements to lineage interactions by @gabe-lyons in #3478
refactor(nocode): Final part of No-Code cleanup by @jjoyce0510 in #3477
fix(browse paths): Adjust Default browse path logic for datasets by @jjoyce0510 in #3495
fix(lineage backend): fix ownership timestamps by @gabe-lyons in #3498
tests(smoke): introducing first isolated smoke test: updating tags & terms by @gabe-lyons in #3496
feat(graphql): extend entity client to support aspect methods directly via java by @gabe-lyons in #3489
fix(aspects): fix null aspects case by @gabe-lyons in #3501
Docs: Update to Slack & Townhall details by @maggiehays in #3502
refactor(profiler): add PerfTimer class and fix typos by @hsheth2 in #3497
fix tiny typo by @andrewm4894 in #3484
fix(ingestion): Glue job names by @kevinhu in #3503
fix(fk): fix foreign key styling with modals by @gabe-lyons in #3500
docs: add path fix for 'command not found' by @dannylee8 in #3490
docs: nit, grammar by @dannylee8 in #3491
docs: nit by @dannylee8 in #3492
Docs: nits by @dannylee8 in #3493
add tooltip for owner category in dataset profile page by @saxo-lalrishav in #3470
feat(ingest) : kafka connect source improvements by @mayurinehate in #3481
feat(ingest): adding support for read-modify-write capabilities durin… by @swaroopjagadish in #3506
feat(dbt): Dbt enhancements - dbt nodes, lineage, subtype, etc. by @varunbharill in #3519
docs (Metadata Model): nits by @dannylee8 in #3525
fix(ingestion): Enhance logging and error-handling in bigquery usage connector. by @rslanka in #3521
docs: nits and added hyperlinks by @dannylee8 in #3526
(Docs) Updated troubleshooting tip by @dannylee8 in #3516
test(profiler): make profiling tests more comprehensive by @hsheth2 in #3513
doc (React) Node 17 openssl change causes error by @dannylee8 in #3523
feat(cli): add support for deletion based on filters, soft deletes an… by @swaroopjagadish in #3527
feat(frontend): Proxy GMS API requests by @jjoyce0510 in #3509
feat(deletes): support deletion of non-snapshot aspects by @gabe-lyons in #3518
fix(ingest): restrict botocore version to fix urllib3 build issue by @hsheth2 in #3534
ui: Migrate UI to use "properties" field of entity for descriptions by @jjoyce0510 in #3515
fix(cli): fix name of cli arg by @aseembansal-gogo in #3536
feat(git-version): Encode the latest release included in the build by @dexter-mh-lee in #3535
Revert "feat(git-version): Encode the latest release included in the build" by @dexter-mh-lee in #3539
[feat] Add multiplatform docker support by @treff7es in #3537
feat(gms): Adding yaml configuration for metadata-service by @jjoyce0510 in #3514
fix(docker): fix multi-platform build for arm by @treff7es in #3543
fix(glossary terms): Fix 'Glossary Term on Dataset Columns doesn't show in Related Entities' by @jjoyce0510 in #3542
fix(ingestion): Make AVRO schema parsing robust to exceptions. by @rslanka in #3541
docs: cleanup old links, misc updates by @swaroopjagadish in #3545
docs(graphql): Updating example CURLs to work on copy and paste + misc FAQ doc improvements by @jjoyce0510 in #3538
feat(cli): add support for m1 laptops during quickstart by @swaroopjagadish in #3547
feat(ingestion): Support for converting AVRO schemas with logical types to MCE fields. by @rslanka in #3546
feat(profiler): streamline profiler by @hsheth2 in #3510
feat(ingest): add transformer to add properties by @nomarlo in #3480
Adding openapi ingestion by @vlavorini in #2706
[fix] Disabling arm frontend build temporary by @treff7es in #3551
fix com.linkedin.dashboard.TagProperties by @andrewm4894 in #3550
[fix] Build frontend docker on build platform instead on target platform by @treff7es in #3552
fix(docker): create multiplatform docker build from elasticsearch-setup by @treff7es in #3562
fix(docs-website): Fix company logo urls by @jjoyce0510 in #3568
fix(frontend): Hush noisy datahub-frontend warnings by @jjoyce0510 in #3559
fix(docs): use absolute links by @swaroopjagadish in #3570
docs: Adding a custom Data Platform doc by @jjoyce0510 in #3561
fix: Glue lineage compatibility by @kevinhu in #3555
fix(bigquery_usage ingestion): add partition decorator to regex, move exception handling to after matching, add table snapshots by @courtney-lang in #3533
fix(ingest): fix bigquery-usage regex for both partitioned and sharde… by @swaroopjagadish in #3571
fix(ingestion, redshift-usage): Do not append email domain to the username if the username is already an email. by @rslanka in #3569
docs: adding links to help with metadata model visualization and documentation by @swaroopjagadish in #3573
feat(GraphService): Add Dgraph implementation of GraphService by @EnricoMi in #3261
feat(ingest): adding snowflake app name to connection string by @swaroopjagadish in #3574
Run metadata-io tests in parallel by @EnricoMi in #3577
fix(users): fix ownership count on user page by @gabe-lyons in #3575
fix(graphql): making glossaryTermInfo nullable in glossaryTerm. by @varunbharill in #3576
fix(autocomplete): fix case where autocomplete interferes with search by @gabe-lyons in #3580
fix(cli): m1 check breaks on windows by @swaroopjagadish in #3579
Feat: Allow logs to be filtered in Bigquery Usage plugin by @tha23rd in #3567
fix(ingestion): Fix snowflake documentation. by @rslanka in #3585
feat(ci): adding support for env variables in python release script by @swaroopjagadish in #3587
fix(docs): Add documentation on how to connect to custom ES instance. by @varunbharill in #3581
fix(ci): SKIP_RELEASE_UPLOAD flag was not being respected by python r… by @swaroopjagadish in #3588
feat(ingestion): Adds --dry-run and --preview options to datahub ingest command. by @rslanka in #3584
fix(ingest): fix dbt source platform when disable_dbt_node_creation is False by @remisalmon in #3592
Add docs nav links by @jeffmerrick in #3594
feat(ingest): add bigquery sink connector lineage in kafka connect source by @mayurinehate in #3590
feat(model): adding a field to capture unmodeled field level properties by @swaroopjagadish in #3593
fix(browse): Fix browse response size issue when there are too many browse paths by @dexter-mh-lee in #3595
fix(docs): Add docs on accessing datahub CLI by @aseembansal-gogo in #3589
feat(ingest): Starburst Trino usage by @treff7es in #3558
fix(ingestion): Emitter api examples + Documentation by @rslanka in #3599

New Contributors

@maggiehays made their first contribution in #3420
@WaStCo made their first contribution in #3354
@DSchmidtDev made their first contribution in #3324
@andrewm4894 made their first contribution in #3484
@dannylee8 made their first contribution in #3490
@nomarlo made their first contribution in #3480
@courtney-lang made their first contribution in #3533
@tha23rd made their first contribution in #3567

Full Changelog: v0.8.16...v0.8.17