Release Highlights
User Experience
We're excited to announce View-based RBAC Policies! You can now create and apply view-only permissions to your DataHub end-users, providing more robust access controls.
We've also included some small (but impactful!) improvements to UX, including:
- Display recent search terms when beginning the search flow
- Consistently displaying entity subtypes for dbt, Looker, Kafka, & more. Think: Kafka entities are displayed as "topics" instead of "datasets"
Ingestion Highlights
- New! Protobuf ingestion (shoutout to @leifker for this Community-led contribution!)
- Initial work to support a "Notebook" entity (shoutout to @tc350981 for spearheading this work!!)
- Stateful ingestion for dbt is now supported
- Ongoing improvements to our Tableau ingestion source from @nandacamargo & @cuong-pham
- Improvements to handling database aliases for Redshift ingestion
- Improvements to S3 source:
- Add containers for datasets
- Support platform_instance
- Support for folder level datasets
- Increased flexibility to specify dataset paths
- Ingestion Fixes:
- Snowflake Usage - log warning instead of error out & other error handling
- Snowflake allow/deny patterns
- Examples of allow/deny patterns added to docs
Full Commit Log
- #4570 @gabe-lyons fix(search): handle commas in search queries in the UI
- #4557 @daha fix: replace direct and indirect references to linkedin with datahub-project
- #4569 @dexter-mh-lee fix(policy): Add view entity page priv to all entity types
- #4567@anshbansal fix(bigquery): missing dependency
- #4548 @mayurinehate fix(tableau): gracefully stop ingestion if tableau sign in not successful
- #4564 @dexter-mh-lee fix(docs): fix logo links on ingestion docs
- #4396 @Abhiram98 feat(ingestion): schema, table filtering for redshift-usage
- #4554 @darapuk (fix): Update path generated when creating LookML URL
- #4549 @maggiehays docs: add sumup logo
- #4560 @mhw docs: Fix PostgreSQL typo in
features.md - #4562 @gabe-lyons feat(lineage): show fully qualified dataset name on expansion
- #4561 @anshbansal fix: dependencies for usage sources
- #3782 @CorentinDuhamel feat(ingest): indent sql queries for usage sources
- #4551 @pedro93 fix(rollback) Removes status & key aspects from affected aspects count during rollback
- #4538 @dexter-mh-lee fix(policy): Remove all from the resource type choices
- #4544 @anshbansal fix(ingest): snowflake-usage - log warning instead of error out
- #4542 @mattmatravers build(ui): allow custom nodeDistBaseUrl
- #4545 @mayurinehate fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map
- #4547 @anshbansal chore: update pull request template
- #4537 @RyanHolstien fix(dataPlatformInstance): add data platform instance to entity registry
- #4375 @mayurinehate fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d…
- #4492 @RyanHolstien fix(cli): delete - handle case insensitive entity types
- #4130 @sgomezvillamor feat(ingest): glue - adds platform instance capability
- #4456 @mohdsiddique feat(stateful dbt): add stateful ingestion capability in dbt source
- #4482 @pedro93 feat(platform): adds side-effect report for rollbacks
- #4275 @maggiehays docs: Ingestion Source Docs Template
- #4470 @mayurinehate feat(tableau): emit lineage edge from embedded datasource to upstream…
- #4535 @pppsunil feat(ingestion): Support pluggable Schema Registry for Kafka Source
- #4369 @eburairu feat(ui): Add new loading pattern logo
- #4493 @leifker feat(integration): protobuf - additional annotations and features
- #4435 @zhoxie-cisco perf(docker): datahub-gms - add jetty configuration xml
- #4532 @anshbansal doc: add example of profiling in default example
- #4533 @anshbansal doc: clarify CLI releases
- #4528 @daha fix(doc): Change to forward slash-separated strings, as in the example
- #4315 @daha fix(docs): Minor fixes to pip install commands
- #4521 @anshbansal doc: update docker docs for mentioning Python CLI
- #4523 @kevinhu fix(ingest): mssql - support database_alias
- #4501 @arunvasudevan feat(ingest): kafka-connect - support mapping for multiple DB instances
- #4477 @jjoyce0510 feat(metadata service): Introducing Platform Events
- #4526 @jjoyce0510 Adding has container
- #4494 @cuong-pham fix(ingest): make tableau ingestion more resilient to error
- #4519 @anshbansal doc(ingestion): add examples of running in docker and Kubernetes
- #4485 @andres-lowrie docs(metadata-ingestion): callout props in para
- #4511 @RyanHolstien Oss/urn validation
- #4525 @dexter-mh-lee feat(policy): Add tooltip and view button
- #4507 @mayurinehate feat(assertion): update python example, assertion entity doc
- #4513 @kevinhu feat(ingestion): detect and disable telemetry in CI
- #4516 @dexter-mh-lee feat(policy): Add domain based and view based policies
- #4520 @rslanka Fix: Snowflake Table to View lineage
- #4510 @anshbansal feat(ingest): Add config to improve user exp for initial ingestion and fix docs
- #4517 @anshbansal feat(ingest): option for number of workunits in preview
- #4503 @treff7es feat(ingest): athena - set Athena location as upstream
- #4490 @MugdhaHardikar-GSLab feat(s3): add s3 source
- #4468 @tc350981 feat(notebook): graphqul related logic change for notebook
- #4401 @anshbansal doc: update instructions for updating DataHub on quickstart
- #4500 @ShubhamThakre SecretBuilderModal -> name field validation updated
- #4505 @anshbansal docs: add example of database and schema allow/deny patterns
- #4504 @anshbansal fix(snowflake): allow/deny patterns
- #4496 @shirshanka feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently
- #4480 @anshbansal feat(snowflake): stop querying for usage data when no mix/max dates
- #4483 @anshbansal fix(snowflake-usage): do not ingest for stage as a dataaset
- #4497 @shirshanka moving to dockerhub for actions container
- #4475 @darapuk fix: Update GroupProfile to read from properties over deprecated info aspect
- #4489 @anshbansal fix(ingestion): pin Jinja2 to version < 3.1.0
- #4484 @anshbansal fix(ingestion): stop CLI build failures
- #4467 @anshbansal doc: add caveats to snowflake doc
- #4481 @anshbansal fix(snowflake): don't recommend accountadmin role for snowflake
- #4479 @anshbansal fix: change log level to debug
- #4476 @dexter-mh-lee Add recent searches filtering
- #4469 @tc350981 feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn
- #4439 @ShubhamThakre Feature/modal-validation-and-UI-fixes-updates
- #4398 @kevinhu feat(ingest): simplify event IDs for function invocations
- #4474 @sgomezvillamor chore: acryl-data 0.6.12
- #4473 @treff7es fix(redshift) Properly handling database alias in redshift usage and redshift lineage generation
- #4471 @gabe-lyons enabling ml tabs
- #4460 @eclaassen-pb fix: java dependency vulnerabilities
- #4453 @treff7es feat(ingest) data-lake: Add s3 properties metadata when ingesting s3 files
- #4464 @anshbansal fix: change for repository change
- #4466 @anshbansal fix(snowflake-usage): add more error handling
- #4445 @nandacamargo fix(ingest): add fix to tableau connector when table has None fields
- #4457 @mayurinehate docs(hive): update recipe with example to specify kerberos auth
- #4313 @pedro-iatzky fix(ingest): bigquery - fix ingestion of external tables
- #4462 @gabe-lyons adding final transport options
- #4223 @tc350981 feat(notebook): add data models for Notebook entity
- #4450 @pedro93 feat(frontend) Adds multiple group claim support
- #4442 @anshbansal feat(ingestion): snowflake, bigquery - enhancements to log and bugfix
- #4333 @anshbansal doc: add guide for ui tabs
- #4443 @jjoyce0510 Fixing privilege option display bug
- #4451 @gabe-lyons fix tableau connector when it cannot connect to URI
- #4237 @tc350981 (docs) add RFC file to introduce Notebook entity data model
- #4446 @kevinneville fix: Replace old repository link with new link
- #4447 @cuong-pham getting database directly from upstream tables incase there are multiple databases in upstreamDatabases
- #4436 @treff7es Passing entity properly on deletion
- #4433 @rslanka Fix bug in the SchemaField type computation for AVRO logical types.