datahub-project/datahub v0.13.3rc1 on GitHub

What's Changed

fix(ingest/bigquery): Supporting lineage extraction in case the select query result's target table is set on job by @treff7es in #10191
fix(retention): fix time-based retention by @trialiya in #10118
feat(lineage): give via and paths in entity lineage response by @RyanHolstien in #10192
fix(ingestion/datahub): implemented the filter to ignore/include URN for ingestion by @dushayntAW in #10174
fix(ingestion/glue): fix to ingest the comment for partition key as description by @dushayntAW in #10189
feat(ingest/looker): cleanup usage generation code by @hsheth2 in #10153
fix(dev): fix env file overrides for profiles by @hsheth2 in #10194
fix(ingestion/hive): ignore sampling for tagged column/table by @dushayntAW in #10096
fix(ui/property): add collapse for long dataset properties by @gaurav2733 in #10203
saas release v0.3.1 release notes by @david-leifker in #10205
fix(ingest/databricks): pin pandas for databricks ingestion by @mayurinehate in #10204
Fixed issue where the custom defined aspects were missing from the API specification. by @ajoymajumdar in #10208
feat(ingestion/transformer): Handle overlapping while mapping in extract ownership from tags transformer by @shubhamjagtap639 in #10201
fix(build): avoid nested gradle commands by @hsheth2 in #10198
feat(ingest/great_expectations): support in-memory (Pandas) data assets by @bouaouda-achraf in #9811
ci(workflow): publish docker from pr with label by @david-leifker in #10193
bump(version): bump classgraph version, add early package filter by @david-leifker in #10207
fix(ingestion/mongodb): MongoDB source unable to parse datetimes with years > 9999 by @jonasHanhan in #10110
fix(graphql-core): DomainEntitiesResolver does not support values FacetFilterInput parameter by @siladitya2 in #10188
fix(graphql-core):Auto completion/suggestion of Domains are not working by @siladitya2 in #10150
chore(usage-stats): measure time for getting buckets and aggregations by @darnaut in #10220
test(search): introduce retry for search test by @david-leifker in #10206
feat(ingest/bigquery): fix support for incremental column lineage by @hsheth2 in #10222
fix(ingest/dbt): better dbt timestamp parsing by @hsheth2 in #10223
feat(ingest/sql): normalize bigquery partitioned tables when parsing by @hsheth2 in #10224
docs: fix feedback button design by @yoonhyejin in #10182
docs: add discourse to community tab by @yoonhyejin in #10181
docs: edit the text and destination for sign up link by @yoonhyejin in #10183
fix(ingestion/datahub): moved urn_pattern config to source config by @dushayntAW in #10215
fix(ingestion/airflow-plugin): ingesting the tags along with the data by @dushayntAW in #10216
fix(ingest): suppress all column-level parsing errors by @hsheth2 in #10211
fix(ci): unified workflow login logic by @david-leifker in #10235
fix(lineage): fix lighting cache dataJob platform by @david-leifker in #10233
feat(vianode): v3 of cll via datajob update by @david-leifker in #10221
chore(build): bump actions versions by @david-leifker in #10240
fix(ingest): avoid requiring sqlalchemy for dynamodb classification by @hsheth2 in #10213
docs(cli/init): make datahub init docs more clear by @gabe-lyons in #10245
feat(ingest/redshift): filter out system queries from usage by @hsheth2 in #10247
feat(gql): support operationName by @hsheth2 in #10210
fix(frontend): fix frontend script used in release checklist by @david-leifker in #10243
docs(init): Update entrypoints.py to be more clear about acryl init by @gabe-lyons in #10248
fix(airflow): disable OL regardless of plugin status by @hsheth2 in #10250
fix(ingestion/salesforce): added additional check for description by @dushayntAW in #10239
feat(api): Add description parameter to editable dataset change entity event by @eboneil in #10237
fix(ingest/bigquery): fix lineage if multiple sql expression passed in and destination table set by @treff7es in #10212
feat(ingest/nifi): ingest process group as browse path v2, incremental lineage by @mayurinehate in #10202
fix publish-datahub-jars workflow by @david-leifker in #10244
fix(ingest/unity): Fix bug around unity notebook ingestion by @asikowitz in #10253
feat(ingest/cockroachdb): add cockroachdb ingestion by @dotan-mor in #10226
feat(ingestion/bigquery): support patterns for label -> tag capture by @olgapenedo in #10146
feat(ingest/fivetran): use emails in owner user urns by @shubhamjagtap639 in #10229
feat(cli): Make yaml loaders compatible with pydantic v2 by @eboneil in #10257
fix(ingest): support pydantic v2 with properties subcommand by @hsheth2 in #10256
feat(ingestion): Add -e flag to uv command in ingestion Dockerfiles by @skrydal in #10114
fix(quickstart): remove unneeded init.sql by @darnaut in #10266
fix(ingestion/airflow-plugin): replace deprecated calls by @ms32035 in #10238
build(deps): bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /docs-website by @dependabot in #10109
fix(metadata-io):Recently viewed, Recently Edited and Recently Searched section is missing in datahub home page by @siladitya2 in #10234
Update datahub-executor docs by @noggi in #10263
feat(access): Improve external role retrieval by @filipe-caetano-ovo in #10160
fix(openapi): fix structured properties mapping by @david-leifker in #10260
fix(authorization): fix restricted entity privmitives by @david-leifker in #10265
fix(ingest/mongodb): schema_metadata referenced before assignment by @sid-acryl in #10169
feat(ui/folder-structure-sort): sort folder structure alphabetically by @gaurav2733 in #10268
feat(ui/ingestion): add pagination on ingestion executions by @gaurav2733 in #10269
feat(access): Experimental policy debugger by @anshbansal in #9833
feat(docs) Update updating-datahub.md for GA4 analytics change by @chriscollins3456 in #10196
fix(docs): update docs for SchemaField.label by @hsheth2 in #10251
feat(ingest): show custom model info by @hsheth2 in #10259
fix(ingest/bigquery): Adding way to change api's batch size on schema init by @treff7es in #10255
feat(ingest/mode): Mode improvements: by @treff7es in #10273
fix(ingestion/powerbi): patch column lineage for powerbi report by @dushayntAW in #10270
fix(ingestion/lite): An index with the name aspect_idxalready exists … by @jonasHanhan in #10267
feat(ingest/looker): browse path followups by @mayurinehate in #10217
fix: revert signup page by @yoonhyejin in #10282
feat: add posts to quickstart sample data by @yoonhyejin in #10276
fix(ingestion/transformer): tranformer to replace the externalUrl in dataset properties by @dushayntAW in #10281
fix(ingestion/csv): add to support multiple ownership type for the sa… by @dushayntAW in #10287
docs: update welcome acryl doc by @anshbansal in #10280
feat(ui/backend/openapi/docs) : Add support for Business Attributes by @deepgarg-visa in #9863
feat(ingest/sigma): Sigma connector integration by @shubhamjagtap639 in #10037
feat(graph-retriever): implement graph retriever by @david-leifker in #10241
fix(ingestion/scheduler): add extraArgs support for Ingestion Scheduler (e.g. for extra pip packages) by @Nelvin73 in #10195
fix(spring): refactor spring configuration by @david-leifker in #10290
fix(ingest): improve performance of get_allowed_list in AllowDenyPattern when dealing with large lists by @Masterchen09 in #10219
fix(oidc settings): use correct path for preferredJwsAlgorithm by @darnaut in #10302
chore(ingest/presto-on-hive): Renaming presto-on-hive to hive-metastore source by @treff7es in #10278
fix(ingest): disallow src.* imports, fix powerbi/sigma by @hsheth2 in #10292
Cc fix broken cll impact analysis by @chriscollins3456 in #10303
docs: add content describing diff between datahub and acryl datahub by @shirshanka in #10301
docs: versions bump for 0.13.1 by @yoonhyejin in #10285
doc(gms/scim): SCIM API user guide by @sid-acryl in #10311
chore(docker): bump kafka docker base image by @david-leifker in #10313
fix(ui) Show edited field descriptions in schema table by @chriscollins3456 in #10314
chore(pyiceburg): set minimum version by @david-leifker in #10318
fix(ingest/tableau): handle very large filter queries by @mayurinehate in #10295
fix(ingest/databricks): handle and report config parse failure, updat… by @mayurinehate in #10261
feat(ingest/airflow): support disabling iolet materialization by @hsheth2 in #10305
feat(ingest/sigma): fix stateful ingestion by @hsheth2 in #10321
fix(ingest/profiling): compute sample row count correctly by @hsheth2 in #10319
fix(ingest/transformers): Use set to store tags in AddDatasetTags by @asikowitz in #10317
feat(views): apply views to homepage entity counts & recommendations by @ksrinath in #10283
fix(ingest): make gms url configuration resilient in rest emitter by @anshbansal in #10316
feat(ingest/profiling): allow unique count queries to be combined by @hsheth2 in #10322
fix(ingest/kafka): clarify meta-mapping docs by @hsheth2 in #10320
feat(ingest): materialize terms produced by ingestion by @hsheth2 in #10249
openapi-v3 by @david-leifker in #9550
chore(kafka-setup): bump kafka version by @david-leifker in #10329
fix: make next as default version & create redirection by @yoonhyejin in #10309
feat(ui/tasks): add pagination on tasks listing page by @gaurav2733 in #10293
feat(ingest): mark acryl cloud package first-party for logging by @hsheth2 in #10334
feat(ingest/classify): add pip dependency by @hsheth2 in #10335
feat(ingest/metabase): add ability to exclude other users collections by @paguos in #10330
chore(metadata) Addressing vulnerabilities by @rtekal in #10296
fix(ingest/bigquery): set default max_overflow to -1 by @treff7es in #10342
fix(auth-impl): handle empty entities in field resolver providers by @david-leifker in #10341
feat(ingest): bump acryl-sqlglot dep by @hsheth2 in #10343
fix(ingestion/transformer): updated transformer to avoid duplicating … by @dushayntAW in #10348
feat(schema-registry): exclude schema reg onboot check from schema re… by @david-leifker in #10349
fix(ingest/starburst): parse create_time datetime format by @ishtartec in #10345
test(ingestion/sigma): Add integration test cases by @shubhamjagtap639 in #10356
fix(ingestion/salesforce): escape markdown char for multiline description by @dushayntAW in #10351
fix(mae): fix mae standalone platform consumer by @david-leifker in #10352
fix(ingestion/qlik): Unable to ingest more than ten spaces by @shubhamjagtap639 in #10228
docker(ingestion-base): set certificate location for python by @david-leifker in #10364
build(ci): unified workflow update 1 by @david-leifker in #10353
feat(ui): Adding new analytics charts for new users, top users past month by @jjoyce0510 in #10344
feat(ingestion/bigquery): support for table clones by @camilogutierrez in #10274
build(ingest): update base requirements file by @anshbansal in #10368
feat(ingest/mssql): improve docs on using odbc by @mrjefflewis in #10370
feat(ingest/dbt): handle complex dbt sql + improve docs by @hsheth2 in #10323
feat:(entity-registry): add ability to search for float and double by @Rosmirose in #10324
fix(hazelcast): fix cache value classloading by @RyanHolstien in #10373
docs(business-attribute):add info businessAttributeEntityEnable flag by @deepgarg-visa in #10379
fix(ingest/bigquery): map date types correctly by @hsheth2 in #10383
feat(ingest/dbt): use columns from manifest as a fallback by @hsheth2 in #10374
fix(ingest/profiling): Filter tables early based on profile pattern filter by @treff7es in #10378
feat(ingest/dbt): support a datahub section in meta mappings by @hsheth2 in #10371
docs(observe): update docs for remote executor, databricks by @mayurinehate in #10393
fix(graphql) Fix entity type filter clash with legacy filters by @chriscollins3456 in #10362
fix(backend): do not lower-case dataset key parts i.e. data platform … by @ksrinath in #10385
docs(search): document default search operator by @darnaut in #10397
fix: add redirection for the past versions by @yoonhyejin in #10395
feat: add keywords for SEO by @yoonhyejin in #10358
docs: add slack utm component in docs by @yoonhyejin in #10214
perf(ingestion/fivetran): Connector performance optimization by @shubhamjagtap639 in #10346
feat(graphql): Improve logging of GraphQL requests by @darnaut in #10404
fix(ingest): map bigquery nested types properly by @hsheth2 in #10409
fix(ingestion/looker): fix lineage for dimension group column by @sid-acryl in #10382
feat(metabase): add stateful ingestion by @paguos in #10360
docs(apis): Update datahub-apis.md to add link to search example by @gabe-lyons in #10412
feat(graphql): log query name if operation name is not provided by @darnaut in #10420
DynamoDB IAM auth by @eboneil in #10419
fix(ingest/bigquery): Fixing double sanitization of urns by @treff7es in #10386
fix(ingestion/transformer): new transformer to clean user URN for DatasetUsageStatistics aspect by @dushayntAW in #10398
fix(ingestion/airflow-plugin): emit the operation aspect by @dushayntAW in #10402
feat(search): allow overriding case-sensitivity to zero by @david-leifker in #10422
fix(ci): add labeled to list of pr types for ci by @david-leifker in #10363
docs(ingest): update datahub sink doc to include an acryl example by @gabe-lyons in #10411
feat(ui) Support rich text for form descriptions by @chriscollins3456 in #10425
feat(auth): improve authentication flow logging by @darnaut in #10428
feat(upgrade): common base for mcl upgrades by @david-leifker in #10429
feat(search): autocomplete custom configuration by @david-leifker in #10426
fix(upgrade): fix upgrade npe by @david-leifker in #10436
fix(docker): use distinct empty env files by @hsheth2 in #10438
feat(ingest/snowflake): use system sampling on very large tables by @hsheth2 in #10430
fix(ingest/bigquery): remove last modified timestamp fallback by @hsheth2 in #10431
feat(cli): cache sql parsing intermediates by @hsheth2 in #10399
docs: fix blog link by @yoonhyejin in #10441
fix(ingestion/tableau): Fix tableau custom sql lineage gap by @shubhamjagtap639 in #10359
fix(changeEvents): add description-parameter to the change-event of a schemaField-description by @ksrinath in #10414
feat(ci): add linting for cypress tests by @anshbansal in #10424
feat(spark/openlineage): Openlineage 1.13.1 upgrade by @treff7es in #10433
feat(ingestion): Copy urns from previous checkpoint state on ingestion failure by @shubhamjagtap639 in #10347
fix(ingest/snowflake): add more reporting for usage aggregation, handle lineage errors by @mayurinehate in #10279
feat(docker): Enable and expose Jetty statistics by @darnaut in #10448
fix(ingest/mode): Improve query lineage by @treff7es in #10284
feat(ingest): add actorUrn for ingestion through UI by @anshbansal in #10447
fix(ingestion/airflow-plugin): warning log for non-materialized iolets by @dushayntAW in #10421
fix(ingestion/salesforce): handle the label with none value scenario by @dushayntAW in #10446
fix(ingestion): Explicitly set requirement on snowflake-connector-python to be newer or equal to 3.4.0 by @skrydal in #10445
perf(ingest): speed up urn encode happy path by @hsheth2 in #10451
feat(ingest/tableau): Fetch Upstreams From Columns by @egemenberk in #9874
docs(ingest): fix typos and clarify ingestion recipe docs by @guyr-ziprecruiter in #10405
fix(patch): update json patch library by @david-leifker in #10449
fix(metadata-service): add PE processor to component scan by @darnaut in #10462
fix(ingestion/airflow-plugin): bumping up the openlineage-airflow version by @dushayntAW in #10457
fix(ingest/tableau): catch exception during sign out by @sgomezvillamor in #10459
fix(ingest/dbt): failures due to API change by @anshbansal in #10467
fix(ingestion/kafka-connect): fixed the issue with ingestion requiring multiple substitutes by @dushayntAW in #10443
feat(ingest/cli): add some URNs per aspect for easier debugging by @anshbansal in #10468
fix(ingest/dbt): Adding fix if dbt data type is null by @treff7es in #10471
fix(docs): adjust new requirements for DynamoDB ingestion by @darnaut in #10470
feat(ingest/redshift): add timers for lineage v2 by @hsheth2 in #10460
feat(fabricType): add fabric type RVW by @eboneil in #10472
feat(structured-properties): immutable flag by @david-leifker in #10461
fix(docker): mount newly added jetty-jmx.xml by @darnaut in #10475
feat(plugins): spring custom plugins by @david-leifker in #10389
docs(impact analysis): Add column level impact analysis graphql example by @gabe-lyons in #10427
fix(entity-registry): fix plugin load error by @david-leifker in #10476
fix(openapi): fix lookupAspectSpec by @david-leifker in #10478
fix(openapi-v3): comprehensive aspect name casing fix by @david-leifker in #10484
feat(ingest/slack): Support profile ingestion using users:info by @asikowitz in #10410
docs: fix docs utms & slack footer by @yoonhyejin in #10481
feat(docs): Updating assertion docs + adding schema assertion doc by @jjoyce0510 in #10473
fix(misc): misc fixes for OSS release by @david-leifker in #10493
docs: sort feature section alphabetically by @yoonhyejin in #10400
docs(developers): add section regarding symbolic links on Windows 10/11 to developer's guide by @Masterchen09 in #10487
fix(ingestion/transformer): Add dataset domains based on tags using transformer by @sagar-salvi-apptware in #10458
chore(ingest/presto-on-hive) Set enable_properties_merge to True by default by @dushayntAW in #10469
fix(ci): documentation build fix by @anshbansal in #10507
docs: 0.3.2 Acryl by @anshbansal in #10377
feat(ingest/tableau): support platform instance mapping based off database server hostname by @richenc in #10254
fix(ingestion/looker): deduplicate the view field by @sid-acryl in #10482
fix(graphql): Support querying Posts and Queries by @asikowitz in #10502
fix(ebean): fix auto-closeable ebean dao streams by @david-leifker in #10506
feat(ingest/airflow): support BigQueryInsertJobOperator by @hsheth2 in #10452
fix(ingest): avoid using _inner_dict in urn iterator by @hsheth2 in #10492
fix(ingest/snowflake): use block sampling more conservatively by @hsheth2 in #10494
feat(sdk): add DataHubGraph.get_timeseries_values() method by @hsheth2 in #10501
fix(mcp): fix mcp key aspect by @david-leifker in #10503
fix(ingest): fix bug in incremental lineage by @hsheth2 in #10515
chore(ingest): run pyupgrade for python 3.8 by @hsheth2 in #10513
docs: update cli recommendation by @anshbansal in #10518
Wrap non-required $ref properties in an object to mark as nullable by @timothyjin in #10514
Fix formatting for #10514 by @timothyjin in #10525
feat(ingestion/glue): delta schemas by @sgomezvillamor in #10299
fix(ingestion/snowflake): fix dataclass defaults for SnowflakeReport by @ms32035 in #10529
Security/CWE 200 graphql introspection toggle by @erikkvale in #10531
feat(neo4j): neo4j pagination as per v2 scrollApi for related entities by @deepgarg-visa in #10537
docs: add api templates by @yoonhyejin in #10521
fix(ingestion/powerbi): handle special character #(tab) in native query parsing by @sid-acryl in #10520
OpenAPI v3 Spec bug fixes: by @kevin1chun in #10548
fix(assertions) aligned graphql AssertionType definition with the AssertionType defined in metadata-models by @jayacryl in #10534
fix(smoke-test): pin requests to 2.31.0 by @darnaut in #10549
fix(ingest/dbt): improve handling for CLL via ephemeral nodes by @hsheth2 in #10535
feat(connections) Add Connection entity type and graphql endpoints by @chriscollins3456 in #10550
doc(gms/scim-api): fix title and add overview by @sid-acryl in #10388
docs: add guides on forms & structured properties by @yoonhyejin in #10340
fix(graphl): fix introspection setting by @david-leifker in #10560
feat(ingest): bump acryl-sqlglot dep by @hsheth2 in #10554
feat(ingest): auto-fix duplicate schema fieldPaths by @hsheth2 in #10526
refactor(ingest): defer ctx.graph initialization by @hsheth2 in #10504
consider all values of FabricType enum in DatasetUrn util by @kevin1chun in #10564
fix(ingest/airflow): fix support for bigquery insert job operator by @hsheth2 in #10567
fix(ingest/mode): Adding Dashboards into containers by @treff7es in #10563
feat: update lineage feature guide by @yoonhyejin in #10401
docs: improve lineage docs by @yoonhyejin in #10396
fix(ingestion/powerbi): Databricks support for table lineage by @sid-acryl in #10416
fix(ingest/dbt): resolve more dbt ephemeral node lineage gaps by @hsheth2 in #10553
fix(ui) Fix preventing users from deleting personal views by @chriscollins3456 in #10510
fix(lint): fix linting by @david-leifker in #10572
build(jar): enable custom plugin lib by @david-leifker in #10552

New Contributors

@bouaouda-achraf made their first contribution in #9811
@jonasHanhan made their first contribution in #10110
@dotan-mor made their first contribution in #10226
@olgapenedo made their first contribution in #10146
@paguos made their first contribution in #10330
@ishtartec made their first contribution in #10345
@camilogutierrez made their first contribution in #10274
@mrjefflewis made their first contribution in #10370
@Rosmirose made their first contribution in #10324
@guyr-ziprecruiter made their first contribution in #10405
@sagar-salvi-apptware made their first contribution in #10458
@timothyjin made their first contribution in #10514
@erikkvale made their first contribution in #10531

Full Changelog: v0.13.1...v0.13.3rc1