What's Changed
- fix(ingest/bigquery): Supporting lineage extraction in case the select query result's target table is set on job by @treff7es in #10191
- fix(retention): fix time-based retention by @trialiya in #10118
- feat(lineage): give via and paths in entity lineage response by @RyanHolstien in #10192
- fix(ingestion/datahub): implemented the filter to ignore/include URN for ingestion by @dushayntAW in #10174
- fix(ingestion/glue): fix to ingest the comment for partition key as description by @dushayntAW in #10189
- feat(ingest/looker): cleanup usage generation code by @hsheth2 in #10153
- fix(dev): fix env file overrides for profiles by @hsheth2 in #10194
- fix(ingestion/hive): ignore sampling for tagged column/table by @dushayntAW in #10096
- fix(ui/property): add collapse for long dataset properties by @gaurav2733 in #10203
- saas release v0.3.1 release notes by @david-leifker in #10205
- fix(ingest/databricks): pin pandas for databricks ingestion by @mayurinehate in #10204
- Fixed issue where the custom defined aspects were missing from the API specification. by @ajoymajumdar in #10208
- feat(ingestion/transformer): Handle overlapping while mapping in extract ownership from tags transformer by @shubhamjagtap639 in #10201
- fix(build): avoid nested gradle commands by @hsheth2 in #10198
- feat(ingest/great_expectations): support in-memory (Pandas) data assets by @bouaouda-achraf in #9811
- ci(workflow): publish docker from pr with label by @david-leifker in #10193
- bump(version): bump classgraph version, add early package filter by @david-leifker in #10207
- fix(ingestion/mongodb): MongoDB source unable to parse datetimes with years > 9999 by @jonasHanhan in #10110
- fix(graphql-core): DomainEntitiesResolver does not support values FacetFilterInput parameter by @siladitya2 in #10188
- fix(graphql-core):Auto completion/suggestion of Domains are not working by @siladitya2 in #10150
- chore(usage-stats): measure time for getting buckets and aggregations by @darnaut in #10220
- test(search): introduce retry for search test by @david-leifker in #10206
- feat(ingest/bigquery): fix support for incremental column lineage by @hsheth2 in #10222
- fix(ingest/dbt): better dbt timestamp parsing by @hsheth2 in #10223
- feat(ingest/sql): normalize bigquery partitioned tables when parsing by @hsheth2 in #10224
- docs: fix feedback button design by @yoonhyejin in #10182
- docs: add discourse to community tab by @yoonhyejin in #10181
- docs: edit the text and destination for sign up link by @yoonhyejin in #10183
- fix(ingestion/datahub): moved urn_pattern config to source config by @dushayntAW in #10215
- fix(ingestion/airflow-plugin): ingesting the tags along with the data by @dushayntAW in #10216
- fix(ingest): suppress all column-level parsing errors by @hsheth2 in #10211
- fix(ci): unified workflow login logic by @david-leifker in #10235
- fix(lineage): fix lighting cache dataJob platform by @david-leifker in #10233
- feat(vianode): v3 of cll via datajob update by @david-leifker in #10221
- chore(build): bump actions versions by @david-leifker in #10240
- fix(ingest): avoid requiring sqlalchemy for dynamodb classification by @hsheth2 in #10213
- docs(cli/init): make datahub init docs more clear by @gabe-lyons in #10245
- feat(ingest/redshift): filter out system queries from usage by @hsheth2 in #10247
- feat(gql): support operationName by @hsheth2 in #10210
- fix(frontend): fix frontend script used in release checklist by @david-leifker in #10243
- docs(init): Update entrypoints.py to be more clear about acryl init by @gabe-lyons in #10248
- fix(airflow): disable OL regardless of plugin status by @hsheth2 in #10250
- fix(ingestion/salesforce): added additional check for description by @dushayntAW in #10239
- feat(api): Add description parameter to editable dataset change entity event by @eboneil in #10237
- fix(ingest/bigquery): fix lineage if multiple sql expression passed in and destination table set by @treff7es in #10212
- feat(ingest/nifi): ingest process group as browse path v2, incremental lineage by @mayurinehate in #10202
- fix publish-datahub-jars workflow by @david-leifker in #10244
- fix(ingest/unity): Fix bug around unity notebook ingestion by @asikowitz in #10253
- feat(ingest/cockroachdb): add cockroachdb ingestion by @dotan-mor in #10226
- feat(ingestion/bigquery): support patterns for label -> tag capture by @olgapenedo in #10146
- feat(ingest/fivetran): use emails in owner user urns by @shubhamjagtap639 in #10229
- feat(cli): Make yaml loaders compatible with pydantic v2 by @eboneil in #10257
- fix(ingest): support pydantic v2 with properties subcommand by @hsheth2 in #10256
- feat(ingestion): Add
-e
flag touv
command in ingestion Dockerfiles by @skrydal in #10114 - fix(quickstart): remove unneeded init.sql by @darnaut in #10266
- fix(ingestion/airflow-plugin): replace deprecated calls by @ms32035 in #10238
- build(deps): bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /docs-website by @dependabot in #10109
- fix(metadata-io):Recently viewed, Recently Edited and Recently Searched section is missing in datahub home page by @siladitya2 in #10234
- Update datahub-executor docs by @noggi in #10263
- feat(access): Improve external role retrieval by @filipe-caetano-ovo in #10160
- fix(openapi): fix structured properties mapping by @david-leifker in #10260
- fix(authorization): fix restricted entity privmitives by @david-leifker in #10265
- fix(ingest/mongodb): schema_metadata referenced before assignment by @sid-acryl in #10169
- feat(ui/folder-structure-sort): sort folder structure alphabetically by @gaurav2733 in #10268
- feat(ui/ingestion): add pagination on ingestion executions by @gaurav2733 in #10269
- feat(access): Experimental policy debugger by @anshbansal in #9833
- feat(docs) Update updating-datahub.md for GA4 analytics change by @chriscollins3456 in #10196
- fix(docs): update docs for SchemaField.label by @hsheth2 in #10251
- feat(ingest): show custom model info by @hsheth2 in #10259
- fix(ingest/bigquery): Adding way to change api's batch size on schema init by @treff7es in #10255
- feat(ingest/mode): Mode improvements: by @treff7es in #10273
- fix(ingestion/powerbi): patch column lineage for powerbi report by @dushayntAW in #10270
- fix(ingestion/lite): An index with the name aspect_idxalready exists … by @jonasHanhan in #10267
- feat(ingest/looker): browse path followups by @mayurinehate in #10217
- fix: revert signup page by @yoonhyejin in #10282
- feat: add posts to quickstart sample data by @yoonhyejin in #10276
- fix(ingestion/transformer): tranformer to replace the externalUrl in dataset properties by @dushayntAW in #10281
- fix(ingestion/csv): add to support multiple ownership type for the sa… by @dushayntAW in #10287
- docs: update welcome acryl doc by @anshbansal in #10280
- feat(ui/backend/openapi/docs) : Add support for Business Attributes by @deepgarg-visa in #9863
- feat(ingest/sigma): Sigma connector integration by @shubhamjagtap639 in #10037
- feat(graph-retriever): implement graph retriever by @david-leifker in #10241
- fix(ingestion/scheduler): add extraArgs support for Ingestion Scheduler (e.g. for extra pip packages) by @Nelvin73 in #10195
- fix(spring): refactor spring configuration by @david-leifker in #10290
- fix(ingest): improve performance of get_allowed_list in AllowDenyPattern when dealing with large lists by @Masterchen09 in #10219
- fix(oidc settings): use correct path for preferredJwsAlgorithm by @darnaut in #10302
- chore(ingest/presto-on-hive): Renaming presto-on-hive to hive-metastore source by @treff7es in #10278
- fix(ingest): disallow src.* imports, fix powerbi/sigma by @hsheth2 in #10292
- Cc fix broken cll impact analysis by @chriscollins3456 in #10303
- docs: add content describing diff between datahub and acryl datahub by @shirshanka in #10301
- docs: versions bump for 0.13.1 by @yoonhyejin in #10285
- doc(gms/scim): SCIM API user guide by @sid-acryl in #10311
- chore(docker): bump kafka docker base image by @david-leifker in #10313
- fix(ui) Show edited field descriptions in schema table by @chriscollins3456 in #10314
- chore(pyiceburg): set minimum version by @david-leifker in #10318
- fix(ingest/tableau): handle very large filter queries by @mayurinehate in #10295
- fix(ingest/databricks): handle and report config parse failure, updat… by @mayurinehate in #10261
- feat(ingest/airflow): support disabling iolet materialization by @hsheth2 in #10305
- feat(ingest/sigma): fix stateful ingestion by @hsheth2 in #10321
- fix(ingest/profiling): compute sample row count correctly by @hsheth2 in #10319
- fix(ingest/transformers): Use set to store tags in AddDatasetTags by @asikowitz in #10317
- feat(views): apply views to homepage entity counts & recommendations by @ksrinath in #10283
- fix(ingest): make gms url configuration resilient in rest emitter by @anshbansal in #10316
- feat(ingest/profiling): allow unique count queries to be combined by @hsheth2 in #10322
- fix(ingest/kafka): clarify meta-mapping docs by @hsheth2 in #10320
- feat(ingest): materialize terms produced by ingestion by @hsheth2 in #10249
- openapi-v3 by @david-leifker in #9550
- chore(kafka-setup): bump kafka version by @david-leifker in #10329
- fix: make next as default version & create redirection by @yoonhyejin in #10309
- feat(ui/tasks): add pagination on tasks listing page by @gaurav2733 in #10293
- feat(ingest): mark acryl cloud package first-party for logging by @hsheth2 in #10334
- feat(ingest/classify): add pip dependency by @hsheth2 in #10335
- feat(ingest/metabase): add ability to exclude other users collections by @paguos in #10330
- chore(metadata) Addressing vulnerabilities by @rtekal in #10296
- fix(ingest/bigquery): set default
max_overflow
to -1 by @treff7es in #10342 - fix(auth-impl): handle empty entities in field resolver providers by @david-leifker in #10341
- feat(ingest): bump acryl-sqlglot dep by @hsheth2 in #10343
- fix(ingestion/transformer): updated transformer to avoid duplicating … by @dushayntAW in #10348
- feat(schema-registry): exclude schema reg onboot check from schema re… by @david-leifker in #10349
- fix(ingest/starburst): parse create_time datetime format by @ishtartec in #10345
- test(ingestion/sigma): Add integration test cases by @shubhamjagtap639 in #10356
- fix(ingestion/salesforce): escape markdown char for multiline description by @dushayntAW in #10351
- fix(mae): fix mae standalone platform consumer by @david-leifker in #10352
- fix(ingestion/qlik): Unable to ingest more than ten spaces by @shubhamjagtap639 in #10228
- docker(ingestion-base): set certificate location for python by @david-leifker in #10364
- build(ci): unified workflow update 1 by @david-leifker in #10353
- feat(ui): Adding new analytics charts for new users, top users past month by @jjoyce0510 in #10344
- feat(ingestion/bigquery): support for table clones by @camilogutierrez in #10274
- build(ingest): update base requirements file by @anshbansal in #10368
- feat(ingest/mssql): improve docs on using odbc by @mrjefflewis in #10370
- feat(ingest/dbt): handle complex dbt sql + improve docs by @hsheth2 in #10323
- feat:(entity-registry): add ability to search for float and double by @Rosmirose in #10324
- fix(hazelcast): fix cache value classloading by @RyanHolstien in #10373
- docs(business-attribute):add info businessAttributeEntityEnable flag by @deepgarg-visa in #10379
- fix(ingest/bigquery): map date types correctly by @hsheth2 in #10383
- feat(ingest/dbt): use columns from manifest as a fallback by @hsheth2 in #10374
- fix(ingest/profiling): Filter tables early based on profile pattern filter by @treff7es in #10378
- feat(ingest/dbt): support a
datahub
section in meta mappings by @hsheth2 in #10371 - docs(observe): update docs for remote executor, databricks by @mayurinehate in #10393
- fix(graphql) Fix entity type filter clash with legacy filters by @chriscollins3456 in #10362
- fix(backend): do not lower-case dataset key parts i.e. data platform … by @ksrinath in #10385
- docs(search): document default search operator by @darnaut in #10397
- fix: add redirection for the past versions by @yoonhyejin in #10395
- feat: add keywords for SEO by @yoonhyejin in #10358
- docs: add slack utm component in docs by @yoonhyejin in #10214
- perf(ingestion/fivetran): Connector performance optimization by @shubhamjagtap639 in #10346
- feat(graphql): Improve logging of GraphQL requests by @darnaut in #10404
- fix(ingest): map bigquery nested types properly by @hsheth2 in #10409
- fix(ingestion/looker): fix lineage for dimension group column by @sid-acryl in #10382
- feat(metabase): add stateful ingestion by @paguos in #10360
- docs(apis): Update datahub-apis.md to add link to search example by @gabe-lyons in #10412
- feat(graphql): log query name if operation name is not provided by @darnaut in #10420
- DynamoDB IAM auth by @eboneil in #10419
- fix(ingest/bigquery): Fixing double sanitization of urns by @treff7es in #10386
- fix(ingestion/transformer): new transformer to clean user URN for DatasetUsageStatistics aspect by @dushayntAW in #10398
- fix(ingestion/airflow-plugin): emit the operation aspect by @dushayntAW in #10402
- feat(search): allow overriding case-sensitivity to zero by @david-leifker in #10422
- fix(ci): add labeled to list of pr types for ci by @david-leifker in #10363
- docs(ingest): update datahub sink doc to include an acryl example by @gabe-lyons in #10411
- feat(ui) Support rich text for form descriptions by @chriscollins3456 in #10425
- feat(auth): improve authentication flow logging by @darnaut in #10428
- feat(upgrade): common base for mcl upgrades by @david-leifker in #10429
- feat(search): autocomplete custom configuration by @david-leifker in #10426
- fix(upgrade): fix upgrade npe by @david-leifker in #10436
- fix(docker): use distinct empty env files by @hsheth2 in #10438
- feat(ingest/snowflake): use system sampling on very large tables by @hsheth2 in #10430
- fix(ingest/bigquery): remove last modified timestamp fallback by @hsheth2 in #10431
- feat(cli): cache sql parsing intermediates by @hsheth2 in #10399
- docs: fix blog link by @yoonhyejin in #10441
- fix(ingestion/tableau): Fix tableau custom sql lineage gap by @shubhamjagtap639 in #10359
- fix(changeEvents): add description-parameter to the change-event of a schemaField-description by @ksrinath in #10414
- feat(ci): add linting for cypress tests by @anshbansal in #10424
- feat(spark/openlineage): Openlineage 1.13.1 upgrade by @treff7es in #10433
- feat(ingestion): Copy urns from previous checkpoint state on ingestion failure by @shubhamjagtap639 in #10347
- fix(ingest/snowflake): add more reporting for usage aggregation, handle lineage errors by @mayurinehate in #10279
- feat(docker): Enable and expose Jetty statistics by @darnaut in #10448
- fix(ingest/mode): Improve query lineage by @treff7es in #10284
- feat(ingest): add actorUrn for ingestion through UI by @anshbansal in #10447
- fix(ingestion/airflow-plugin): warning log for non-materialized iolets by @dushayntAW in #10421
- fix(ingestion/salesforce): handle the label with none value scenario by @dushayntAW in #10446
- fix(ingestion): Explicitly set requirement on snowflake-connector-python to be newer or equal to 3.4.0 by @skrydal in #10445
- perf(ingest): speed up urn encode happy path by @hsheth2 in #10451
- feat(ingest/tableau): Fetch Upstreams From Columns by @egemenberk in #9874
- docs(ingest): fix typos and clarify ingestion recipe docs by @guyr-ziprecruiter in #10405
- fix(patch): update json patch library by @david-leifker in #10449
- fix(metadata-service): add PE processor to component scan by @darnaut in #10462
- fix(ingestion/airflow-plugin): bumping up the openlineage-airflow version by @dushayntAW in #10457
- fix(ingest/tableau): catch exception during sign out by @sgomezvillamor in #10459
- fix(ingest/dbt): failures due to API change by @anshbansal in #10467
- fix(ingestion/kafka-connect): fixed the issue with ingestion requiring multiple substitutes by @dushayntAW in #10443
- feat(ingest/cli): add some URNs per aspect for easier debugging by @anshbansal in #10468
- fix(ingest/dbt): Adding fix if dbt data type is null by @treff7es in #10471
- fix(docs): adjust new requirements for DynamoDB ingestion by @darnaut in #10470
- feat(ingest/redshift): add timers for lineage v2 by @hsheth2 in #10460
- feat(fabricType): add fabric type RVW by @eboneil in #10472
- feat(structured-properties): immutable flag by @david-leifker in #10461
- fix(docker): mount newly added jetty-jmx.xml by @darnaut in #10475
- feat(plugins): spring custom plugins by @david-leifker in #10389
- docs(impact analysis): Add column level impact analysis graphql example by @gabe-lyons in #10427
- fix(entity-registry): fix plugin load error by @david-leifker in #10476
- fix(openapi): fix lookupAspectSpec by @david-leifker in #10478
- fix(openapi-v3): comprehensive aspect name casing fix by @david-leifker in #10484
- feat(ingest/slack): Support profile ingestion using users:info by @asikowitz in #10410
- docs: fix docs utms & slack footer by @yoonhyejin in #10481
- feat(docs): Updating assertion docs + adding schema assertion doc by @jjoyce0510 in #10473
- fix(misc): misc fixes for OSS release by @david-leifker in #10493
- docs: sort feature section alphabetically by @yoonhyejin in #10400
- docs(developers): add section regarding symbolic links on Windows 10/11 to developer's guide by @Masterchen09 in #10487
- fix(ingestion/transformer): Add dataset domains based on tags using transformer by @sagar-salvi-apptware in #10458
- chore(ingest/presto-on-hive) Set enable_properties_merge to True by default by @dushayntAW in #10469
- fix(ci): documentation build fix by @anshbansal in #10507
- docs: 0.3.2 Acryl by @anshbansal in #10377
- feat(ingest/tableau): support platform instance mapping based off database server hostname by @richenc in #10254
- fix(ingestion/looker): deduplicate the view field by @sid-acryl in #10482
- fix(graphql): Support querying Posts and Queries by @asikowitz in #10502
- fix(ebean): fix auto-closeable ebean dao streams by @david-leifker in #10506
- feat(ingest/airflow): support BigQueryInsertJobOperator by @hsheth2 in #10452
- fix(ingest): avoid using
_inner_dict
in urn iterator by @hsheth2 in #10492 - fix(ingest/snowflake): use block sampling more conservatively by @hsheth2 in #10494
- feat(sdk): add DataHubGraph.get_timeseries_values() method by @hsheth2 in #10501
- fix(mcp): fix mcp key aspect by @david-leifker in #10503
- fix(ingest): fix bug in incremental lineage by @hsheth2 in #10515
- chore(ingest): run pyupgrade for python 3.8 by @hsheth2 in #10513
- docs: update cli recommendation by @anshbansal in #10518
- Wrap non-required $ref properties in an object to mark as nullable by @timothyjin in #10514
- Fix formatting for #10514 by @timothyjin in #10525
- feat(ingestion/glue): delta schemas by @sgomezvillamor in #10299
- fix(ingestion/snowflake): fix dataclass defaults for SnowflakeReport by @ms32035 in #10529
- Security/CWE 200 graphql introspection toggle by @erikkvale in #10531
- feat(neo4j): neo4j pagination as per v2 scrollApi for related entities by @deepgarg-visa in #10537
- docs: add api templates by @yoonhyejin in #10521
- fix(ingestion/powerbi): handle special character #(tab) in native query parsing by @sid-acryl in #10520
- OpenAPI v3 Spec bug fixes: by @kevin1chun in #10548
- fix(assertions) aligned graphql AssertionType definition with the AssertionType defined in metadata-models by @jayacryl in #10534
- fix(smoke-test): pin requests to 2.31.0 by @darnaut in #10549
- fix(ingest/dbt): improve handling for CLL via ephemeral nodes by @hsheth2 in #10535
- feat(connections) Add Connection entity type and graphql endpoints by @chriscollins3456 in #10550
- doc(gms/scim-api): fix title and add overview by @sid-acryl in #10388
- docs: add guides on forms & structured properties by @yoonhyejin in #10340
- fix(graphl): fix introspection setting by @david-leifker in #10560
- feat(ingest): bump acryl-sqlglot dep by @hsheth2 in #10554
- feat(ingest): auto-fix duplicate schema fieldPaths by @hsheth2 in #10526
- refactor(ingest): defer ctx.graph initialization by @hsheth2 in #10504
- consider all values of FabricType enum in DatasetUrn util by @kevin1chun in #10564
- fix(ingest/airflow): fix support for bigquery insert job operator by @hsheth2 in #10567
- fix(ingest/mode): Adding Dashboards into containers by @treff7es in #10563
- feat: update lineage feature guide by @yoonhyejin in #10401
- docs: improve lineage docs by @yoonhyejin in #10396
- fix(ingestion/powerbi): Databricks support for table lineage by @sid-acryl in #10416
- fix(ingest/dbt): resolve more dbt ephemeral node lineage gaps by @hsheth2 in #10553
- fix(ui) Fix preventing users from deleting personal views by @chriscollins3456 in #10510
- fix(lint): fix linting by @david-leifker in #10572
- build(jar): enable custom plugin lib by @david-leifker in #10552
New Contributors
- @bouaouda-achraf made their first contribution in #9811
- @jonasHanhan made their first contribution in #10110
- @dotan-mor made their first contribution in #10226
- @olgapenedo made their first contribution in #10146
- @paguos made their first contribution in #10330
- @ishtartec made their first contribution in #10345
- @camilogutierrez made their first contribution in #10274
- @mrjefflewis made their first contribution in #10370
- @Rosmirose made their first contribution in #10324
- @guyr-ziprecruiter made their first contribution in #10405
- @sagar-salvi-apptware made their first contribution in #10458
- @timothyjin made their first contribution in #10514
- @erikkvale made their first contribution in #10531
Full Changelog: v0.13.1...v0.13.3