This release adds several interesting improvements and many bugfixes. Lancedb destination now uses duckdb extension to let you query lance tables with SQL, ibis or sqlglot via our standard .dataset() interface. We introduced several iceberg-relates improvements (catalog support, s3 tables for Athena, advanced partitioning). There's also new fabric destination and additional options in `clickhouse_adapter. Finally: we have test environment for Oracle and we stared to fix Oracle related bugs.
Core Library
- feat(lancedb): implement
SqlClientBase- query lance tables with ibis, sqlglot or raw SQL by @zilto in #3527 - feat: S3 Tables support for
athenadestination by @jorritsandbrink in #3434 - feat:
fabricdestination (by @mattiasthalen) by @jorritsandbrink in #3535 - set iceberg advanced partitioning options by @rakesh-tmdc in #3053
- allow to use any catalog in filesystem+iceberg destination by @lfagliano in #3325
- feat: Snowflake-provided OAuth token auth by @jorritsandbrink in #3322
- Add column description support for BigQuery destination by @Travior in #3483
- feat:
clickhouse_adapterextensions by @jorritsandbrink in #3511 - 3480 chore update weaviate destination to use v4 client by @djudjuu in #3481
- add support for gcp adc with iceberg by @benjibuiltit in #3460
- 3312: feat(snowflake) - add column_comment/description hint support by @somasays in #3462
- Fix-3292: Load retrieval to only show loads that contain a schema name that is in the pipeline.schema_names by @alkaline-0 in #3446
- feat: Support OAuth and base GCP credentials for BigQuery destination by @daniel-nagish in #3382
- helper to filter dlt-changes out of schema updates by @djudjuu in #2968
Bugfixes
- pyarrow: respect resource hints before extract by @djudjuu in #3436
- Fix: 3490 better error message in schema contract application by @anuunchin in #3498
- Fix state file being ignored when pipline_name includes FILENAME_SEPA… by @Travior in #3448
- Fix/3464 sync error results in success label (workspace dashboard) by @anuunchin in #3492
- Fix/3376 load state changes in load package that changed it by @anuunchin in #3521
- 3353 normalize start method spawn seems to ignore environment variables by @djudjuu in #3463
- Fix: Connectorx arrow_stream timestamp conversion issue by @louiewhw and @anuunchin in #3528
- fix/3141 - process Oracle "table not found" exception by @ivasio in #3509
- fix: 3514 mermaid reference label by @zilto in #3515
- Fix cluster hint overriding partition hint on bigquery by @Travior in #3497
- Fix: Special handling of numeric type for oracle by @ivasio in #3144
- Fix/3159 pydantic model incorrect serialization by @tetelio in #3421
Chores
- fix(typing): add None to Container._Instance typing by @zilto in #3469
- Fix dashboard e2e tests by @burnash in #3453
- Chore: poke tests disabled by @anuunchin in #3487
- Feat/3362 - add oracle testing setup - able to test oracle-specific edge cases now by @ivasio in #3496
- chore/Omit git private key tests by @tetelio in #3502
- adds checks for plugins version in hub extra by @rudolfix in #3474
- do progress=log in vibe source template by @djudjuu in #3543
Docs
- Fix: 3467 PipelineTasksGroup Docstring fix by @anuunchin in #3482
- Docs: Converting Jupyter notebooks in education to marimo notebooks by @anuunchin in #3068
- Docs/data quality by @adrianbr in #3466
- docs: fix typo in dlt fundamentals lesson 8 by @dmschauer in #3508
- docs: fix typos on parquet page by @dmschauer in #3506
- docs/snowflake-native-app-how-to by @kaliole in #3455
- change "remove" to "keep" in Advanced Lesson 1 by @AstrakhantsevaAA in #3522
- Docs/explicit note for row hash calculation with non deterministic API responses by @anuunchin in #3524
- docs: update lancedb documentation links by @ccmao1130 in #3542
- Added docs for chunksize and use_pyarrow for parquet reader by @rishabh1815769 in #3526
New Contributors
- @rakesh-tmdc made their first contribution in #3053
- @somasays made their first contribution in #3462
- @daniel-nagish made their first contribution in #3382
- @tetelio made their first contribution in #3502
- @dmschauer made their first contribution in #3508
- @Travior made their first contribution in #3448
- @lfagliano made their first contribution in #3325
- @ccmao1130 made their first contribution in #3542
- @benjibuiltit made their first contribution in #3460
- @rishabh1815769 made their first contribution in #3526
Full Changelog: 1.20.0...1.21.0