dlt-hub/dlt 1.3.0

on GitHub

Core Library

• Fix try/except in from_reference shadowing MissingDependencyException by @burnash in #1939
• prefers uv over pip if found (when creating virtual envs) by @rudolfix in #1940
• allows to plug new or updated dlt cli commands by @sh-rp in #1938
• Feat/557 rest api add oauth2clientcredentials to built in auth methods by @willi-mueller in #1871
• uses path normalize for columns in arrow tables by @rudolfix in #1947
• Added extended jsonpath_ng parser (rest_api) by @francescomucio in #1941
• Fix/1897 support https endpoints clickhouse by @sh-rp in #1931
• Fix for multiple ignores is not working (rest_api) by @burnash in #1956
• SQL Database: Support including/excluding NULL cursor values by @steinitzu in #1946
• Add references table hint and reflect them in sql_database by @steinitzu in #1925
• only truncate or delete from existing tables in refresh modes by @sh-rp in #1926
• adds bigquery partition expiration and motherduck connection string by @rudolfix in #1968

Experimental interfaces

Below we expose a new pipeline._dataset and dlt._dataset interfaces that provide unified access to data loaded into destination. We also implement duckdb-based SQL client on a filesystem destination to access data in data lakes. We'll add documentation once we stabilize dataset interface. However already now you can benefit from new cursor implementation of sql_client that allows to take data frames, arrow tables also in batches:

• dataset factory by @sh-rp in #1945
• expose readable datasets as dataframes and arrow tables by @sh-rp in #1507

PRs below adds pluggy and a few first plugin hooks. The idea is to make a lot of functionalities in dlt pluggable. Currently you can plug new cli command (or upgrade existing) and you can also plug your own runtime environment (how dlt looks for data, secrets etc.)

• adds registries and plugins by @rudolfix in #1894
• unifies run configuration and run context by @rudolfix in #1944

Docs

• Update url in deploy-with-airflow-composer.md by @FriedrichtenHagen in #1942
• Added info about backend kwargs in pyarrow by @dat-a-man in #1903
• Docs: sync styles with dlthub by @burnash in #1936
• Docs: styles: remove underline for cards in dark mode by @burnash in #1967

New Contributors

• @FriedrichtenHagen made their first contribution in #1942

Full Changelog: 1.2.0...1.3.0