Core Library
Python version support changes:
- we drop Python 3.8 support
- we add Python 3.13 support
Python 3.13 package depends on dlt-pendulum (datetime library). It is a fork of pendulum that we are going to support until we are able to merge our changes into the original repo. There are a few nice things in it (we got rid of test dependencies and now it works on Alpine linux etc.)
Note: other Python versions still use original pendulum.
A big improvement of Databricks destination:
- uses internal staging by default (Volumes), no need to declare
stagingfilesystempipeline - uses predefined staging volume or creates one automatically
- can figure out credentials if running in Databricks notebook
- can take any credentials and warehouse settings when present in the environment
overall you can run dlt pipelines in a Databrick notebook without any additional configuration
Improvements to SQLALchemy destination:
- we configured several destinations like oracle, db2, sap hana, exasol, trino etc. but still need our community help to try them out.
More stuff
- recognizes ARRAY data type and converts to nested, fixes minimal nested types handling for sqlalchemy backend by @rudolfix in #2177
- #2020 #2151 - fixes dbt 1.8.6 and arrow dict types by @rudolfix in #2175
- fix: unravel Optional to inner generic arg from instance by @otosky in #2172
- bug: support lag for non-UTC datetime cursor fields by @hairrrrr in #2170
- Added note for missing credentials error for windows authentication by @dat-a-man in #2095
- unifies handling of naive datetime for arrow and object incrementals by @rudolfix in #2184
- Force use of typeddict from typingexentions, pin poetry in tests, simple disable python 3.8 by @sh-rp in #2185
- drop python 3.8, enable python 3.13, and enable full linting for 3.12 by @sh-rp in #2194
- add downloads badge by @sh-rp in #2211
- fix: streamlit sidebar button by @zilto in #2217
- Declare pip dependency for cli extras by @jonbiemond in #2207
- #2110 adds several not found error message patterns to sqlalchemy destination by @rudolfix in #2213
- Add python 3.12 and 3.13 to bug report template by @sh-rp in #2239
- (REST API source) Enable resolve for query and json parameters by @francescomucio in #2208
- Update branch naming guide in contributing doc to add a "keep" prefix by @sh-rp in #2242
- typed entity registries by @rudolfix in #2236
- Fix: Support azure default credentials by @gaatjeniksaan in #2244
- added exception for unsupported arrow types by @zilto in #2246
- Nicer cli help output and generated cli reference by @sh-rp in #2232
- #2226 enables dbt 1.9 (also enables or fixes clickhouse, databricks and synapse profiles) by @rudolfix in #2250
- #2166 databricks direct loading by @donotpush in #2219
- #2249 sqlalchemy indexes off by default by @rudolfix in #2253
- chore: takes dlt-pendulum from pypi by @rudolfix in #2254
Docs
- Docs: fix credentials link (add slash) by @AstrakhantsevaAA in #2162
- docs: Fix typo in file name in "Create a pipeline" walkthrough by @lpillmann in #2255
- Performance page update with parallelism pitfalls section by @sh-rp in #2240
- Updated credentials docs "Added section Destination configuration via Python" by @dat-a-man in #2182
New Contributors
- @otosky made their first contribution in #2172
- @zilto made their first contribution in #2217
- @jonbiemond made their first contribution in #2207
- @gaatjeniksaan made their first contribution in #2244
- @lpillmann made their first contribution in #2255
Full Changelog: 1.5.0...1.6.0