dlt-hub/dlt 1.15.0 on GitHub

Breaking changes

This version will add .gz extensions to files that are compressed. That includes filesystem destinations, internal working directory and staging locations used to feed other destinations. A few practical hints:

Existing filesystem destination will continue storing files without gz extension and they are not affected by the change (existing datasets will retain their behavior where this extension is not added for backwards compatibility)
Compressed files uploaded to staging destinations will now have the .gz extension, also if dlt is configured to keep data in stage
This does not apply to parquet files.
More information can be found in the filesystem destination docs: https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem#file-compression

Core Library

[Databricks destination] Adding comment and tags for table and columns and applying primary and foreign key constraints in Unity Catalog by @bayees in #2674
feat - add crlf support for csv exports by @7amza79 in #2783
feat: add has_more boolean flag logic to RESTClient OffsetPaginator by @michaelconan in #2817
rest_api: fix: make ProcessingSteps filter and map fields optional by @burnash in #2913
Enable and test python 3.14 support by @sh-rp in #2789
removes init files from dlt tables in filesystem by @rudolfix in #2868
restclient: json param range paginator by @Giackgamba in #2917
fix sync destination warning logging call by @sh-rp in #2927
fix: missing __repr__ for @dlt.transformation by @zilto in #2940
fix: restclient: handle null data in response by @burnash in #2936
Fix: saving compressed load files with .gz extension by @anuunchin in #2835
fix: prevent DuplicateSchema error when using public schema in Redshift by @franloza in #2953
feat: Schema.to_dbml(), auto export schemas in dbml format by @zilto in #2929
QoL: improve DataValidationError output: use identifying columns if present by @djudjuu in #2915
callback collector by @djudjuu in #2922
skips inferring incomplete column when already incomplete by @rudolfix in #2935
2946 sqlalchemy destination fixes (full support for mssq, partial for trino) by @rudolfix in #2951
adds precision to _dlt_load_id and _dlt_id columns by @rudolfix in #2951
adds json field support for mssql by @rudolfix in #2951
fixes clickhouse temporary table engine not propagate to nodes (failed merges fix) by @rudolfix in #2951
fixes BIGQUERY numeric creation (when scale was set to 0) by @rudolfix in #2951
fix: replace arrow2 with arrow backend for connectorx, enables newest connectorx versions by @zilto in #2933
AI Command: extended with IDEs (rules for all major IDEs are supported) by @anuunchin in #2937
duckdb bumped to 1.3.2, iceberg scanners updated by @rudolfix in #2958
Feat: Allow control over streamed_exec in delta merge upsert by @anuunchin in #2961
fix failing top level module imports on projects in dirs that start with a dot by @sh-rp in #2963

Docs

Fix docusaurus / netlify trailing slash issue by @sh-rp in #2878
make docs snippets tests use local secrets by @sh-rp in #2903
2784: update dlt+ pojects.md sources example by @kaliole in #2861
Docs: Some additions to the filesystem gdrive source by @anuunchin in #2912
Release highlights 1.12.3-1.14.1 by @AstrakhantsevaAA in #2939
Updating custom configurations with @configspec decorator by @dat-a-man in #2826
docs: rest_api: add tip for escaping curly braces by @burnash in #2925
link to add_map and add_yield_map usage example by @molkazhani2001 in #2916
adjust cursor docs to new flow by @adrianbr in #2885
Optimize add_limit docs title for the search by @VioletM in #2949
Docs/dlt plus project docs rest api restructuring by @kaliole in #2911
Updated merge loading docs on scd2 strategy handling nested structures by @dat-a-man in #2944

New Contributors

@bayees made their first contribution in #2674
@7amza79 made their first contribution in #2783
@michaelconan made their first contribution in #2817
@Giackgamba made their first contribution in #2917

Full Changelog: 1.14.1...1.15.0