dlt-hub/dlt 1.18.0

on GitHub

A few cool feats in this release that also need your attention if you use them:

• databricks destination now supports iceberg table format, cluster and partition hints. note that previously they were ignored and mixing of cluster and partition is not allowed. in super rare cases you can get error messages that were not there previously
• signal handling went through full overhaul, (https://dlthub.com/docs/devel/running-in-production/running#allow-a-graceful-shutdown) this allows dlt to shut down load pools gracefully (previously we were raising exceptions). with console attached double CTRL-C will raise immediately. we also support graceful shutdowns for pipelines that run in thread pools. this is a behavioral change compared to 1.17.0
• we now support named destinations with shorthands dlt.pipeline(destination="warehouse") and via dlt.destination which now serves both as decorator and a factory, overall super helpful when you go from dev to production but in rare cases you may get linter and runtime error on custom destinations @dlt.destination that were not using kwargs to pass options to decorator
• Relation got to_ibis method that works both on tables and queries and which uses dlt as a backed for ibis so you can read data from them
• chores: we have new internal _workspace modules where all code for cli, dashboards, mcp (and other things that you typically not use in runtime). there were changes in internal private interfaces. as a user you do not to worry about it

Core Library

• feat: unify dlt.Relation API and create bound Ibis tables by @zilto in #3179
• fix(dashboard): 3077 remove pandas deps by @zilto in #3157
• Add custom header generation with quote_none to ArrowToCsvWriter by @burnash in #3178
• [Databricks destination] Feat/2863 databricks table optimization by @bayees in #3137
• feat:3162 add resource name to incremental extract duplicate checks logging. by @and2reak in #3164
• Fix: make store_decimal_as_integer patching conditional on pyiceberg version by @ivasio in #3185
• QoL: accept destination name as shorthand form of destination by @anuunchin in #3122
• chore/moves cli to _workspace module by @rudolfix in #3215
• ignores native config values if config spec does not implement those by @rudolfix in #3233
• avoids passing naming conventions as modules by @rudolfix in #3229
• feat: extend TTableReference by @zilto in #3093
• Feat: dlt.destination as named destination factory by @anuunchin in #3204
• Feat: resource.add_metrics implementation by @anuunchin in #3240
• Feature: Introduce support of http based resources for fs source by @TheLazzziest in #3029
• Fix: add support for yield_map in rest resource by @ivasio in #3211
• Skip cluster by in bigquery on alter statements by @adrian-173 in #3239
• Re-enable python 3.14 common tests by @sh-rp in #3242
• graceful signal handler by @rudolfix in #3234
• replace .table(..., table_type="ibis") with .to_ibis() by @zilto in #3225
• Fix: Empty columns that were previously flattened into compound ones violate freeze contract by @anuunchin in #3226
• adds more signal options by @rudolfix in #3248

Chores

• adds workspace module by @rudolfix in #3171
• chore/moves cli to _workspace module by @rudolfix in #3215
• databricks: removes cluster on create table test and allows only partition by @rudolfix in #3191
• repo(pytest): migrate to pyproject.toml and reduce verbosity by @zilto in #3205
• Feat: workspace file selector, package builder by @anuunchin in #3207
• Feat/adds workspace configuration by @rudolfix in #3221
• chore/fixes pokemon table counts by @rudolfix in #3232
• Feat/3154 convert script preprocess docs to python and add destination capabilities section to destination pages by @alkaline-0 in #3188
• Fix: workspace package hash is dependent on file order by @anuunchin in #3251

Docs

• documents vault provider by @rudolfix in #3160
• Docs: dataset access doesn't work in tutorial by @anuunchin in #3197
• Update Databricks init scipt documentation by @AndreiBondarenko in #3202
• Databricks documentation typo fix by @Magicbeanbuyer in #3217
• Fixes docs on schema file naming convention by @willi-mueller in #3244

New Contributors

• @and2reak made their first contribution in #3164
• @ivasio made their first contribution in #3185
• @Magicbeanbuyer made their first contribution in #3217
• @TheLazzziest made their first contribution in #3029
• @adrian-173 made their first contribution in #3239

Full Changelog: 1.17.1...1.18.0