github dlt-hub/dlt 1.29.0
on GitHub

6 hours ago

dlt 1.29.0 Release Notes

Highlights

  • ClickHouse staging-optimized replace strategy (#3927 @filipesilva) — ClickHouse now supports the staging-optimized replace strategy, performing atomic table swaps via EXCHANGE TABLES so a full replace is instantaneous and never leaves the destination in a half-loaded state.
  • AWS Secret Manager config provider (#4162 @rudolfix) — dlt can now resolve config and secrets directly from AWS Secret Manager, matching the existing Google Secret Manager provider. Also stops re-creating the Google secrets client on every lookup.
  • Explicit joins in Relation.join() (#3868 @burnash) — The dataset relation API gains explicit join support, giving you control over join conditions when composing relations.
  • Cross-destination join compatibility (#3905 @Travior) — Destination configs now expose a physical_location() accessor and can_join_with rules, so dlt knows when relations that live on different destinations (or datasets) can actually be joined together.

Core Library

Features

  • BigQuery atomic replace (#4130 @oystersuki) — New opt-in enable_atomic_replace config flag makes the truncate-and-insert strategy do a single-job, metadata-preserving WRITE_TRUNCATE_DATA load from GCS staging, giving BigQuery an atomic, transactional full replace.
  • DuckDB SQLAlchemy destination (#4161 @rudolfix) — Fully tests and fixes the SQLAlchemy destination running on DuckDB.
  • Expose Parquet compression codec (#4183 @Travior) — Configure the Parquet compression codec via DATA_WRITER__COMPRESSION or ParquetFormatConfiguration(compression=...).
  • Cross-batch schema evolution in ArrowToParquetWriter (#3896 @AyushPatel101) — Safe Arrow type promotions (e.g. float32 → float64) now work across flush batches instead of crashing once data spans multiple batches.
  • Additional Redshift COPY options (#4088 @timH6502) — New Redshift config to append extra options (e.g. FILLRECORD) to the generated COPY command for staged loads.
  • Cross-destination join compatibility (#3905 @Travior) — See Highlights.
  • Disable dlt→dlthub command proxying in workspace context (#4037 @rudolfix) — Instead of silently running a dlthub command, dlt now shows a note and points to the correct dlthub counterpart, avoiding misleading behavior.

Fixes

  • Fix: Normalize crash on empty Arrow tables with add_dlt_id (#4187 @nizar-zerrad) — Adding _dlt_id to an empty Arrow table no longer fails with ArrowInvalid.
  • Fix: correct table/resource metric aggregation (#4155 @tetelio) — Writer metrics are now aggregated correctly when jobs for the same table interleave across normalize workers or file rotation (previously undercounted items_count).
  • Fix: Airflow parallel staging truncation (#4079 @DRACULA1729) — Auto-disables staging truncation for decompose="parallel" / parallel-isolated runs so concurrent tasks no longer truncate each other's staging files.
  • Fix: move misplaced merge subqueries to post-qualify (#4169 @rudolfix) — Corrects merge SQL generation where pre-qualify subquery collapse could not work correctly.
  • Fix: switch replace strategy correctly (#4164 @rudolfix)
  • Fix: use truncate mechanism for empty "replace" tables (#4153 @rudolfix) — Empty "replace" resources are truncated via load-package state rather than empty files, avoiding hint side-effects.
  • Fix: use empty files to truncate top-level tables (#4168 @rudolfix) — Partially reverts #4153 for optimized replace strategies, where truncation happens upfront rather than inside the job transaction.
  • Fix: reuse job client when truncating staging dataset (#4207 @rudolfix)
  • Fix: keep the pipeline selector when no pipeline is selected (#4096 @lis365b) — Dashboard keeps the pipeline dropdown visible and shows a clearer empty-state message when the selection is cleared.
  • Fix: logging more frequent than log_period (#3962 @cbility) — Stops resetting the log timer on every counter update, so logs respect the configured log_period.
  • Bump jsonpath-ng and improve test coverage (#4105 @Travior) — Upgrades to jsonpath-ng 1.8.0, dropping the archived ply transitive dependency flagged by security scanners (CVE).

Docs

Chores

  • Add git commit-message rule, hook, and /commit skill (#4098 @lis365b) — Claude Code tooling to keep commit messages Conventional-Commit clean (no footers/emojis).
  • Refactor ibis handover — create ibis backend on the Destination factory (#4170 @rudolfix)
  • Relax dlthub-client deps (#4202 @rudolfix)
  • Fix Windows path for griffe tests (#4104 @rudolfix)

New Contributors

Check out latest releases or
releases around dlt-hub/dlt 1.29.0

Don't miss a new dlt release

NewReleases is sending notifications on new releases.

Get notifications