github dlt-hub/dlt 1.32.0
on GitHub

4 hours ago

dlt 1.32.0 Release Notes

Breaking Changes

  1. Naive scd2 timestamps are no longer read as UTC (#4547 @rudolfix)
    • A naive boundary_timestamp or active_record_timestamp is sent as it is, so the database session timezone reads it. A naive active_record_timestamp may stop matching existing active records on servers whose session timezone is not UTC.
    • On destinations without timezone support, scd2 timestamps are naive UTC instead of the wall clock of the context timezone. Stored values change only if you set a non-UTC context timezone.
  2. refresh="drop_resources", pipeline.drop() and dlt pipeline drop attempt to drop all tables belonging to resources (#4582 @rudolfix) — They drop every table of the dropped resources that exists in the destination. Before, only tables that were marked as saw data were dropped.
  3. Arrow extract shares the resolved schema contract per table (#4582 @rudolfix) — Two resources with different schema_contract that write the same table share the contract resolved for the first one, as object items always did.
  4. jira and salesforce verified sources are rewritten from scratch (dlt-hub/verified-sources#717, dlt-hub/verified-sources#718 @rudolfix) — If you use the old sources (versions before 1.0.0), keep using them or refresh the pipeline.

Other Behavioral Changes

  1. scd2 boundary_timestamp applies only to the load that declares it (#4547 @rudolfix) — The boundary is stored in the load package, not in the schema. dlt no longer writes or reads the x-boundary-timestamp table hint, and a leftover hint in an existing schema is ignored. This is actually a bugfix - boundary_timestamp should be ephemeral.
  2. Load package state and trace shape (#4547 @rudolfix)
    • nested_hints of a resource are a list in the trace, each entry with its path as a list of identifiers, instead of a dict keyed by path. Each entry carries load_id, extract_idx and resource_name, and nested hint columns are YAML text. A trace loaded to a destination puts them in ..._hints__nested_hints and ..._hints__nested_hints__path.
  3. Relation schema raises when lineage fails (#4538 @rudolfix)
    • Relation.columns, columns_schema and schema raise LineageFailedException when a column cannot be resolved (an unqualified column of a table outside the dataset schema, * over such a table, a non-SELECT query). Before, they returned a guessed or empty schema while execution failed anyway. Raw query relations and relations with _allow_partial_lineage return columns without data type instead.
    • dlt.dataset.lineage.compute_columns_schema no longer accepts infer_sqlglot_schema and allow_anonymous_columns.
    • Relations derived from a raw query relation (ie. .limit()) execute as raw query. Before, they were qualified and bound.
    • On sqlglot < 27.14, a NULL literal column has no data type (was text), like on newer sqlglot.
  4. add_limit on transformers and unselected parents (#4582 @rudolfix) — A limit on a transformer is set on the resource at the top of its chain, so that resource and all its transformers get the limited pages. source.add_limit() also limits parents that are not selected, for example the filesystem resource behind readers() or a REST API parent with selected: False. Before, a limit on a transformer had no effect and unselected parents ran unlimited. sql_table(defer_table_reflect=True).add_limit(n) loads n pages instead of n - 1.
  5. Traces written by 1.32 cannot be read by older dlt (#4582 @rudolfix) — Older versions report no last trace. 1.32 reads older traces, with file counts and load sizes shown as unknown.

Highlights

  • Placeholders in merge conditions and partition pruning on nested tables (#4547 @rudolfix)
    • source_filter and destination_scope accept value placeholders: {load_id}, {table_name}, {load_package_timestamp}, {boundary_timestamp}, {curr_date}, date parts ({YYYY}, {MM}, ...) and your own extra_placeholders. These can be constants, or functions called after extract with the resource and the table name.
    • Values are stored in the load package, not in the schema, so a partition that changes on every load no longer creates a new schema version.
    • Values render as escaped destination literals. A list expands for IN, e.g. event_day IN {days}, so you can compute the partitions to replace from the extracted data.
    • A nested table with its own destination_scope in nested_hints deletes by that scope instead of a subquery on the root table, so its partitions can be pruned. With every nested table scoped, root_key can be disabled.
    • New cookbook page: "Prune BigQuery partitions when merging data".
  • Faster extract, balanced normalize and trace metrics (#4582 @rudolfix)
    • dlt pipeline <name> trace metrics shows files, rows and sizes of each step of the last run: by resource and table for extract, by table for normalize, and files, sizes, job time and failed jobs by table for load.
    • Normalize gives each worker a contiguous group of files with a similar total uncompressed size. It used to split by file count, so one worker could end up with all the files of a large table.
    • Extract warns when the files of a load package would leave at least half of the normalize workers idle, and tells you which file_max_items / file_max_bytes to set.
    • Arrow extract caches the computed table. Extract of small arrow items is up to ~3x faster.
  • replace can keep the resource state (#4575 @rudolfix) — The dict form {"disposition": "replace"} keeps the resource state by default, so incremental cursors survive a full table replace. Set exclude_resource_state: False to reset the state like the "replace" shorthand does. Closes the long-standing #2252.
  • dbt runner: microbatch intervals, load id and row counts (#4562 @rudolfix)
    • run_all() and run() accept interval, e.g. dlt.current.interval(), and pass it to dbt as --event-time-start / --event-time-end.
    • load_id is passed to the package as a variable, so models can write it to _dlt_load_id.
    • Results carry row counts.
  • salesforce: load any sObject with salesforce_table (dlt-hub/verified-sources#717, dlt-hub/verified-sources#725 @rudolfix)
    • salesforce_table(sobject=...) loads any sObject, standard or custom, like sql_table does for tables. salesforce_source keeps its 15 resources and table names and runs them in parallel.
    • Bulk API 2.0 query jobs replace Bulk API 1.0. sObjects that the Bulk API does not support are read with the REST API (api="auto" | "bulk" | "rest").
    • Column types come from the sObject describe: decimals with precision and scale for currency, date, time, timestamp, bigint and json.
    • incremental="auto" picks the cursor per sObject, start_date sets its initial value, and any dlt.sources.incremental is translated to SOQL WHERE / ORDER BY. add_limit is pushed down as SOQL LIMIT.
    • Deleted records are read with queryAll and removed from the destination (include_deleted).
    • pyarrow and python backends.
  • jira: incremental issues with comments, changelogs and worklogs (dlt-hub/verified-sources#718 @rudolfix)
    • issues load incrementally from start_date and merge on id. Pages of issues are bulk fetched in parallel.
    • Every field is a top level column typed from /field. included_columns and excluded_columns select fields.
    • New resources: issue_comments, issue_changelogs, worklogs and issue_fields.
  • pg_replication: reliable runs, direct snapshots and an arrow backend (dlt-hub/verified-sources#719, dlt-hub/verified-sources#722 @rudolfix)
    • Changes stream over one connection per run, and a run stops at the WAL position flushed when it started. The position is stored with every batch, so runs bounded with add_limit make progress.
    • init_replication(initial_snapshot="direct") reads the tables with sql_table in the snapshot exported by the slot, without copies. "unlogged_table" makes unlogged copies.
    • backend="pyarrow" decodes inserts column-wise: 173k rows/s against 23k before, for 500k rows from Postgres 17 into filesystem.
    • The snapshot and replicated changes load the same column types, and columns loaded before keep their type.
    • A role with only REPLICATION and SELECT can replicate tables it does not own. hard_delete=False keeps deleted rows with _dlt_deleted_ts set.

Core Library

  • Merge condition placeholders and nested partition pruning: see Highlights (#4547 @rudolfix)
  • New propagate_to_nested map step and merge staging options (#4547 @rudolfix)
    • propagate_to_nested in dlt.sources.helpers.transform copies root keys into nested lists of dicts, with a nesting cutoff.
    • load.truncate_staging_merge_tables_before_load keeps the staging tables of merge tables between loads.
    • Fixes: with a source_filter, dedup ranks only the merge source; the trace serializes to JSON when nested_hints use tuple paths; the merge warning on a nested table no longer fires when the root uses a merge dict.
  • trace metrics, balanced normalize and faster arrow extract: see Highlights (#4582 @rudolfix)
  • replace keeps resource state: see Highlights (#4575 @rudolfix). replace on filesystem also deletes files in parallel.
  • dbt runner context: see Highlights (#4562 @rudolfix)
  • add_limit on transformers and on unselected parents (#4582 @rudolfix) — Derived REST API endpoints and filesystem readers can now be limited. dlt warns when this replaces a different limit.
  • table_name function dispatches arrow, pandas and polars items (#4582 @rudolfix)
  • Unified lineage computation (#4538 @rudolfix) — Relation columns and query binding use the same options. _allow_partial_lineage works. Raw query relations compute partial lineage for columns / schema. dlt warns when untyped columns are dropped in the normalizer. Supports sqlglot >= 30.22.
  • connectorx 0.4.6 for decimals (#4567 @rudolfix) — dlt warns when connectorx rescales decimals in a way that loses precision. MSSQL MONEY is handled.
  • Default code input for resources without recorded inputs (#4609 @sh-rp) — Resources that record no input with add_input() get a code input pointing at their source or resource function, so custom sources show an upstream in workspace lineage. Also fixes inputs lost when a resource is renamed or cloned, e.g. sql_table(...).with_name(...).
  • Fix: compound SCD2 merge keys are matched exactly (#4245 @mattfaltyn) — Keys were concatenated, so ("a", "bc") and ("ab", "c") collided and retired an unrelated active record.
  • Fix: existing tables get ALTER when a new schema hash is first stored (#4542 @raunak2007) — A column added to a table without data in that load was never applied, and later loads failed with a missing column.
  • Fix: changing a column data type drops the old type modifiers (#4467 @developer-rpai) — precision, scale, timezone and x-nested-type of the previous type no longer stay in the schema. Fixes TypeError: not all arguments converted on Snowflake VARCHAR after decimal -> text.
  • Fix: descending incremental keeps extracting after a deduplicated boundary row (#4525 @bunnysayzz) — New rows with the same cursor value on later pages were skipped.
  • Fix: incremental step keeps its position when hints are re-applied or the resource is cloned (#4540 @rudolfix) — Steps added with insert_at after the Incremental step were moved before it in pipeline.run().
  • Fix: where() edge cases on Relation (#4570 @rudolfix) — Filters on aggregating, grouping, DISTINCT and window relations wrap the query instead of producing invalid SQL. Filters on aliases that shadow a source column filter the alias. Typed where() on a column without data type no longer raises KeyError. Explicit joins work on table functions.
  • Fix: iceberg tables with nested columns can be partitioned (#4558 @nanjeshramesh)
  • Fix: Oracle TIMESTAMP WITH LOCAL TIME ZONE maps to timezone: true (#4343 @Aryan-Pardeshi)
  • Fix: filesystem destination_fingerprint uses scheme://netloc of the bucket url (#4576 @rudolfix) — Fingerprints change for buckets whose name repeats in the path (ie. s3://data/data), bucket roots with a trailing slash, urls with a query string or inline password, upper case schemes and hf:// namespaces. s3://bucket and s3://bucket/path do not change.
  • Fix: workspace launchers set a lower bound on every dependency they add (#4521 @tetelio) — Without bounds, s3fs could resolve to 0.4.2 from 2020.
  • Fix: release fixes collected in the version bump (#4582 @rudolfix)
    • extract: contract filters of the previous arrow item no longer apply to the next one, so discard_row does not raise KeyError and data_type contracts keep valid values.
    • extract: add_limit no longer counts hints-only items as pages (sql_table(defer_table_reflect=True).add_limit(1) loaded no rows).
    • extract: dlt.transformer accepts the dict form of write_disposition in its type hints.
    • pipeline: refresh with drop_sources or drop_resources recreates dropped tables from the import schema. Importing a schema keeps the normalizer hints of existing tables.
    • destinations: the nested destination scope is verified on the prepared root table. Delta merge checks ignore nested tables without data.
    • workspace: the entry point profile is active before launchers resolve config, the workspace .gitignore no longer ignores lock files, and agent loops recognize retried tool calls on newer pydantic-ai.

Verified Sources

Docs

  • Parallel loads to one dataset from multi-tenant pipelines (#4536 @rudolfix). Also fixes ClickHouse to derive the primary key from the sort key when no primary key hint is set.
  • Adding the dlt root key to existing nested tables (#4569 @rudolfix)
  • Release highlights page for dlt 1.31 (#4533 @dlt-oss-docs-agent)
  • Generated dltHub CLI, REST API and Python SDK references (#4550 @sh-rp)
  • Syntax highlighting for CLI, tree and endpoint code blocks, and a light-mode code theme (#4616 @sh-rp)
  • Docs search moved from Algolia to pagefind (#4515 @zilto)
  • Background agents marked as public preview, with updated setup dependencies (#4522 @lis365b)
  • dltHub section in the README (#4593 @ShreyasGS)
  • Links to dltHub product pages from the dltHub docs pages (#4612 @njaltran)
  • Cross-links between related destination and source pages (#4492), distinct page titles (#4493), "Go further" links on the most-linked pages (#4494), direct /context links and aligned source counts (#4491) (@martinibach)
  • Docs worker: single permanent redirects and real 404s (#4489 @martinibach); request logs exported to the traffic-logs dataset (#4561 @djudjuu)

Chores

New Contributors

Check out latest releases or
releases around dlt-hub/dlt 1.32.0

Don't miss a new dlt release

NewReleases is sending notifications on new releases.

Get notifications