dlt-hub/dlt 1.30.0 on GitHub

dlt 1.30.0 Release Notes

Breaking Changes

Failed load packages no longer auto-abort (#3557 @anuunchin) — auto_abort_on_terminal_error defaults to False. With raise_on_failed_jobs=True the job is now retried and you get a LoadClientJobTerminalRetry instead of LoadClientJobFailed, and the package stays pending. Pipeline.drop_pending_packages() and the drop-pending-packages CLI command are deprecated in favour of abort_packages / abort-packages. This has no effect on ephemeral storage, which auto-aborts by wiping the working directory. Set the flag to True to restore the old behavior.
Filesystem layouts that omit {ext} now get the extension appended (#4220 @mattfaltyn) — the documented extension fallback was unreachable, so these layouts wrote extension-less files. They now write mocked-table.jsonl, and .jsonl.gz stays intact for compressed jobs. Pipelines already running an ext-less layout will start writing different file names.
Table prefix keeps its separator (#4283 @AnnasMazhar) — with layout="{table_name}" the prefix is now event. rather than event, which is what stops a replace of event from deleting events. A new warn_unsafe_layout_separators option (default True) warns when the separator around {table_name} can also occur inside table names; set it to False to keep a layout as it is.

Highlights

Cross-destination joins (#4286 @rudolfix) — join datasets that live on different destinations. dlt attaches the foreign dataset into the query engine of the destination you read from, so the join runs in a single engine:
```
users = duckdb_pipeline.dataset().table("users")
orders = s3_pipeline.dataset().table("orders")
joined = users.join(orders, on="users.id = orders.user_id")
```
Supported for duckdb, motherduck, ducklake, lance, lancedb and filesystem, with eager and lazy materialization
Snowflake nested types (#4276 @rudolfix) — native nested-type support behind a new use_nested_types flag: automatic for parquet with nested fields, and for jsonl/dicts with an upfront columns definition via a pyarrow schema. Schema and data-type evolution both work. The user-facing interface piggy-backs on the json data type and is experimental for now.
Input/output lineage in traces (#4289 @rudolfix) — traces now carry lineage in a relational form close to OpenLineage but ingestible by dlt itself. Inputs (per resource and data location) land on the extract step, and dlt derives the outputs onto the load step.
Manual load package abort (#3557 @anuunchin) — abort_packages records what happened instead of silently deleting, and gives you the chance to inspect or retry a failed job first. Adds list_pending_retry_jobs_in_package(), fail_pending_job() and retry_failed_job() on Pipeline, a dlt pipeline <name> load-package <load-id> fail-job <job> CLI command, and an .exceptions folder in the load package where every retry writes its exception message. See Breaking Changes for the behavior change this implies.

Retryable schema migrations (#4266 @rudolfix) — a retry_schema_update helper for use with tenacity, so a failing schema migration retries with jitter and backoff instead of failing the load:

from tenacity import retry, retry_if_exception, stop_after_attempt, wait_random_exponential
from dlt.pipeline.helpers import retry_schema_update

@retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=1, max=30),
    retry=retry_if_exception(retry_schema_update()),
    reraise=True,
)
def load():
    return pipeline.run(chess_source(...))

Core Library

Cross-destination joins (#4286 @rudolfix) — see Highlights.
Snowflake nested types (#4276 @rudolfix) — see Highlights.
Input/output lineage in traces (#4289 @rudolfix) — see Highlights.
Manual load package abort (#3557 @anuunchin) — see Highlights.
Retryable schema migrations (#4266 @rudolfix) — see Highlights. Also stops creating comments on _dlt tables on databricks, which made CREATE TABLE non-atomic, and adds an option to disable comment creation altogether via TBLPROPS.
Configurable destination session time zone (#4330 @rudolfix) — session_timezone can be set on clickhouse, databricks, duckdb, ducklake, postgres, redshift, snowflake and others, guarded by a new supports_session_timezone capability. Defaults are unchanged: UTC on duckdb and snowflake, where snowflake's was previously hardcoded. duckdb's TimeZone moves from global to per-connection config, so dlt no longer changes the global timezone of a duckdb connection you pass in; set session_timezone to an empty string to keep the duckdb default of the machine timezone.
add_limit on the source factory (#4223 @richacode007-byte) — @dlt.source factories accept .add_limit() before instantiation, for both sync and async sources, and return the factory for chaining. Factory clones preserve the limit.
Profile and config-value location in traces (#4303 @rudolfix) — the run context in a trace now reports the active profile, and the resolver trace records where each config or secret value came from, down to the exact profile file (dev.secrets.toml), vault identity or AWS account, region and prefix. See Breaking Changes for the trace shape change.
instance key in the job require spec (#4262 @tetelio) — a job can declare its runner instance requirements as an open dict, with instance.size (for example {"size": "medium"}) read today. The legacy machine key still works and is still serialized, but now emits a DltDeprecationWarning, so existing manifests and deployments are unaffected.
Readable __repr__ for WorkspaceRunContext (#4022 @sohamwaghe) — shows the run dir and profile instead of an opaque object in a REPL or debugger.
Fix: include initialize_storage in schema-update error handling (#4325 @rudolfix) — CREATE SCHEMA was excluded from the retry-only-the-schema-update path.
Fix: add a dot separator to get_table_prefix_layout (#4283 @AnnasMazhar) — stops a replace of event from deleting events. See Breaking Changes for the delta and iceberg table move and the new warn_unsafe_layout_separators option.
Fix: preserve the import schema after a restore (#4251 @AkhilTrivediX) — schemas restored from a destination were linked to the current import schema hash without containing the imported changes, so the next load skipped the import.
Fix: migrate stored schemas read from a destination (#4239 @bartcode) — four raw-JSON loaders used Schema.from_stored_schema, which skips migrate_schema and apply_defaults. dlt.dataset() and the dashboard schema-version loader therefore could not read schemas written by older dlt versions, raising KeyError: 'name' on any naming normalization.
Fix: build boolean sqlglot literals as sge.Boolean (#4231 @chuenchen309) — bool fell into the int/float branch and produced a malformed numeric literal that crashed annotate_types() with decimal.InvalidOperation in Relation.where() and .filter().
Fix: keep identifiers case-sensitive in the sqlglot schema (#4269 @rudolfix) — sqlglot 30.13.0 started normalizing identifiers in more cases.
Fix: pass the load-id column name to remove_columns as a sequence (#4241 @chuenchen309) — add_dlt_load_id_column silently dropped RecordBatch columns whose name is a substring of _dlt_load_id (id, load_id, load, dlt, …), because membership against a bare str is a substring test.
Fix: quote stage references and file paths in Snowflake local file loads (#4281 @burnash)
Fix: default the athena dbt work_group to primary (#4282 @burnash) — newer dbt-athena passes an empty value straight through.
Fix: apply extra_placeholders in get_table_prefix (#4226 @axelray-dev) — when extra_placeholders overrode {schema_name}, replace-truncation and table listing matched zero files and silently left old data in place.
Fix: append the extension when a filesystem layout omits {ext} (#4220 @mattfaltyn) — see Breaking Changes for the resulting file name change.
Fix: allow JWT auth without scopes (#4235 @mattfaltyn) — OAuthJWTAuth.scopes is optional and defaults to None, but __post_init__ passed it to str.join, raising TypeError before authentication could begin.
Fix: handle a non-scalar total in the range paginator (#4186 @anxkhn) — int(total) raises TypeError, not ValueError, when total_path resolves to an object or array, so pagination failed opaquely instead of raising the paginator's actionable error.
Fix: preserve paginator stop conditions (#4227 @mattfaltyn) — an API returning has_more=true could override a stop already decided by maximum_offset, maximum_page or the response total, or re-enable a cursor paginator with no valid cursor. has_more=false can still stop pagination early.
Fix: remove ModuleType.__dict__ usage infringing PEP 562 (#4215 @zilto)
Warn about ephemeral storage in load package errors (#4211 @tetelio) — the error now says that a machine with ephemeral storage will not retry the package on the next run, because the files are gone from disk.

Docs

AI Harness section (#4270 @ShreyasGS) — new hub pages covering what the AI Harness is and its four artifact kinds, a catalog of the shipped toolkits mapped to the ingest → validate → transform → deploy → observe lifecycle, and installation with uvx dlthub-init@latest.
Inviting members, org and workspace roles (#4082 @sh-rp) — invite-by-email and accept-on-sign-in, revoking pending invites, the full role model including the organization guest and workspace developer roles, how invites and roles combine, and the safety rules for managing members.
Instance size in job configuration (#4293, #4292 @tetelio) — configuring a job's instance size, including disk, with a pointer to optimizing dlt.
Instance-size preview banner, platform-tutorial removed (#4294 @ShreyasGS) — deletes hub/getting-started/platform-tutorial.md and repoints the redirects at pipeline-operations/deployments so old URLs keep resolving.
README refresh (#4083 @elviskahoro) — rewritten as a feature tour with runnable code: a no-auth rest_api_source doing the full describe → load → dataset().pokemon.df() round trip, plus sql_database, filesystem | read_csv_duckdb and direct pandas/Polars/Arrow. Meta sections are kept intact.
Replace the dead playground link in the README (#4267 @elviskahoro) — points at the dlt + Hugging Face marimo notebook on molab alongside the existing Colab demo.
Revised contribution guidelines and a new AI-assisted policy (#4252 @burnash) — a shorter, more predictable path from "I want to help" to "merged", steering new contributors to issues labelled help wanted or good first issue.
Remove an outdated IP from the docs (#4238 @VioletM)
Fix typos in comments, docstrings and docs (#4189 @maxtaran2010)

Chores

agentic-docs workflow (#4314 @ShreyasGS) — the trigger for the bot that drafts documentation pages from labelled issues and revises them from review feedback. The workflow only checks an allowlist and starts a Cloud Run job, so only numeric IDs cross the boundary and no external text passes through a shell step.
Do not start an agentic-docs run for the bot's own comments (#4323 @ShreyasGS) — the bot's own /position example in an issue comment matched the workflow's trigger and the run began writing the wrong page.
Arm the agentic-docs run on the agent label, not the label set (#4338 @ShreyasGS) — applying documentation and agent together sends two labeled webhooks and both matched. agent now arms the run, and the /position comment arm is removed.
Preserve dependency resolution across test jobs (#4264 @Travior) — propagates matrix resolution settings through UV_SYNC_ARGS and stops uv run from resyncing prepared CI environments via UV_NO_SYNC. Also raises the GitPython minimum and keeps the SQLGlot compatibility test working across supported versions.
Rerun remote destinations on transient connection errors (#4243 @burnash) — widens the pytest-rerunfailures filters for mssql, synapse and fabric ODBC transients and for databricks DatabaseTransientException. Benign assertion failures still fail fast.
Stop the flaky gather_metrics test on an early sleep return (#4236 @burnash)

New Contributors

@sohamwaghe made their first contribution in #4022
@anxkhn made their first contribution in #4186
@maxtaran2010 made their first contribution in #4189
@mattfaltyn made their first contribution in #4220
@richacode007-byte made their first contribution in #4223
@axelray-dev made their first contribution in #4226
@chuenchen309 made their first contribution in #4231
@bartcode made their first contribution in #4239
@AkhilTrivediX made their first contribution in #4251