github substrait-io/duckdb-substrait-extension v1.5.6

7 hours ago

Highlights

  • DuckDB 1.5.6 and Substrait v0.102.0.
  • Subquery support in the consumer: uncorrelated subquery expressions can now be read from Substrait plans. Correlated subqueries will follow in a later release.
  • Emit mappings are honored on every relation, not only ProjectRel. Before this, a mapping such as [2, 0] on another relation type could quietly return the wrong columns.
  • Fewer crashes on malformed or unusual plans. Several plans that used to kill the process with SIGSEGV now fail with a clear error instead. These include an unset Rel or Expression, unrecognized enum values, and enum arguments on aggregates.

Behavior changes

  • NOT LIKE is now emitted as not(like(x, pattern)). It used to be emitted as not_equal, and not_equal is now always read as inequality. Plans written by earlier versions of this extension encoded NOT LIKE as not_equal, so they will now be read as !=. Those plans did not follow the spec, and the old workaround also broke real comparisons: WHERE s != 'a%c' came back as NOT LIKE. (#286)
  • When an aggregate has more than one grouping set, its grouping set index column is no longer included in the output, so the root names match the output columns. (#291)
  • Stricter emit validation. These are now rejected: mappings that are explicitly empty or whose indices are out of range, an emit on a WriteRel, and an emit on a lateral right-side FilterRel that is folded into the join condition. (#274)
  • Delete plans must filter the table they delete from. A plan that filters one table and names a different table as the delete target is now rejected, because it used to delete rows the plan never selected. (#274)
  • local_files reads bind columns to baseSchema by name, including nested fields and Hive partition columns. A mismatch with baseSchema now raises an error. (#298)

What's Changed

Features

Fixes

  • Fix local files base schema by name by @mbrobbel in #298
  • fix: support aggregates whose variant is selected by an enum argument (std_dev, variance, median) by @andrew-coleman in #294
  • fix: honor emit mappings beyond ProjectRel by @alexandrefimov in #274
  • fix: do not crash on a plan with an unset Rel or Expression oneof by @andrew-coleman in #296
  • fix: empty-table optimizer folding in get_substrait by @thgschwind in #269
  • fix: name an unsupported enum value instead of crashing on it by @alexandrefimov in #295
  • fix: emit NOT LIKE as not(like(..)) instead of not_equal by @andrew-coleman in #286
  • fix: preserve timestamp literal precision by @alexandrefimov in #275
  • fix: leave the grouping set index out of an aggregate's output by @alexandrefimov in #291
  • fix: dialect has missing variants for arithmetic aggregates by @andrew-coleman in #284
  • fix: correct two wrong DuckDB mappings in duckdb_dialect.yaml (bitwise_xor, PRECISION_TIMESTAMP_TZ) by @andrew-coleman in #283
  • fix: emit integer literals for TINYINT and unsigned integer types by @thgschwind in #280
  • fix: map substrait extract(UNIX_TIME) to date_part('epoch') by @thgschwind in #271
  • fix: emit TIMESTAMPTZ constants as precision_timestamp_tz literals by @thgschwind in #273

Build

  • build: update duckdb to 1.5.6 by @andrew-coleman in #304
  • build: update Substrait to v0.102.0 (includes window bound validation and typed RANGE offset fixes) by @bvolpato in #265

Tests & Chores

  • test: document upcast-width intent and correct TIMETZ skip comment by @thgschwind in #281
  • chore: keep CodeRabbit's summary out of the PR description by @nielspardon in #299

New Contributors

Full Changelog: v1.5.5.1...v1.5.6

Don't miss a new duckdb-substrait-extension release

NewReleases is sending notifications on new releases.