dataform-co/dataform 3.0.0-beta.0

on GitHub

TL;DR of What's Changed Since 2.9.0

dataform.json -> workflow_settings.yaml

dataform.json is being deprecated in favor of workflow_settings.yaml. This means that:

• Workflow settings are now strictly typed, in Protobuf format.
• The Dataform Core version can be specified directly in the workflow_settings.yaml file. Note: to have more than just @dataform/core as a dependency, a package.json must still be used.

Example conversion of workflow_settings.yaml:

defaultProject: dataform-demos
defaultLocation: us
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
version: 3.0.0-beta.0

The above is equivalent to the dataform.json file:

{
  "warehouse": "bigquery",
  "defaultDatabase": "dataform-demos",
  "defaultLocation": "us",
  "defaultSchema": "dataform",
  "assertionSchema": "dataform_assertions"
}

Notebooks Actions and actions.yaml

Notebooks as Dataform actions are on their way - but not quite yet! They're part of the compiled graph, and soon they'll be executable.

A new way of configuring action configs through actions.yaml has been implemented to support this.

An example of loading a notebook in Dataform can be seen at https://github.com/dataform-co/dataform/tree/main/examples/extreme_weather_programming.

Stateless Package Installation by @dataform/cli

Package installation by @dataform/cli is now stateless! The CLI will install NPM packages during compilation if version is defined in the workflow_settings.yaml file.

This means no node_modules folder has to be seen in the project, and Dataform users no longer need to be familiar with NPM.

Compilation Output is Now Warehouse Agnostic

Previously the output of compilation results from @dataform/core would insert warehouse specific SQL into the compiled graph. Where possible, this has been removed - transferring the responsibility of inserting warehouse specific SQL into whichever execution engine is running Dataform.

Additionally, support for non-BigQuery warehouses has been dropped. We're in discussions with Datashell for them to provide a warehouse-agnostic CLI execution engine based off of Dataform compiled graphs. In the meantime however, if you need support for a non-BigQuery warehouse, please continue using the latest version starting with 2.x.x!

What's Changed

• Remove non bigquery warehouses by @Ekrekr in #1550
• Remove dynamic warehouse inference, and more clearly differentiate core and api's SQL adapters by @Ekrekr in #1554
• Consolidate cli package by @Ekrekr in #1557
• Merge main to v3 by @Ekrekr in #1563
• Deprecate support for old type .sql files by @Ekrekr in #1564
• Add initial action config proto definitions by @Ekrekr in #1576
• Merge main to main_v3 by @Ekrekr in #1579
• Remove gen index in favor of main by @Ekrekr in #1571
• Read workflow_settings.yaml by @Ekrekr in #1580
• Validate workflow settings fields by @Ekrekr in #1581
• Remove deprecated run cache and gcloud project ID fields by @Ekrekr in #1584
• Move concurrency and action retries from project config proto to the CLI by @Ekrekr in #1585
• Merge main to main_v3 3 by @Ekrekr in #1590
• Populate warehouse as BigQuery by default by @Ekrekr in #1591
• Update v3 to use the apache 2.0 license by @Ekrekr in #1594
• Stop using ProtobufJS's verify method (it doesn't do much) by @Ekrekr in #1596
• Clean up action proto constructors in prep for proto config definitions by @Ekrekr in #1597
• Add basic actions.yaml and notebook support by @Ekrekr in #1595
• Move custom variables test to the new testing structure by @Ekrekr in #1601
• Add auto assertion database override to core test by @Ekrekr in #1602
• Move core/tasks.ts to CLI, and minor cleanup by @Ekrekr in #1598
• Add more test coverage of variables and project config overrides for Dataform Core by @Ekrekr in #1603
• Add licenses as prefixes to output bundles by @Ekrekr in #1605
• Throw a more interpretable error when a =v3 Core version by @Ekrekr in #1607
• Action builders and path utils fixes by @Ekrekr in #1606
• Prevent non bq warehouse from being set in workflow settings by @Ekrekr in #1609
• Strip notebook cell outputs by @Ekrekr in #1613
• Add version to workflow settings by @Ekrekr in #1610
• Move common target constructor methods to the action builder by @Ekrekr in #1626
• Add basic support for actions.yaml reading SQL files as operations by @Ekrekr in #1628
• Add support for declarations in action config files by @Ekrekr in #1630
• Add support for Tables to action config files by @Ekrekr in #1631
• Add action config support for the remaining SQL based action types by @Ekrekr in #1633
• Improve proto validation errors where possible by @Ekrekr in #1635
• Make filenames defined in action config files be treated as relative to the action config file by @Ekrekr in #1636
• Better errors for invalid declarations, add more test coverage by @Ekrekr in #1637
• Add an example of a workflow containing notebooks/SQL scripts, with tests by @Ekrekr in #1644
• Make the CLI use and default to workflow_settings.yaml by @Ekrekr in #1648
• Update Dataform CLI npm installs to best-effort by @Ekrekr in #1649
• Merge to v3 4 by @Ekrekr in #1654
• Remove dead resolve code by @Ekrekr in #1652
• Make main_test more concise by making filenames optional by @Ekrekr in #1651
• Remove more dead code, including navigator column descriptors and tools by @Ekrekr in #1655
• Remove warehouse option from the CLI, more tidying by @Ekrekr in #1653
• Make v3 JS API act the same as v2 by @Ekrekr in #1657
• Remove inline tables functionality by @Ekrekr in #1658
• Update proto to use a separate compiled graph, with UX review changes by @Ekrekr in #1656
• Core path tidying by @Ekrekr in #1659
• Export configs proto, bump version by @Ekrekr in #1661
• Migrate core sqlx syntax and assertions context functions tests to _test style by @Ekrekr in #1665
• Better cli errors during compilation by @Ekrekr in #1669
• Replace cli spec tests with _test format by @Ekrekr in #1670
• Improve generated workflow_settings.yaml field order by @Ekrekr in #1671
• Fix config targets by @Ekrekr in #1673
• Remove JS context from SQL files by @Ekrekr in #1672
• Lazy / stateless installation by @Ekrekr in #1675
• Don't check we are on main branch for pre-releases by @lewish in #1664
• Merge main to main_v3 5 by @Ekrekr in #1678
• Bump ip from 1.1.5 to 1.1.9 by @dependabot in #1679
• Merge v3 into main by @Ekrekr in #1680
• Bump v3 to beta by @Ekrekr in #1682

Full Changelog: 2.9.0...3.0.0-beta.0