databricks/dbt-databricks v1.9.0

on GitHub

What's Changed

Features

• Add config for generating unique tmp table names for enabling parralel merge (thanks @huangxingyi-git!) (854)
• Add support for serverless job clusters on python models (706)
• Add 'user_folder_for_python' behavior to switch writing python model notebooks to the user's folder (835)
• Merge capabilities are extended (739) to include the support for the following features (thanks @mi-volodin):
  • with schema evolution clause (requires Databricks Runtime 15.2 or above);
  • when not matched by source clause, only for delete action
  • matched, not matched and not matched by source condition clauses;
  • custom aliases for source and target tables can be specified and used in condition clauses;
  • matched and not matched steps can now be skipped;
• Allow for the use of custom constraints, using the custom constraint type with an expression as the constraint (thanks @roydobbe). (792)
• Add "use_info_schema_for_columns" behavior flag to turn on use of information_schema to get column info where possible. This may have more latency but will not truncate complex data types the way that 'describe' can. (808)
• Add support for table_format: iceberg. This uses UniForm under the hood to provide iceberg compatibility for tables or incrementals. (815)
• Add include_full_name_in_path config boolean for external locations. This writes tables to {location_root}/{catalog}/{schema}/{table} (823)
• Add a new workflow_job submission method for python, which creates a long-lived Databricks Workflow instead of a one-time run (thanks @kdazzle!) (762)
• Allow for additional options to be passed to the Databricks Job API when using other python submission methods. For example, enable email_notifications (thanks @kdazzle!) (762)
• Support microbatch incremental strategy using replace_where (825)

Fixes

• Replace array indexing with 'get' in split_part so as not to raise exception when indexing beyond bounds (839)
• Ensure columns that are added get backticked (859)

Under the Hood

• Significant refactoring and increased testing of python_submissions (830)
• Fix places where we were not properly closing cursors, and other test warnings (713)
• Drop support for Python 3.8 (713)
• Upgrade databricks-sql-connector dependency to 3.5.0 (833)
• Prepare for python typing deprecations (837)
• Fix behavior flag use in init of DatabricksAdapter (thanks @VersusFacit!) (836)
• Restrict pydantic to V1 per dbt Labs' request (843)
• Switching to Ruff for formatting and linting (847)
• Refactoring location of DLT polling code (849)
• Switching to Hatch and pyproject.toml for project config (853)

New Contributors

• @huangxingyi-git made their first contribution in #811
• @jelmerk made their first contribution in #820
• @VersusFacit made their first contribution in #836

Full Changelog: v1.8.6...v1.9.0