dbt-labs/dbt-core v1.8.0b1

dbt-core v1.8.0b1

on GitHub

dbt-core 1.8.0-b1 - February 28, 2024

Breaking Changes

• Remove adapter.get_compiler interface (#9148)
• Move AdapterLogger to adapters folder (#9151)
• Rm --dry-run flag from 'dbt deps --add-package', in favor of just 'dbt deps --lock' (#9100)
• move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions (#9150)
• Remove dbt-tests-adapter and dbt-postgres packages from dbt-core (#9455)

Features

• Initial implementation of unit testing (#8287)
• Unit test manifest artifacts and selection (#8295)
• Support config with tags & meta for unit tests (#8294)
• Allow adapters to include package logs in dbt standard logging (#7859)
• Enable inline csv fixtures in unit tests (#8626)
• Add drop_schema_named macro (#8025)
• migrate utils to common and adapters folders (#8924)
• Move Agate helper client into common (#8926)
• remove usage of dbt.config.PartialProject from dbt/adapters (#8928)
• Add exports to SavedQuery spec (#8892)
• Support unit testing incremental models (#8422)
• Add support of csv file fixtures to unit testing (#8290)
• Remove legacy logger (#8027)
• Unit tests support --defer and state:modified (#8517)
• Support setting export configs hierarchically via saved query and project configs (#8956)
• Support source inputs in unit tests (#8507)
• Use daff to render diff displayed in stdout when unit test fails (#8558)
• Move unit testing to test command (#8979)
• Support --empty flag for schema-only dry runs (#8971)
• Support unit tests in non-root packages (#8285)
• Convert the tests config to data_tests in both dbt_project.yml and schema files. in schema files. (#8699)
• Make fixture files full-fledged parts of the manifest and enable partial parsing (#9067)
• Adds support for parsing conversion metric related properties for the semantic layer. (#9203)
• In build command run unit tests before models (#9128)
• Move flags from UserConfig in profiles.yml to flags in dbt_project.yml (#9183)
• Added hook support for dbt source freshness (#5609)
• Align with order of unit test output when actual differs from expected (#9370)
• Added support for external nodes in unit test nodes (#8944)
• Enable unit testing versioned models (#9344)
• Enable list command for unit tests (#8508)
• Integration Test Optimizations (#9498)
• Accelerate integration tests with caching. (#9498)
• Cache environment variables (#9489)
• Support meta at the config level for Metric nodes (#9441)
• Add cache to SavedQuery config (#9540)

Fixes

• For packages installed with tarball method, fetch metadata to resolve nested dependencies (#8621)
• Fix partial parsing not working for semantic model change (#8859)
• Handle unknown type_code for model contracts (#8877, #8353)
• Rework get_catalog implementation to retain previous adapter interface semantics (#8846)
• Add back contract enforcement for temporary tables on postgres (#8857)
• Add version to fqn when version==0 (#8836)
• Fix cased comparison in catalog-retrieval function. (#8939)
• Catalog queries now assign the correct type to materialized views (#8864)
• Fix compilation exception running empty seed file and support new Integer agate data_type (#8895)
• Make relation filtering None-tolerant for maximal flexibility across adapters. (#8974)
• Update run_results.json from previous versions of dbt to support deferral and rerun from failure (#9010)
• Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution (#9016)
• Fix git repository with subdirectory for Deps (#9000)
• Use seed file from disk for unit testing if rows not specified in YAML config (#8652)
• Fix formatting of tarball information in packages-lock.yml (#9062)
• deps: Lock git packages to commit SHA during resolution (#9050)
• deps: Use PackageRenderer to read package-lock.json (#9127)
• Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources (#8991)
• Get sources working again in dbt docs generate (#9119)
• Fix parsing f-strings in python models (#6976)
• Preserve the value of vars and the --full-refresh flags when using retry. (#9112)
• Support reasonably long unit test names (#9015)
• Fix back-compat parsing for model-level 'tests', source table-level 'tests', and 'tests' defined on model versions (#9411)
• Fix retry command run from CLI (#9444)
• Fix seed and source selection in dbt docs generate (#9161)
• Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate (#9456)
• Fix node type plurals in FoundStats log message (#9464)
• Run manifest upgrade preprocessing on any older manifest version, including v11 (#9487)
• Update 'compiled_code' context member logic to route based on command ('clone' or not). Reimplement 'sql' context member as wrapper of 'compiled_code'. (#9502)
• Fix bug where Semantic Layer filter strings are parsed into lists. (#9507)
• Initialize invocation context before test fixtures are built. (##9489)
• When patching versioned models, set constraints after config (#9364)
• only include unmodified semantic mdodels in state:modified selection (#9548)
• Set query headers when manifest is passed in to dbtRunner (#9546)
• Store node_info in node associated logging events (#9557)
• Fix Semantic Model Compare node relations (#9548)
• Clearer no-op logging in stubbed SavedQueryRunner (#9533)
• Fix node_info contextvar handling so incorrect node_info doesn't persist (#8866)
• Add target-path to retry (#8948)

Docs

• fix get_custom_database docstring (dbt-docs/#9003)

Under the Hood

• Added more type annotations. (#8537)
• Add unit testing functional tests (#8512)
• Remove usage of dbt.include.global_project in dbt/adapters (#8925)
• Add a no-op runner for Saved Qeury (#8893)
• remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8967)
• Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters (#8969)
• Move CatalogRelationTypes test case to the shared test suite to be reused by adapter maintainers (#8952)
• Treat SystemExit as an interrupt if raised during node execution. (#n/a)
• Removing unused 'documentable' (#8871)
• Remove use of dbt/core exceptions in dbt/adapter (#8920)
• Cache dbt plugin modules to improve integration test performance (#9029)
• Consolidate deferral methods & flags (#7965, #8715)
• Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock variance (#9057)
• Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific event types and protos (#8927, #8918)
• Clean up unused adaptor folders (#9123)
• Move column constraints into common/contracts, removing another dependency of adapters on core. (#9024)
• Move dbt.semver to dbt.common.semver and update references. (#9039)
• Move lowercase utils method to common (#9180)
• Remove usages of dbt.clients.jinja in dbt/adapters (#9205)
• Remove usage of dbt.contracts in dbt/adapters (#9208)
• Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters (#9214)
• Introduce RelationConfig Protocol, consolidate Relation.create_from (#9215)
• remove manifest from adapter.set_relations_cache signature (#9217)
• remove manifest from adapter catalog method signatures (#9218)
• Move BaseConfig, Metadata and various other contract classes from model_config to common/contracts/config (#8919)
• Add MacroResolverProtocol, remove lazy loading of manifest in adapter.execute_macro (#9244)
• pass query header context to MacroQueryStringSetter (#9249, #9250)
• add macro_context_generator on adapter (#9247)
• pass mp_context to adapter factory as argument instead of import (#9025)
• have dbt-postgres use RelationConfig protocol for materialized views' (#9292)
• move system.py to common as dbt-bigquery relies on it to call gcloud (#9293)
• Reorganizing event definitions to define core events in dbt/events rather than dbt/common (#9152)
• move exceptions used only in dbt/common to dbt/common/exceptions (#9332)
• Remove usage of dbt.adapters.factory in dbt/common (#9334)
• Accept valid_error_names in WarnErrorOptions constructor, remove global usage of event modules (#9337)
• Move result objects to dbt.artifacts (#9193)
• dbt Labs OSS standardization of docs and templates. (#9252)
• Add dbt-common as a dependency and remove dbt/common (#9357)
• move cache exceptions to dbt/adapters (#9362)
• Clean up macro contexts. (#9422)
• Add the @requires.manifest decorator to the retry command. (#9426)
• Move WritableManifest + Documentation to dbt/artifacts (#9378, #9379)
• Define Macro and Group resources in dbt/artifacts (#9381, #9382)
• Move SavedQuery data definition to dbt/artifacts (#9386)
• Migrate data parts of Metric node to dbt/artifacts (#9383)
• Move data portion of SemanticModel to dbt/artifacts (#9387)
• Move data parts of Exposure class to dbt/artifacts (#9380)
• Start using Mergeable from dbt-common (#9505)
• Move manifest nodes to artifacts (#9388)
• Move data parts of SourceDefinition class to dbt/artifacts (#9384)
• Remove uses of Replaceable class (#7802)
• Make dbt-core compatible with Python 3.12 (#9007)
• Restrict protobuf to major version 4. (#9566)
• Remove references to dbt.tracking and dbt.flags from dbt/artifacts (#9390)
• Implement primary key inference for model nodes (#9652)
• Define UnitTestDefinition resource in dbt/artifacts/resources (#9667)
• Use Manifest instead of WritableManifest in PreviousState and _get_deferred_manifest (#9567)

Dependencies

• Bump actions/checkout from 3 to 4 (#8781)
• Begin using DSI 0.4.x (#8892)
• Update typing-extensions version to >=4.4 (#9012)
• Bump ddtrace from 2.1.7 to 2.3.0 (#9132)
• Bump freezegun from 0.3.12 to 1.3.0 (#9197)
• Bump actions/setup-python from 4 to 5 (#9267)
• Bump actions/download-artifact from 3 to 4 (#9374)
• remove dbt/adapters and add dependency on dbt-adapters (#9430)
• Bump actions/cache from 3 to 4 (#9471)
• Bump peter-evans/create-pull-request from 5 to 6 (#9552)
• Cap dbt-semantic-interfaces version range to <0.6 (#9671)
• bump dbt-common to accept major version 1 (#9690)

Security

• Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 (#CVE-2024-22195)

Contributors

• @LeoTheGriff (#9003)
• @WilliamDee (#9203)
• @adamlopez (#8621)
• @aliceliu (#9652)
• @benmosher (#n/a)
• @colin-rorgers-dbt (#8919)
• @courtneyholcomb (#9507)
• @l1xnan (#9007)
• @mederka (#6976)
• @ofek1weiss (#5609)
• @peterallenwebb, (#9112)
• @tlento (#9012, #9671)
• @tonayya (#9252)