dbt-labs/dbt-core v0.20.0 on GitHub

dbt 0.20.0 - Margaret Mead (July 12, 2021)

💬 Discussion: v0.20.0
📖 New and changed documentation

Breaking changes

Add Jinja tag for generic test definitions. Replacement for macros prefixed test_ (#1173, #3261)
Update schema/generic tests to expect a set of rows instead of a single numeric value, and to use test materialization when executing. (#3192, #3286)
Plugin maintainers: For adapters that inherit from other adapters (e.g. dbt-postgres → dbt-redshift), adapter.dispatch() will now include parent macro implementations as viable candidates (#2923, #3296)
Fix adapter.dispatch macro resolution when statically extracting macros. Introduce new project-level dispatch config. The packages argument to dispatch no longer supports macro calls; there is backwards compatibility for existing packages. The argument will no longer be supported in a future release, instead provide the macro_namespace argument. (#3362, #3363, #3383, #3403)
Bump schema versions for run results and manifest artifacts (#3422, #3421)

Features

Tests

Add a new materialization for tests, update data tests to use test materialization when executing. (#3154, #3181)
Support disabling schema tests, and configuring tests from dbt_project.yml (#3252,
#3253, #3257)
Less greedy test selection: expand to select unselected tests if and only if all parents are selected (#2891, #3235)
Store test failures in the database (#517, #903, #2593, #3316)
Add new test configs: where, limit, warn_if, error_if, fail_calc (#3258, #3321, #3336)

Parsing

Add the opt-in --use-experimental-parser flag (#3307, #3374)
Move partial parsing to end of parsing and implement new partial parsing method. (#3217, #3364)
Save doc file node references and use in partial parsing. (#3425, #3432)

Packages

Support commit hashes in dbt deps package revision (#3268, #3270)
Add optional subdirectory key to install dbt packages that are not hosted at the root of a Git repository (#275, #3267)

Core

Add tags and meta properties to the exposure resource schema. (#3404, #3405)
Support optional updated_at config parameter with check strategy snapshots. If not supplied, will use current timestamp (default). (#1844, #3376)

Plugins

Add optional configs for require_partition_filter and partition_expiration_days in BigQuery (#1843, #2928)
Fix for EOL SQL comments prevent entire line execution (#2731, #2974)
Add optional merge_update_columns config to specify columns to update for merge statements in BigQuery and Snowflake (#1862, #3100)
Use query comment JSON as job labels for BigQuery adapter when query-comment.job-label is set to true (#2483), (#3145)
Set application_name for Postgres connections (#885, #3182)
Add native support for Postgres index creation (#804, 3106)
Prevent locks in Redshift during full refresh in incremental materialization. (#2426, #2998)

Docs

Reversed the rendering direction of relationship tests so that the test renders in the model it is defined in (docs#181, docs#183)
Support dots in model names: display them in the graphs (docs#184, docs#185)
Render meta tags for sources (docs#192, docs#193)
Display tags on exposures (docs#194, docs#195)
Update dbt logo and links (docs#197)

Fixes

Fix exit code from dbt debug not returning a failure when one of the tests fail (#3017, #3018)
Auto-generated CTEs in tests and ephemeral models have lowercase names to comply with dbt coding conventions (#3027, #3028)
Fix incorrect error message when a selector does not match any node #3036)
Fix variable _dbt_max_partition declaration and initialization for BigQuery incremental models (#2940, #2976)
Moving from 'master' to 'HEAD' default branch in git (#3057, #3104, #3117))
Requirement on dataclasses is relaxed to be between >=0.6,<0.9 allowing dbt to cohabit with other libraries which required higher versions. (#3150, #3151)
Add feature to add _n alias to same column names in SQL query (#3147, #3158)
Raise a proper error message if dbt parses a macro twice due to macro duplication or misconfiguration. (#2449, #3165)
Fix exposures missing in graph context variable. (#3241, #3243)
Ensure that schema test macros are properly processed (#3229, #3272)
Use absolute path for profiles directory instead of a path relative to the project directory. Note: If a user supplies a relative path to the profiles directory, the value of args.profiles_dir will still be absolute. (#3133, #3176)
Fix FQN selector unable to find models whose name contains dots (#3246, #3247)
Handle quoted values within test configs, such as where (#3458, #3459)
Avoid slowdown in column-level persist_docs on Snowflake, while preserving the error-avoidance from #3149 (#3541, #3543)
Partial parsing: handle already deleted nodes when schema block also deleted (#3516, #3522)
Fix compiled sql for ephemeral models (#3317, #3318)
Now generating run_results.json even when no nodes are selected (#3313, #3315)
Add missing packaging dependency (#3312, #3339)
Fix references to macros with package names when rendering schema tests (#3324, #3345)
Stop clobbering default keyword arguments for jinja test definitions (#3329, #3340)
Fix unique_id generation for generic tests so tests with the same FQN but different configuration will run. (#3254, #3335)
Update the snowflake adapter to only comment on a column if it exists when using the persist_docs config (#3039, #3149)
Add a better error messages for undefined macros and when there are less packages installed than specified in packages.yml. (#2999)
Separate compiled_path from build_path, and print the former alongside node error messages (#1985, #3327)
Fix exception caused when running dbt debug with BigQuery connections (#3314, #3351)
Raise better error if snapshot is missing required configurations (#3381, #3385)
Fix dbt run errors caused from receiving non-JSON responses from Snowflake with Oauth (#3350)
Fix deserialization of Manifest lock attribute (#3435, #3445)
Fix dbt run errors caused from receiving non-JSON responses from Snowflake with Oauth (#3350
Fix infinite recursion when parsing schema tests due to loops in macro calls (#3444, #3454)

Under the hood

Add dependabot configuration for alerting maintainers about keeping dependencies up to date and secure. (#3061, #3062)
Update script to collect and write json schema for dbt artifacts (#2870, #3065)
Relax Google Cloud dependency pins to major versions. (#3155, #3155)
Bump snowflake-connector-python and releated dependencies, support Python 3.9 (#2985, #3148)
General development environment clean up and improve experience running tests locally (#3194, #3204, #3228)
Switch from externally storing parsing state in ParseResult object to using Manifest (#3163, #3219)
Switch from loading project files in separate parsers to loading in one place(#3244, #3248)
Added logic for registry requests to raise a timeout error after a response hangs out for 30 seconds and 5 attempts have been made to reach the endpoint (#3177, #3275)
Added support for invoking the list task via the RPC server (#3311, #3384)
Added unique_id and original_file_path as keys to json responses from the list task (#3356, #3384)
Use shutil.which so Windows can pick up git.bat as a git executable (#3035, #3134)
Add ssh-client and update git version (using buster backports) in Docker image (#3337, #3338)
Update test sub-query alias (#3398, #3414)
Add deprecation warning for using packages argument with adapter.dispatch (#3419, #3420)
Swap experimental parser implementation to use Rust #3497
Dispatch the core SQL statement of the new test materialization, to benefit adapter maintainers (#3465, #3461)
Minimal validation of yaml dictionaries prior to partial parsing (#3246, #3460)
Add partial parsing tests and improve partial parsing handling of macros (#3449, #3505)
Update project loading event data to include experimental parser information. (#3438, #3495)
Add tracking for experimental parser accuracy (3503, 3553)

Contributors

Thank you all for your contributions to dbt! 🎉

Margaret Mead (1901–1978)

Thanks to @amyd123 for writing this biography of Margaret Mead—a famous Philadelphian, and the namesake of this dbt release.

The most striking thing about Margaret Mead is that although she died more than 40 years ago, her interests and advocacy are as relevant as ever - she was a campaigner for reproductive rights, fought against discrmination against LGBT scientists, and for a greater understanding of how racism affects individuals in American society. Her interests as a researcher focused on how we can help adolescent girls navigate the complexities of the modern world - in the 1920s!

Born in Philadelphia to a Quaker family in 1901, Mead became one of the nation’s most public intellectuals, popularizing the field of anthropology, and tirelessly campaigning for various social issues until her death in 1978. Mead began her field work in the 1920s, leading to her first book Coming of Age in Samoa. This began more than two decades of anthropological field work focusing on the lives of families across the preliterate cultures of the Pacific.

One of the major themes of her scholarly work was that gender roles are largely determined by culture, not biology, and as a corollary, that we have the ability as a society to build a culture that allows all people to realize their full potential. Although Mead’s work focused on the Pacific, her interest was squarely on how to use the knowledge gained from anthropology to create a better society in the US. She brought these ideas out of academia and made them more accessible to the public by writing for magazines like Life, Time, Parents Magazine, and Redbook.

In more recent years, Mead’s work has come under fire for being a product of some of the colonialist biases of her time, and based on fieldwork considered less rigorous than current standards dictate. However, Mead’s fundamental contribution remains -- in the 1920s, when Mead began her career, genetic determinism was the scientific consensus in the West. Mead was one of the first researchers to illustrate that many of our beliefs and actions are shaped by culture and our environment, and to use that understanding as a way to challenge ourselves to build the society we want to live in.

dbt-labs/dbt-core v0.20.0 dbt 0.20.0 on GitHub