dbt-labs/dbt-core v0.19.0 on GitHub

dbt 0.19.0 - Kiyoshi Kuromiya (January 27, 2021)

Links:

💬 Discussion: v0.19.0
📖 New and changed documentation

Breaking changes

The format for sources.json, run-results.json, manifest.json, and catalog.json has changed:
- Each now has a common metadata dictionary (#2761, #2778). The contents include: schema and dbt versions (#2670, #2767); invocation_id (#2763, #2784); custom environment variables prefixed with DBT_ENV_CUSTOM_ENV_ (#2764, #2785); cli and rpc arguments in the run_results.json (#2510, #2813).
- Remove injected_sql from manifest nodes, use compiled_sql instead (#2762, #2834)
Defer if and only if upstream reference does not exist in current environment namespace (#2909, #2946)
Rationalize run result status reporting and clean up artifact schema (#2493, #2943)
Add adapter specific query execution info to run results and source freshness results artifacts. Statement call blocks return response instead of status, and the adapter method get_status is now get_response (#2747, #2961)

Deprecations

Removed support for config-version: 1 of dbt_project.yml, which was deprecated in v0.17.0. Use config-version: 2 in all projects and installed packages.

Features

Core

dbt will compare configurations using the un-rendered form of the config block in dbt_project.yml. (#2713, #2735)
Added state and defer arguments to the RPC client, matching the CLI (#2678, #2736)
Added ability to snapshot hard-deleted records (opt-in with invalidate_hard_deletes config option). (#249, #2749)
Added revival for snapshotting hard-deleted records. (#2819, #2821)
Improved error messages for YAML selectors (#2700, #2781)
Added --defer flag for dbt test as well (#2701, #2954)
Added native python re module for regex in jinja templates (#1755, #1755)
Store resolved node names in manifest (#2647, #2837)
Save selectors dictionary to manifest, allow descriptions (#2693, #2866)
Normalize cli-style-strings in manifest selectors dictionary (#2879, #2895)

Docs website

Add select/deselect option in DAG view dropups. (docs#98, docs#138)
Fixed issue where sources with tags were not showing up in graph viz (docs#93, docs#139)
Use compiled_sql instead of injected_sql for "Compiled" (docs#146, docs#148)
Fixed issue where data tests with tags were not showing up in graph viz (docs#147, docs#157)

BigQuery

Added dbt_invocation_id for each BigQuery job to enable performance analysis (#2808, #2809)
Added support for BigQuery connections using refresh tokens (#2344, #2805)
Added macro get_partitions_metadata(table) to return partition metadata for BigQuery partitioned tables (#2552, #2596)
Hourly, monthly and yearly partitions available in BigQuery (#2476, #2903)
Allow BigQuery to default to the environment's default project (#2828, #2908)

Fixes

Respect --project-dir in dbt clean command (#2840, #2841)
Fix Redshift adapter get_columns_in_relation macro to push schema filter down to the svv_external_columns view (#2854, #2854)
Increased the supported relation name length in postgres from 29 to 51 (#2850)
dbt list command always return 0 as exit code (#2886, #2892)
Set default materialized for test node configs to test (#2806, #2902)
Allow docs blocks in exposure descriptions (#2913, #2920)
Use original file path instead of absolute path as checksum for big seeds (#2927, #2939)
Fix KeyError if deferring to a manifest with a since-deleted source, ephemeral model, or test (#2875, #2958)
Fix regression with defining exposures and other resources with the same name (#2969, #3009)
Remove ellipses printed while parsing (#2971, #2996)

Under the hood

Save manifest.json at the same time we save the run_results.json at the end of a run (#2765, #2799)
Added strategy-specific validation to improve the relevancy of compilation errors for the timestamp and check snapshot strategies. ((#2787, #2791)
Changed rpc test timeouts to avoid locally run test failures (#2803,#2804)
Added a debug_query on the base adapter that will allow plugin authors to create custom debug queries (#2751,#2871)
Add unixodbc-dev package to testing docker image (#2859)
Add event tracking for project parser/load times (#2823,#2893)
Bump cryptography version to >= 3.2 and bump snowflake connector to 2.3.6 (#2896, #2922)
Widen supported Google Cloud libraries dependencies (#2794, #2877).
Bump hologram version to 0.0.11. Add scripts/dtr.py (#2888,#2889)
Bump hologram version to 0.0.12. Add testing support for python3.9 (#2822,#2960)
Bump the version requirements for boto3 in dbt-redshift to the upper limit 1.16 to match dbt-redshift and the snowflake-python-connector as of version 2.3.6. (#2931, (#2963))
Rewrite macro for snapshot_merge_sql to make compatible with other SQL dialects (#3003)
Rewrite logic in snapshot_check_strategy() to make compatible with other SQL dialects (#3000, #3001)
Remove version restrictions on botocore (#3006)
Include exposures in start-of-invocation stdout summary: Found ... (#3007, #3008)
Cleanup docker resources, use single docker/Dockerfile for publishing dbt as a docker image (dbt-release#3, #3019)

Contributors

Thank you all for your contributions to dbt! 🎉

Kiyoshi Kuromiya (1943–2000)

Thanks to @amychen1776 for writing this biography of Kiyoshi Kuromiya―a famous Philadelphian, and the namesake of this dbt release.

To boil down all of Kiyoshi Kuromiya's accomplishments to a few sentences would be an act not unlike trying to boil down the Pacific Ocean to mere salt. There will be many remnants missing but you will at least have a glimpse at how he has shaped the world.

Born at a Japanese American internment camp in Wyoming, Kiyoshi's life began in conflict. He would go on to live a life centered around being a voice for the marginalized. His intersectional activism span from racial injustice to LGBTQA rights as well as protesting war. Though not born and raised in West Philadelphia, Kiyoshi became a Philadelphian in 1961 when he started at the University of Pennsylvania. There, he began to explore his multi-faceted identities as well as progress his activism.

One of the ways he attracted attention to his causes was theatrics. He once staged an event on the library steps at Penn, threatening to burn a dog alive. When thousands turn to protest, he flipped the narrative by delivering the message: "Congratulations on your anti-napalm protest. You saved the life of a dog. Now, how about saving the lives of tens of thousands of people in Vietnam?"

Kiyoshi was also one of the few Asian Americans who openly supported the Civil Rights Movement. He was a close confidant to Dr. Martin Luther King Jr, to the point of taking care of his children in the days after King's assassination. As an openly gay delegate to the Black Panther Convention, he played a part in the organization's recognization of the connections between racial injustice and the Gay Rights Movement. He was also a founding member of the Gay Liberation Front - Philadelphia group.

Outside of his involvement in the civil rights movement, Kiyoshi was widely known for his work for the AIDS community. He was actively involved in several AIDS organizations in Philadelphia, including being the editor of the ACT-UP Standard of Care, the first such publication for the care of people living with HIV. He was a treatment activist and clinical trials participant, fighting for research that involved the community in its design. His advocacy for people of color, drug users, women, and incarcerated individuals would ensure hundreds of folks had access to up-to-date treatment information and resources.

He passed away in 2000 due to AIDS-related complications but his legacy to social justice will forever live on. ✊🏿

dbt-labs/dbt-core v0.19.0 dbt 0.19.0 on GitHub