dbt-labs/dbt-core v0.17.0 on GitHub

dbt 0.17.0 - Octavius Catto (June 8th, 2020)

Links:

📖 Migration guide + docs

Breaking changes

Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version (#2300, #2312)
The graph object available in some dbt contexts now has an additional member sources (along side the existing nodes). Sources have been removed from nodes and added to sources instead (#2312)
The 'location' field has been removed from bigquery catalogs (#2382)
The list_relations_without_caching, drop_schema, and create_schema macros and methods now accept a single argument of a Relation object with no identifier field. (#2411)

Features

Core

Add --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. (#1649, #2224)
Support for appending query comments to SQL queries. (#2138, #2199)
Add a get-manifest API call. (#2168, #2232)
Support adapter-specific aliases (like project and dataset on BigQuery) in source definitions. (#2133, #2244)
Users can now use jinja as arguments to tests. Test arguments are rendered in the native context and injected into the test execution context directly. (#2149, #2220)
Users can supply paths as arguments to --models and --select, either explicitily by prefixing with path: or implicitly with no prefix. (#454, #2258)
dbt now builds the relation cache for "dbt compile" and "dbt ls" as well as "dbt run" (#1705, #2319)
Add a 'depends_on' attribute to the log record extra field (#2316, #2341)
Add a '--no-browser' argument to "dbt docs serve" so you can serve docs in an environment that only has a CLI browser which would otherwise deadlock dbt (#2004, #2364)
Sources (and therefore freshness tests) can be enabled and disabled via dbt_project.yml (#2283, #2312, #2357)
schema.yml files are now fully rendered in a context that is aware of vars declared in from dbt_project.yml files (#2269, #2357)
Sources from dependencies can be overridden in schema.yml files (#2287, #2357)
Add a filter named as_text to the native environment rendering code that allows users to mark a value as always being a string (#2384, #2395)
Add warning to nodes selector if nothing was matched (#2115, #2343)

Snowflake

Snowflake now uses "show terse objects" to build the relations cache instead of selecting from the information schema (#2174, #2322)
Snowflake now uses "describe table" to get the columns in a relation (#2260, #2324)
Relation comments supported for Snowflake tables and views. Column comments supported for tables. (#1722, #2321)

BigQuery

Suport column descriptions for BigQuery models (#2335, #2402)
Add BigQuery option maximum_bytes_billed to set an upper limit for query costs (#2346, #2427)

Postgres & Redshift

Added support for db_groups and autocreate flags in Redshift configurations. (#1995, #2262)
Implement persist_docs for both relation and comments on postgres and redshift, and extract them when getting the catalog. (#2333, #2378)

Docs

Do not render hidden models in the search bar (docs#89, docs#90)

Fixes

When a jinja value is undefined, give a helpful error instead of failing with cryptic "cannot pickle ParserMacroCapture" errors (#2110, #2184)
Added timeout to registry download call (#2195, #2228)
When a macro is called with invalid arguments, include the calling model in the output (#2073, #2238)
When a warn exception is not in a jinja do block, return an empty string instead of None (#2222, #2259)
Add dbt plugin versions to --version(#2272, #2279)
When a Redshift table is defined as "auto", don't provide diststyle (#2246, #2298)
Made file names lookups case-insensitve (.sql, .SQL, .yml, .YML) and if .yaml files are found, raise a warning indicating dbt will parse these files in future releases. (#1681, #2263)
Return error message when profile is empty in profiles.yml. (#2292, #2297)
Fix skipped node count in stdout at the end of a run (#2095, #2310)
Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. (#2188, #2325)
Fix "dbt deps" command so it respects the "--project-dir" arg if specified. (#2338, #2339)
On run_cli API calls that are passed --vars differing from the server's --vars, the RPC server rebuilds the manifest for that call. (#2265, #2363)
Remove the query job SQL from bigquery exceptions (#2383, #2393)
Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures (#2336, #2348)
No longer query the information_schema.schemata view on bigquery (#2320, #2382)
Preserve original subdirectory structure in compiled files. (#2173, #2349)
Add support for sql_header config in incremental models (#2136, #2200)
The ambiguous alias check now examines the node's database value as well as the schema/identifier (#2326, #2387)
Postgres array types can now be returned via run_query macro calls (#2337, #2376)
Add missing comma to dbt compile help text (#2388 #2389)
Fix for non-atomic snapshot staging table creation (#1884, #2390)
Fix for snapshot errors when strategy changes from check to timestamp between runs (#2350, #2391)
On snowflake, get_columns_in_relation now returns an empty list again if the relation does not exist, instead of raising an exception. (#2504, #2509)
Added filename, project, and the value that failed to render to the exception raised when rendering fails. (#2499, #2501)
When no columns are documented and persist_docs.columns is True, skip creating comments instead of failing with errors (#2439, #2440)
Fixed an argument issue with the create_schema macro on bigquery (#2445, #2448)
dbt now logs using the adapter plugin's ideas about how relations should be displayed (dbt-spark/#74, #2450)
The create_adapter_plugin.py script creates a version 2 dbt_project.yml file (#2451, #2455)
Fixed dbt crashing with an AttributeError on duplicate sources (#2463, #2464)
Fixed a number of issues with globally-scoped vars (#2473, #2472, #2469, #2477)
Fixed DBT Docker entrypoint (#2470, #2475)
Fixed a performance regression that occurred even when a user was not using the relevant feature (#2474, #2478)
Substantial performance improvements for parsing on large projects, especially projects with many docs definition. (#2480, #2481)
Expose Snowflake query id in case of an exception raised by connector (#2201, #2358)
When tracking is disabled due to errors, do not reset the invocation ID (#2398, #2400)
Fix for logic error in compilation errors for duplicate data test names (#2406, #2407)
Fix list_schemas macro failing for BigQuery (#2412, #2413)
When plugins are installed in the same folder as dbt core, report their versions. (#2410, #2418)
Fix for making schema tests work for community plugin dbt-sqlserver #2414
Fix a bug where quoted uppercase schemas on snowflake were not processed properly during cache building. (#2403, #2411)
Fix for extra spacing and parentheses when creating views in BigQuery (#2421, #2422)

Under the hood

Added more tests for source inheritance (#2264, #2291)
Update documentation website for 0.17.0 (#2284)
Lock protobufs to the last version that had fully functioning releases on all supported platforms (#2490, #2491)
Removed pytest-logbook dependency from dbt-core (#2505)
Better support for optional database fields in adapters (#2487 #2489)
Track distinct project hashes in anonymous usage metrics for package downloads (#2351, #2429)

Contributors:
Thank you all for your contributions to dbt! 🎉

Octavius Catto

Special thanks to @jtcohen6 for this biography of Octavius Catto, the namesake of this release.

Octavius Valentine Catto, born 1839 in Charleston, SC, lived much of his life at 812 South Street in Philadelphia. He graduated valedictorian and later taught at the Quaker-run Institute for Colored Youth, which still operates today as Cheyney University, the nation's oldest HBCU.

Catto spent much of the 1860's recruiting and supporting Black Pennsylvanians who fought in the Union Army, while he served in the Pennsylvania National Guard and in the fifth brigade of the United States Colored Troops. In the summer of 1863, when the Confederate Army invaded Pennsylvania, he answered the call by raising a company of 90 Black Philadelphian volunteers, many of them his former classmates. When they arrived in Harrisburg, the racism of the commanding officers precluded them from seeing combat with the emergency defense militia.

After the war, Catto grew to be an esteemed civic and community leader. He fought for the desegregation of Philadelphia's trolley car system; his tactics of civil disobedience were covered in the New York Times, ninety years before Rosa Parks' bus sit-in. His work culminated in a Pennsylvania law barring segregation on all transit systems, and the state's ratification of the Fifteenth Amendment in 1869, which granted all African American men the right to vote.

In addition to his teaching and organizing, he was also a top-notch cricket and baseball player. He co-founded the Philadelphia Pythians ball team, which he captained as a starting infielder. He led the team in the first recorded interracial baseball games, drawing massive crowds to the ball fields that were then in use by the Philadelphia (later Oakland) Athletics. (If you're a baseball nut like me, read more.)

Catto's prolific life came to a tragic end on October 10, 1871, when, at the age of 32, he left his house to vote for the very first time. A mob of white Philadelphians was attacking Black citizens on their way to and from the polls; one of them shot and murdered Catto, two doors from his home.

Catto is missing from many history textbooks, but several groups in his hometown have long worked to carry on his legacy. In 2017, at their urging, the City of Philadelphia unveiled "A Quest for Parity: The Octavius V. Catto Memorial," by sculptor Branly Cadet, on the southwest corner of City Hall.

Paired with the removal on June 3, 2020, of a nearby monument to racist and homophobic former mayor Frank Rizzo, the city is reckoning with its long history—one of systemic racism and everyday heroism, in Catto's time and in our own.

"De Tocqueville prophesied that if ever America underwent Revolution, it would be brought about by the presence of the black race, and that it would result from the inequality of their condition." — Octavius Catto (April 21, 1865, Independence Hall)

dbt-labs/dbt-core v0.17.0 dbt 0.17.0 on GitHub