dbt 0.16.0 (March 23, 2020)

This release contains many new features, bug fixes, and performance improvements.

Links:
🔍 Upgrading guide
💬 Discussion: v0.16.0 is released
⏩ Discussion: BigQuery incremental model updates
📈 Discussion: Benchmarking BigQuery incremental strategies

Breaking changes

• Agate type inference is no longer locale-specific. Only a small number of date/datetime formats are supported. If a seed has a specified column type, agate will not perform any type inference (it will instead be cast from a string). (#999, #1639, #1920)
• BigQuery range bucket partitioning must now be configured with the dictionary-style partitioning syntax (#2140) (docs)
• generate_schema_name macros that accept a single argument are no longer supported (#2143)
• Files with a .yml extension found in the data/, macros/, analysis/, tests/, and snapshots/ directories will now be parsed as schema.yml specifications (#2160)
• The syntax of the get_catalog macro has changed (#2037)
• When overriding the snowflake__list_schemas macro, you must now run a result with a column named 'name' instead of the first column (#2171)
• dbt no longer supports building models in Snowflake databases with greater than 10,000 schemas (#2171)
• Arguments to source tests are not parsed in the config-rendering context, and are passed as their literal unparsed values to macros (#2150)
• Update the debug log format (#2099)
• Removed docrefs from output (#2096)

Features

Core

• Compilation contexts have updated to be more consistent and well-defined (#1053, #1981, #1255, #2085) (docs)
• Add support for generate_database_name macro (#1695, #2143) (docs)
• Add meta, docs, and tags, and column quoting options to schema.yml files (docs)
  • Add a meta key to most schema.yml specifications (#1362, #2015)
  • Add tags for sources (like model tags) and columns (tags apply to tests of that column) (#1906, #1586, #2039)
  • Add column-level quoting control for tests (#2106, #2047)
  • Add a "docs" field to models, with a "show" subfield (#1671, #2107)
• Add support for documentation of macros, analyses, snapshots and seeds (docs)
  • Add documentation for macros/analyses (#1041, #2068)
  • Add arguments field to macro schema.yml specifications (#2081, #2083, #2096)
  • Add documentation for snapshots and seeds (#1974, #2051)
• Improve the speed of catalog generation by performing multiple smaller queries instead of one huge query (#2009, #2037)
• Add batching to anonymous usage statistics requests to improve performance (#2008, #2089)
• Add search for docs to include 'data' and 'snapshots' folders, in addition to 'models' (#1832, #2058)
• Add search for docs to include macro-paths and analysis-paths (in addition to source-paths, data-paths, and snapshot-paths) (#2155, #2160)
• Add Column.is_number/Column.is_float methods (#1969, #2046)
• Add detection and error handling for duplicated macro names (#1891, #2045)
• Add support for --select on dbt seed (#1711, #2042) (docs)
• Addtoyaml and fromyaml methods to the base context (#1911, #2036) (docs)
• Add database_schemas to the on-run-end context (#1924, #2031) (docs)
• Add the concept of builtins to the dbt context, make it possible to override functions like ref (#1603, #2028) (docs)
• Add clickable docs URL link in CLI output (#2027, #2131)
• Add parsing model hooks and collect ref statements (#1957, #2025)
• Add the macros every node uses to its depends_on.macros list (#2082, #2103)

Docs website

• Add handling non-array accepted_values test arguments (dbt-docs#70)
• Add support for filtering by resource type (dbt-docs#77)
• Add analyses, macros, and custom data tests (dbt-docs#72, dbt-docs#77, dbt-docs#69)
• Add support for hiding models from the docs (these nodes still render in the DAG view as "hidden") (dbt-docs#71)
• Add meta fields as "details" in node views (dbt-docs#73)
• Add lower-casing for Snowflake columns specified in all-caps (dbt-docs#74)
• Upgrade site dependencies

Postgres/Redshift

• Add role parameter in Postgres target configuration (#1955, #2137) (docs)
• Add sslmode parameter for Postgres target configuration (#2152, #2154) (docs)

Snowflake

• Remove the requirement to have a passphrase when using Snowflake key pair authentication (#1805, #2164) (docs)

BigQuery

• Support a cost-effective approach for incremental models on BigQuery using scripting (#1034, #2140) (docs)
• Add insert_overwrite strategy for BigQuery incremental models (#2153) (docs)
• Add a dbt-{dbt_version} user agent field to the bigquery connector (#2121, #2146)

Fixes

• Fix the help output for dbt docs and dbt source to not include misleading flags (#2038, #2105)
• Allow dbt debug from subdirectories (#2086, #2094)
• Fix the --no-compile flag to dbt docs generate not crash dbt (#2090, #2093)
• Fix issue running dbt debug with an empty dbt_project.yml file (#2116, #2120)
• Ovewrwrite source config fields that should clobber, rather than deep merging them (#2049, #2062)
• Fix a bug in macro search where built-in macros could not be overridden for dbt run-operation (#2032, #2035)
• dbt now detects dependencies with the same name as the current project as an error instead of silently clobbering each other (#2029, #2030)
• Exclude tests of disabled models in compile statistics (#1804, #2026)
• Do not show ephemeral models as being cancelled during ctrl+c (#1993, #2024)
• Improve errors on plugin import failure (#2006, #2022)
• Fix the behavior of the project-dir argument when running dbt debug (#1733, #1989)
• When dbt encounters databases, schemas, or tables with names that look like numbers, treat them as strings (#2206, #2208)
• Fix issue where dbt did not give an error in the presence of duplicate doc names (#2054, #2080)
• Include vars provided to the cli method when running the actual method (#2092, #2104)
• Improved error messages with malformed packages.yml (#2017, #2078)
• Fix an issue where dbt rendered source test args, fix issue where dbt ran an extra compile pass over the wrapped SQL. (#2114, #2150)
• [Regression] If database quoting is enabled, do not attempt to create schemas that already exist (#2186, #2187)

Under the hood

• Set more upper bounds for jinja2,requests, and idna dependencies, upgrade snowflake-connector-python (#2147, #2151)
• Pin cffi to <1.14 to avoid a version conflict with snowflake-connector-python (#2180, #2181)
• Use show terse schemas in database (chosen based on data collected by Michael Weinberg) instead of select ... from information_schema.schemata when collecting the list of schemas in a database (#2166, #2171)
• Parallelize filling the cache and listing schemas in each database during startup (#2127, #2157)
• Increased the lower bound for google-cloud-bigquery (#2213, #2214)
• Improve the CI process for externally-contributed PRs (#2033, #2097)
• Many mypy/typing fixes (#2010)

Contributors:

• @aaronsteers (#2131)
• @alanmcruickshank (#2028)
• @bubbomb (#2080)
• @dholleran-lendico (#2154)
• @franloza (#1989)
• @heisencoder (#2099)
• @mhmcdonald (#2164)
• @nchammas (#2120)
• @NiallRees (#2026)
• @shooka (#2137)
• @sonac (#2078)
• @tayloramurphy (#2015)

Barbara Gittings (1932-2007)

Thanks to @amychen1776 for providing this biography of Barbara Gittings: a famous Philadelphian, and the namesake of this dbt release.

The gritty little city of Philadelphia has always played key roles in the rebellion for a better future, from being the birthplace of America all the way to the modern LGBT Movement. The basic freedom that queer Americans enjoy today can be directly linked to the fight that Barbara Gittings waged for us here in Philadelphia.

Working with Frank Kemeny, Gittings organized the historical Annual Reminders at Independence Hall and the Liberty Bell on the 4th of July. These pickets were designed to remind the general public that LGBT people did not have the same basic civil rights as their heterosexual counterparts. The bravery of these participants should not be taken lightly; homosexuality was illegal in most of the US. These events preceded the Stonewall Riots, acting as a key catalyst for the LGBT civil rights movement.

With Kemeny, one of Gittings’ biggest battles was with the American Psychiatric Association. In 1973, they won, and homosexuality was removed from the Diagnostic and Statistical Manual (an authoritative guide to the diagnosis of mental disorders) as a mental disorder. The celebration? Gittings took photographs with her holding up the Philadelphia news headlines: “Twenty Million Homosexuals Gain Instant Cure”.

In an interview, Barbara Gittings summed up her activism in a quote: “It’s hard work -- but it’s vital, and it’s gratifying and it’s often fun!”.