Highlights
Upgrading to 2.0 brings several important breaking changes:
- All bundled rules have been recoded, both from generic
L00Xformats into groups within similar codes (e.g. an aliasing group with codes of the formatAL0X), but also given names to allow much clearer referencing (e.g.aliasing.column). - Configuring rules now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime.
- A complete re-write of layout and whitespace handling rules (see layout), and with that a change in how layout is configured (see configuring layout) and the combination of some rules that were previously separate. One example of this is that the legacy rules
L001,L005,L006,L008,L023,L024,L039,L048&L071have been combined simply into LT01. - Dropping support for dbt versions before
1.1.
To help users upgrade to 2.0, we've put together a recommended process as part of our release notes.
Beyond the breaking changes, this release brings a load of additional changes:
- Introduces the the
sqlfluff formatCLI command (a lasqlfmtorblack) to auto-format sql files using a known set of fairly safe rules. - Databricks as a distinct new dialect (rather than as previously an alias for
sparksql). - Performance improvements in our parsing engine.
- Dialect improvements to almost all of them.
As a new major release, especially with significant rewrites of large portions of the codebase, we recommend using compatible release specifiers in your dependencies (i.e. sqlfluff~=2.0.0) so that you can automatically take advantage of any bugfix releases in the coming weeks. The alpha releases of 2.0.0 have been tested on a range of large projects, but we know that the range of use cases "in the wild" is very diverse. If you do experience issues, please post them on GitHub in the usual manner.
Finally thanks to everyone who has worked on this release, especially @konnectr, @ValentinCrr, @FabianScheidt, @dflem97, @timcosta, @AidanHarveyNelson, @joar, @jmpfar, @jared-rimmer, @vesatoivonen, @briankravec, @saintamh, @tdurieux, @baa-ableton & @WillAyd who made their first contributions during the development of 2.0.0. Thanks for your contributions, and especially your patience in the slightly slower release of your efforts into the wild. 🙏🎉
What’s Changed
- Revise templating and lexing of calls. #4506 @alanmcruickshank
- Struct Access Spacing #4512 @alanmcruickshank
- Array and Struct Spacing #4511 @alanmcruickshank
- Add a deprecation warning for removed config option. #4509 @alanmcruickshank
- Bigquery spacing (#4508) @alanmcruickshank
- Fix #4433 (more untaken positive indents) #4499 @alanmcruickshank
- Fix parse error on double parentheses #4504 @yoichi
- 2.0.0 Migration Guide #4498 @alanmcruickshank
- Handle missing aliases and align constraints better #4493 @alanmcruickshank
- TSQL: Add support For Clause #4501 @AidanHarveyNelson
- Allow Jinja rule to loop safely #4495 @alanmcruickshank
- Trigger CI tests for merge groups #4503 @alanmcruickshank
- Update Readme and Contributing #4502 @alanmcruickshank
- Update layout docs #4500 @alanmcruickshank
- Bug in operator precedence #4497 @alanmcruickshank
- BigQuery: correct query syntax for single column
UNPIVOTclauses #4494 @imrehg - Fix #4485 #4491 @alanmcruickshank
- Update reserved keywords in Athena language #4490 @ValentinCrr
- Clickhouse support all join types #4488 @konnectr
- Snowflake semi-structured spacing #4487 @alanmcruickshank
- Prep version 2.0.0a6 #4476 @github-actions
- Fix #4367 #4479 @alanmcruickshank
- Teradata: Improve COLLECT STATS parsing #4478 @dflem97
- Add a sqlfluff format CLI command #4473 @alanmcruickshank
- Recode and disable L031 -> AL07 #4471 @alanmcruickshank
- Named Config (part 2) #4470 @alanmcruickshank
- Rule config lookup improvements & config warnings #4465 @alanmcruickshank
- Recode L050 #4468 @alanmcruickshank
- Implicit indent fixes #4467 #4469 @alanmcruickshank
- ANSI: Add IfExistsGrammar to DropTrigger #4466 @WittierDinosaur
- Rules Reorg Mopup #4462 @alanmcruickshank
- Layout Rules Recode (part 2) #4456 @alanmcruickshank
- fix(athena): resolve errors parsing around maps, structs, and arrays #4391 @timcosta
- Layout Rules Recode (part 1) #4432 @alanmcruickshank
- TSQL: EXEC string literal #4458 @jpers36
- Teradata: Added SET QUERY_BAND statement #4459 @dflem97
- Teradata: Added TOP select clause modifier #4461 @dflem97
- Teradata: Addition of comparison operator extensions #4451 @dflem97
- Add extensions and plugin section to the README.md #4454 @jared-rimmer
- Convention rules bundle #4448 @alanmcruickshank
- References rule bundle #4446 @alanmcruickshank
- Structure and Ambiguous rule bundles #4444 @alanmcruickshank
- TSQL: Bare functions #4439 @jpers36
- Pull dbt CI tests forward to 1.1 and 1.4 #4442 @WittierDinosaur
- Teradata: Added "AND STATS" options when creating table #4440 @dflem97
- Add Databricks as a distinct dialect #4438 @WittierDinosaur
- Remove importlib deprecated methods #4437 @alanmcruickshank
- SQLite: Support PRAGMA statements #4431 @WittierDinosaur
- Proposed graceful handling of noqa by L016 (#4248) #4424 @alanmcruickshank
- DuckDb: Allow quoted literals as identifiers #4410 @WittierDinosaur
- SQLite Refactor to reduce statement and keyword scope #4409 @WittierDinosaur
- L046 and L056 recode #4430 @alanmcruickshank
- Recode Aliasing Rules #4427 @alanmcruickshank
- Adjust MySQL dialect to support combination of not-null, default and … #4426 @FabianScheidt
- Revert some changes to tox #4428 @alanmcruickshank
- Migrate capitalisation rules to plugin and recode #4413 @alanmcruickshank
- Prep version 2.0.0a5 #4419 @github-actions
- Handle long lines without trailing newlines gracefully (#4386) #4423 @alanmcruickshank
- Resolve #4184 (index error in L007) #4422 @alanmcruickshank
- Handle untaken positive indents with taken negative pair. #4420 @alanmcruickshank
- Postgres: AS MATERIALIZED support #4417 @saintamh
- Align warnings config with example shown #4421 @briankravec
- BigQuery: parse "AS description" part of assert expressions #4418 @yoichi
- Deprecate doc decorators (replace with metaclass) #4415 @alanmcruickshank
- Enable noqa using aliases and groups #4414 @alanmcruickshank
- Add rule names to CLI outputs #4400 @alanmcruickshank
- Postgres: Remove execution keyword inherited from ANSI #4411 @WittierDinosaur
- Rule names, aliases and more complicated selection. #4399 @alanmcruickshank
- Postgres: Support Recursive View #4412 @WittierDinosaur
- T-SQL: Implement BULK INSERT statement #4381 @borchero
- L062: Add match_source (#4172) #4335 @vesatoivonen
- TSQL: Add SET to ALTER TABLE #4407 @jared-rimmer
- Snowflake: ALTER STORAGE INTEGRATION segment #4406 @jared-rimmer
- Fix incorrect link to pre-commit docs #4405 @pdebelak
- Add Snowflake dialect ALTER ROLE segment #4403 @jared-rimmer
- Improving Postgres create index statement #4356 @jmpfar
- Resolve #4291: Comments forcing unexpected indents. #4384 @alanmcruickshank
- Resolve #4294: Comments affecting indentation #4337 @alanmcruickshank
- Resolve #4292: Window function long line fixes #4383 @alanmcruickshank
- TSQL: ALTER INDEX #4364 @AidanHarveyNelson
- Added Varying Keyword to allowed data type segments #4375 @AidanHarveyNelson
- Add ruff linter #4372 @greg-finley
- Fix postgres column constraint default syntax #4379 @pdebelak
- Allow function names to have a leading underscore #4377 @gavin-tsang
- TSQL: Merge Hints #4354 @AidanHarveyNelson
- TSQL: Temporal Table #4358 @AidanHarveyNelson
- TSQL: ALTER TABLE #4369 @AidanHarveyNelson
- Bugfix: Duckdb SELECT * #4365 @WittierDinosaur
- Postgres: TABLESAMPLE query #4357 @greg-finley
- reindent refactor #4338 @alanmcruickshank
- Snowflake: INSERT INTO #4363 @AidanHarveyNelson
- Docs: correct toml syntax of pyproject.toml file config example #4361 @imrehg
- Allowed Naked Identifiers #4359 @AidanHarveyNelson
- TSQL: TABLESAMPLE #4353 @AidanHarveyNelson
- Tsql: Function Parameters #4352 @AidanHarveyNelson
- Postgres: Storage parameters #4350 @AidanHarveyNelson
- TSQL: Bare Function Set #4351 @AidanHarveyNelson
- Postgres: View options #4340 @AidanHarveyNelson
- BigQuery: SELECT DISTINCT AS STRUCT #4341 @joar
- Snowflake: Fix Alter Warehouse #4344 @AidanHarveyNelson
- Parser: Optimise lookahead_match #4327 @WittierDinosaur
- Add support for dbt test macros #4319 @pdebelak
- Bracket complex expressions before applying :: operator in Rule L067 #4326 @pdebelak
- Prep version 2.0.0a4 #4322 @github-actions
- BigQuery: Alter table alter column #4316 @greg-finley
- Handle renamed dbt exceptions #4317 @greg-finley
- Parser: Fix early exit for simple matchers #4305 @WittierDinosaur
- MySQL: Add CREATE DATABASE and ALTER DATABASE #4307 @yoichi
- BigQuery: Add ALTER VIEW #4306 @yoichi
- toml: only install
tomldependency if < Python 3.11 (otherwise use builtintomllib) #4303 @kevinmarsh - Fix #4024 example plugin unit tests import #4302 @matthieucan
- Prep version 2.0.0a3 #4290 @github-actions
- Move ISSUE from Snwoflake reserved keywords to unreserved ones #4279 @KaoutherElhamdi
- Due to performance and other issues, revert the osmosis implementation of the templater for now #4273 @barrywhart
- Simplify lexing #4289 @alanmcruickshank
- Fix #4255 (Fix exception on mixed indent description) #4288 @alanmcruickshank
- Fix #4253 (incorrect trigger of L006 around placeholders) #4287 @alanmcruickshank
- Fix #4249 (TSQL block comment indents) #4286 @alanmcruickshank
- Fix #4252 (Resolve multiple sensible indents) #4285 @alanmcruickshank
- Parser Performance: Cache segment string repr to reduce function calls #4278 @WittierDinosaur
- Snowflake: GRANT SUPPORT CASES #4283 @WittierDinosaur
- Dialect: duckdb #4284 @WittierDinosaur
- Snowflake: Add variable pattern to CopyIntoTable #4275 @WittierDinosaur
- Postgres: Non-reserved keyword bugfix #4277 @WittierDinosaur
- Hive: Add Table constraints DISABLE VALIDATE #4281 @WittierDinosaur
- Snowflake: Add Python and Java UDF support #4280 @WittierDinosaur
- SparkSQL: Support DIV binary operator #4282 @WittierDinosaur
- BigQuery: Add ALTER TABLE #4272 @yoichi
- Snowflake: Update bare functions #4276 @WittierDinosaur
- Improve Dockerfile to reduce image size #4262 @tdurieux
- Prep version 2.0.0a2 #4247 @github-actions
- Push indents to after comments #4239 @alanmcruickshank
- Templated fix improvements and indentation #4245 @alanmcruickshank
- Fix block comment indent fixes #4224 #4240 @alanmcruickshank
- Fix for #4222 #4236 @alanmcruickshank
- Snowflake: Allow multiple unpivot #4242 @greg-finley
- postgres: add row-level locks to SELECT statements #4209 @Yiwen-Gao
- Add more parsing logic for db2 #4206 @NelsonTorres
- Include the filename in critical exceptions #4225 @alanmcruickshank
- Update Readme Badges #4219 @alanmcruickshank
- diff-quality: Handle the case where there are no files to check #4220 @barrywhart
- Prep version 2.0.0a1 #4203 @github-actions
- Fixed False Positive for L037 #4198 @WillAyd
- Fix #4215 #4217 @alanmcruickshank
- don't consider templated whitespace #4213 @alanmcruickshank
- show fatal errors regardless #4214 @alanmcruickshank
- don't pickle the templater #4208 @alanmcruickshank
- MySQL: Support column character set and collation #4204 @yoichi
- Fix some issues with Docker Compose environment #4201 @barrywhart
- Implicit Indents #4054 @alanmcruickshank
- Tweak Coveralls settings #4199 @barrywhart
- In addition to Codecov, also upload to Coveralls #4197 @barrywhart
- Fix: create table default cast returns unparsable section #4192 @NelsonTorres
- Fix JSON parsing issue with diff-quality plugin #4190 @barrywhart
- Codecov migration #4195 @alanmcruickshank
- Stop adding trailing os.sep if ignore file is on the root of the file… #4182 @baa-ableton
- Port dbt-osmosis templater changes to SQLFluff #3976 @barrywhart
- Reflow 4: Long Lines #4067 @alanmcruickshank
- Fix comment bug on reindent #4179 @alanmcruickshank
- Reflow 3: Reindent #3942 @alanmcruickshank
New Contributors
- @konnectr made their first contribution in #4488
- @ValentinCrr made their first contribution in #4490
- @FabianScheidt made their first contribution in #4426
- @dflem97 made their first contribution in #4440
- @timcosta made their first contribution in #4391
- @AidanHarveyNelson made their first contribution in #4344
- @joar made their first contribution in #4341
- @jmpfar made their first contribution in #4356
- @jared-rimmer made their first contribution in #4403
- @vesatoivonen made their first contribution in #4335
- @briankravec made their first contribution in #4421
- @saintamh made their first contribution in #4417
- @tdurieux made their first contribution in #4262
- @baa-ableton made their first contribution in #4182
- @WillAyd made their first contribution in #4198