Highlights
This release brings several breaking changes to previous releases. Most notably:
-
It drops support for python 3.7, which reached end of life in June 2023.
-
It migrates to
pyproject.tomlrather thansetup.cfgas the python packaging configuration file (although keepingsetuptoolsas the default backend). -
The serialised output for
sqlfluff lint(and the corresponding API methods) now contains more information about the span of linting issues, initial proposed fixes and several statistics which were previously only accessible via csv export. Beside the new fields, the original fields ofline_posandline_nohave been renamed tostart_line_posandstart_line_no, to distinguish them from the new fields startingend_*. -
The default
annotation_levelset by the--annotation-leveloption on thesqlfluff lintcommand has been changed fromnoticetowarning, to better distinguish linting errors from warnings, which always now have the level ofnotice. This is only relevant when using thegithub-annotationorgithub-annotation-nativeformats. -
A change in the default behaviour for
convention.not_equals. The new default is to beconsistent, which is slightly more relaxed than the original behaviour. -
The
--forceoption has been deprecated onsqlfluff fixas that option is now the default behaviour. This is to enable significant reductions in memory overhead when linting large projects. -
The long since deprecated
--disable_progress_baroption has been removed (which was replaced by the kabab-case--disable-progress-barmore than a year ago). -
Plugins are now loaded progressively, and with error handling. If a plugin fails to load, SQLFluff will now continue onward and try to run regardless while also showing a more helpful error message.
On top of these changes, there have a been a whole host of dialect improvements and additions, in particular the inclusion of avertica dialect for the first time. There's also:
-
A new rule (
aliasing.self_alias.column) which prevents aliasing a column as itself. -
A change to disables AL01 (
aliasing.table) by default for Oracle. -
A change to allow AL05 to allow aliasing for a
VALUESclause.
For more specifics please take a look at the release notes.
Thanks to the community for patience during the release cycle for 3.0.0, which has taken a little longer than expected. Thanks also to the TWENTY SEVEN new contributors whose changes are included in this release. πππππ
Whatβs Changed
- Progressively load plugins #5661 @alanmcruickshank
- Postgres: AL05, ignore aliases in values clause #5669 @keraion
- Add Postgres CREATE FOREIGN TABLE statement #5657 @edpft
- Lexer: Handle escaped curly brace slices from the python templater #5666 @keraion
- [CI]: Update pre-commit hook versions #5665 @keraion
- Resolves #5624: Snowflake unparsable unset table options #5664 @andychannery
- Revert Ruff Changes #5662 @alanmcruickshank
- Complete the memory overhead work on cli fix #5653 @alanmcruickshank
- Resolve #5647: Snowflake unparsable variant access after cast #5658 @andychannery
- BQ PK and FK #5654 @OTooleMichael
- Prep version 3.0.0a6 #5652 @github-actions
- Add Support for Databricks
CREATE FUNCTIONSyntax in SparkSQL Parser #5615 @mitchellvanrijkom - Swap fix
--forcefor--check#5650 @alanmcruickshank - Remove
DeprecatedOption#5649 @alanmcruickshank - Resolve broken loop limit test #5651 @alanmcruickshank
- Snowflake: Move NOTIFY to non-reserved words #5645 @greg-finley
- BigQuery: GROUP BY ALL #5646 @greg-finley
- chore: use pre-calculated
_code_indicesin `BaseSegment::raw_segmen⦠#5644 @gvozdvmozgu - Fix Snowflake Semistructured identifier parsing regex-expression #5635 @DannyMor
- Postgres: Update ReferentialActionGrammar to support sets of columns #5628 @WittierDinosaur
- Snowflake: Add syntax for masking policy force #5629 @WittierDinosaur
- Postgres: Allow nested block comments #5630 @WittierDinosaur
- Postgres: Add Create, Alter, Drop Statistics #5631 @WittierDinosaur
- T-SQL Fix relative sql filepath lexer #5632 @WittierDinosaur
- Tech Debt: Replace some sequences with their Ref equivalents #5633 @WittierDinosaur
- ANSI/MYSQL: Support Create Role If Not Exists #5634 @WittierDinosaur
- Add Vertica dialect #5640 @PolitePp
- Add Support for Snowflake Materialised View and Column Masking Policy #5637 @ulixius9
- [snowflake dialect] support ALTER TABLE ... ADD COLUMN IF NOT EXISTS #5621 @gshen7
- SQLite: Make
DISTINCT FROMoptional; SQLite/TSQL/Exasol: Nothing'dNanLiteralSegment#5620 @keraion - Upgrade greenplum dialect #5546 @kkozhakin
- Oracle: parse length qualifier in types #5613 @Jefffrey
- Multiple Dialects: Fix handling of nested sets expressions #5606 @keraion
- DB2: Add labeled durations and special registers #5612 @keraion
- Sparksql: Fix
LATERAL VIEWfollowingJOIN;CLUSTER|SORT|DISTRIBUTE BYorQUALIFYwithoutFROM#5602 @keraion - File helpers and config test parameterisation. #5579 @alanmcruickshank
- Memory overhead optimisations during linting #5585 @alanmcruickshank
- fix: multiple columns foreign key constraint (#5592) #5594 @maoxingda
- [CI] Add
no_implicit_reexportmypy check #5509 @Koyaani - Prep version 3.0.0a5 #5512 @github-actions
- Add support & test for postgres alter policy with multiple clauses #5577 @fnimick
- Update github actions to latest versions #5584 @alanmcruickshank
- Allows using dbt cross project ref in jinja templater #5574 @alangner
- Improve support for Jinja templater plugins with custom tags #5543 @james-johnston-thumbtack
- Databricks: fix
EXCEPTwith qualified column reference #5557 @keraion - Stricter recommended config for not_equals convention #5580 @alanmcruickshank
- CV01: Add options for ANSI and consistent style. #5539 @keraion
- DuckDB: Fix
REPLACEafterEXCLUDE. Fix AL03 linting for wildcard like expression. #5556 @keraion - Clickhouse: Add
GLOBAL JOIN,GLOBAL IN, andPASTE JOIN#5560 @keraion - [Docs] Use extended policy for identifier capitalisation in starter config #5562 @j-svensmark
- Build: linting black 24.1.0 rules update #5573 @keraion
- Snowflake: Updating Snowflake dialect to pass acceptable RLS policy objects #5559 @k1drobot
- Redshift Syntax: ALTER APPEND #5545 @OTooleMichael
- DuckDB: Add ANTI, SEMI, ASOF, and POSITIONAL joins #5544 @keraion
- MySQL: fix FIRST keyword in ALTER ADD/MODIFY #5537 @archer62
- Postgres/DB2/Oracle: Fix comma join
LATERAL. #5533 @keraion - Add new Rule AL09: Avoid Self Alias #5528 @aayushr7
- Rule AL01: disabled for Oracle dialect #5517 @keraion
- Postgres ALTER EXTENSION support: dialect & tests #5527 @remy-gohiring
- SparkSQL: Add
UNPIVOTsyntax. FixTABLESAMPLEaliases. #5524 @keraion - DuckDB: Added support for
PIVOTandUNPIVOT#5514 @keraion - Fix parse error databricks window function starts with order by #5493 @snkekorfus
- Hive: allow UDTF to return multiple column aliases in SELECT #5495 @reata
- DB2: Add support for
DECLARE GLOBAL TEMPORARY TABLES,OFFSET,CALL, and non-bracketedVALUES#5508 @keraion - DuckDB: Add CREATE OR REPLACE TABLE syntax #5511 @keraion
- TSQL: Top and Distinct in same query #5491 @greg-finley
- [Spark/Databricks] Fix: make COLUMNS in APPLY CHANGES INTO optional #5498 @rocwang
- SparkSQL: exclamation mark as logical not #5500 @reata
- SparkSQL: allow value in set_statement to be Java class name #5504 @reata
- SparkSQL: allow distribute/sort/cluster by at end of set operation #5502 @reata
- [CI] Add a few more mypy ckecks #5505 @Koyaani
- Snowflake dialect: Add support for DATABASE ROLE in GRANT/REVOKE #5490 @sfc-gh-dgupta
- DuckDB: Qualify and From-First #5485 @keraion
- MySql: create table: allow null/not null in any position #5473 @archer62
- Snowflake dialect: Support for CREATE DATABASE ROLE #5475 @sfc-gh-dgupta
- Clickhouse Dialect - Support BackQuoted Identifiers #5457 @kaiyannameighu
- Change Color.lightgrey to have a white background - dark theme friendly #5458 @ryaminal
- Fix indentation for single cube clause #5462 @tunetheweb
- Prep version 3.0.0a4 #5455 @github-actions
- Build out rule and fix serialisation #5364 @alanmcruickshank
- Add warning about github actions annotations limit #5450 @alanmcruickshank
- chore: remove unused line initialization in ParseContext #5448 @gvozdvmozgu
- Refine CLI testing fixture #5446 @alanmcruickshank
- Update black linting #5447 @alanmcruickshank
- Prep version 3.0.0a3 #5444 @github-actions
- fix assertion in
test__api__lint_string_specific_exclude_single#5437 @gvozdvmozgu - Databricks CLUSTER BY and OPTIMIZE #5436 @greg-finley
- TSQL: ON DELETE NO ACTION #5434 @greg-finley
- Snowflake dynamic table #5435 @greg-finley
- Support parsing CONSTRAINT definitions when creating Delta Live Tables in SparkSQL/Databricks #5438 @rocwang
- adds few fixes for databricks/sparksql #5431 @markbaas
- TSQL: CREATE USER {FOR|FROM} LOGIN #5426 @greg-finley
- Snowflake Create table order/noorder #5421 @greg-finley
- Simplify Snowflake regexes #5419 @greg-finley
- Permit .* after each tbl_name in multi-table delete syntax #5408 @yoichi
- Fix snowflake add search optimization grant #5412 @jongracecox
- ANSI: Allow combination of UNION clause and WITH clause #5413 @yoichi
- SQLite: Allow block comments to be terminated by end of input #5400 @WittierDinosaur
- Update publish github action to use kebab case #5392 @alanmcruickshank
- Prep version 3.0.0a2 #5391 @github-actions
- Update publish actions and Dockerfile. #5390 @alanmcruickshank
- Prep version 3.0.0a1 #5381 @github-actions
- Move the rest of pytest over to
pyproject.toml#5383 @alanmcruickshank - Move doc8 over to pyproject #5385 @alanmcruickshank
- Remove exceptions for typing stubs that now exist. #5382 @alanmcruickshank
- Migrate to
pyproject.tomlfor the core project. #5373 @alanmcruickshank - Fix the snippet from a pyproject.toml in configuration.rst #5378 @ishiis
- Snowflake: Support AlterNetworkPolicy Statements #5377 @WittierDinosaur
- postgres: add support for bodies of "language sql" functions #5376 @65278
- Add support for SET NAMES statement in MySQL #5374 @joaostorrer
- Fix GRANT ALL PRIVILEGES statement in MySQL #5375 @joaostorrer
- Another extraction of fixing logic. #5365 @alanmcruickshank
- Remove root requirements.txt #5372 @alanmcruickshank
- Drop support of python 3.7 #5288 @zhongjiajie
- API configuration documentation #5369 @golergka
- add listagg extras support in trino dialect #5368 @wjhrdy
- Allow ignoring of comments from indentation entirely #3311 #5363 @alanmcruickshank
- Trino: Support Analyze statements #5361 @WittierDinosaur
- Resolve #5327 (logging to stdout) #5362 @alanmcruickshank
New Contributors
- @edpft made their first contribution in #5657
- @maoxingda made their first contribution in #5594
- @Jefffrey made their first contribution in #5613
- @kkozhakin made their first contribution in #5546
- @gshen7 made their first contribution in #5621
- @PolitePp made their first contribution in #5640
- @DannyMor made their first contribution in #5635
- @mitchellvanrijkom made their first contribution in #5615
- @ryaminal made their first contribution in #5458
- @sfc-gh-dgupta made their first contribution in #5475
- @archer62 made their first contribution in #5473
- @keraion made their first contribution in #5485
- @Koyaani made their first contribution in #5505
- @snkekorfus made their first contribution in #5493
- @remy-gohiring made their first contribution in #5527
- @aayushr7 made their first contribution in #5528
- @k1drobot made their first contribution in #5559
- @alangner made their first contribution in #5574
- @fnimick made their first contribution in #5577
- @jongracecox made their first contribution in #5412
- @markbaas made their first contribution in #5431
- @rocwang made their first contribution in #5438
- @gvozdvmozgu made their first contribution in #5437
- @wjhrdy made their first contribution in #5368
- @golergka made their first contribution in #5369
- @65278 made their first contribution in #5376
- @ishiis made their first contribution in #5378