Highlights
This major release brings support for internal rust optimisations. Rust support is still effectively in beta and is available on an opt in basis. To take advantage of these performance improvements, users can install sqlfluff[rs] to get the parser and lexer running in rust. The current implementation does yield noticeable gains on large files and projects - although for small individual files, the overhead of moving data into rust means that performance is a tiny bit slower. Functionality has proven fairly stable during pre release testing, and we anticipate that the rust implementations will become defaults from 5.0 onwards.
We advise testing the rust package in a development environment before going direct to production.
A massive shout out to @keraion, who's been leading the way on this effort.
Additionally in this release:
- We drop support for dbt 1.4 and older, and include support for dbt 1.10.
- We add the capability to force implicit indents.
- Bufixes for CV12 and RF01.
- Tons of docs and dialect improvements.
What’s Changed
- Databricks: Add support for AUTO CDC INTO and CREATE FLOW (#7360) @TheCleric
- Optimise rust implementation by using sparse representations (#7354) @alanmcruickshank
- Update .gitattributes to exclude generated Rust code (#7356) @keraion
- docs: Update gettingstarted.rst (#7353) @annebelleo
- Sort reserved keywords in anti_template regex for deterministic output (#7339) @keraion
- Initial implementation of the Rust parser (#7342) @keraion
- Add INCLUDE and WHERE clause support to T-SQL inline table indexes (#7352) @copilot-swe-agent[bot]
- docs: Update tox installation command (#7336) @annebelleo
- Adds support for associative array of collections in Oracle. (#7330) @joaostorrer
- databricks: support for named parameters (#7204) @kyletl
- Adds support for
CREATE SYNONYM,DROP SYNONYMandALTER SYNONYMstatements in Oracle. (#7291) @joaostorrer - TSQL: Enhance GROUP BY clause to support T-SQL specific terminators (#7327) @peterbud
- DuckDB - add support for IN operator on strings, lists, and columns (#7273) @stevenlw-porpoise
- Postgres: Add support for composite-value expansion (#7322) @annebelleo
- TSQL: Change DATASOURCE to DATA_SOURCE in openrowset(bulk...) (#7316) @lynndb0
- Postgres: Identify type names for
CREATE TYPEandALTER TYPE(#7314) @annebelleo - Postgres: add support for RETURNING clause (#7317) @annebelleo
- TSQL: add fulltext catalog and openxml plus other fixes (#7311) @nathanthorell
- BigQuery: add support for FORMAT clause (#7310) @annebelleo
- BigQuery: explicitly define data types (#7305) @annebelleo
- CI: Update macos runners (#7270) @keraion
- chore(snowflake): alter dynamic table (un)set delimited values (#7279) @mrlannigan
- fix(rebreak): unhandled
NotImplementedErrorexception (#7285) @peterbud - Adds support for
CREATE DATABASE LINK,DROP DATABASE LINKandALTER DATABASE LINKstatements in Oracle. (#7284) @joaostorrer - update duckdb FILTER clause to make WHERE keyword optional (#7271) @stevenlw-porpoise
- TSQL: allow special characters in variable and table names (#7262) @peterbud
- Adds support for
ENABLE/DISABLEclause after constraints inCREATE TABLEstatements in Oracle. (#7250) @joaostorrer - Fix click version constraint, bumping to 8.3.x (#7297) @serl
- TSQL: new rule to remove empty batches (#7280) @peterbud
- Docs notes on releasing 4.0 (#7264) @alanmcruickshank
- Add instructions for agents (#7261) @peterbud
- Add support for new duckdb lambda syntax (#7256) @stevenlw-porpoise
- refactor: Move Rust parts into their own crates (#7229) @keraion
- fix(duckdb): add support for trailing commas in select * rename/exclude/replace commands (#7254) @stevenlw-porpoise
- Resolve rust templating assertions (#7252) @alanmcruickshank
- TSQL: Remove redundant optional DelimiterGrammar from individual statements (#7231) @peterbud
- Impala: Add INVALIDATE METADATA and REFRESH statements (#7251) @mrebaker
- T-SQL: Enhance Alter Database with Set Recovery Options (#7241) @CactusCollider
- TSQL: Add support for variable assignments in SELECT statements (#7222) @peterbud
- Added alignment_coordinate_space to ALLOWABLE_LAYOUT_CONFIG_KEYS (#7237) @tojnk
- Support aliasing in Snowflake PIVOT clause (#7245) @robert-norberg
- Fix util script to update Cargo.lock (#7249) @keraion
- Postgres: Implement UTF-8 identifier support (#7242) @adnandaut
- TSQL: CreateServerRoleSegment (#7201) @piers-williams
- TSQL: Rework DatatypeSegment (#7200) @peterbud
- Update Datacoves image source in README (#7227) @keraion
- Slightly more obvious logging when rust extensions are used (#7226) @alanmcruickshank
- Lock versions of sqlfluffrs to versions of the main package. (#7218) @alanmcruickshank
- Improve phrasing of ST11 (#7221) @annebelleo
- Another pass on logos (#7219) @alanmcruickshank
- Use distinct upload token for rust package deploy (#7216) @alanmcruickshank
- Remove checks on Rust release action step (#7215) @alanmcruickshank
- DuckDB: Add support for COPY TO statement options (#7209) @tvogels01
- TSQL: Add rule to wrap with BEGIN/END for multi-statement procedures (#7202) @peterbud
- BigQuery: recognize STRUCT and ARRAY data types for CP01/CP05 (#7203) @annebelleo
- Docs logos (#7213) @alanmcruickshank
- Clickhouse: Adds support for parametric aggregate functions (#7205) @franloza
- TSQL: update OutputClauseSegment to match specification (#7207) @peterbud
- Write an implementation of the lexer in Rust (#7132) @keraion
- Disregard CV12 when UNNEST is detected in from_expression_element (#7211) @annebelleo
- TSQL: Update T-SQL keyword identifier usage (#7191) @peterbud
- Add the option to require implicit indents (#7197) @WittierDinosaur
- Update dbt CI matrix (#7196) @WittierDinosaur
- SparkSQL: allow accessing nested array fields for RF01 (#7198) @Ruward
New Contributors
- @serl made their first contribution in #7297
- @lynndb0 made their first contribution in #7316
- @kyletl made their first contribution in #7204
- @piers-williams made their first contribution in #7201
- @adnandaut made their first contribution in #7242
- @robert-norberg made their first contribution in #7245
- @tojnk made their first contribution in #7237
- @CactusCollider made their first contribution in #7241
- @stevenlw-porpoise made their first contribution in #7254
- @Ruward made their first contribution in #7198
- @tvogels01 made their first contribution in #7209