Merged PRs

dolt

• 4408: reorder arguments of dolt_diff() table function
  This PR updates argument ordering of dolt_diff() table function to match dolt_diff_summary() table function, which matches CLI command argument order.
• 4401: go/commands: Add sql-server option for validating query results against a MySQL instance
  depends on dolthub/go-mysql-server#1279
• 4400: allow drop current database directory and fix drop db with '-'
  This PR fixes two issues:
  • if the server is running in the database itself, the database can be dropped regardless of being selected database.
  • drop database with '-' in dir name and '_' in database name.
• 4399: adding prepared tests for dolt stored procedures
  Fix for: #4392
• 4397: print spatial types as hex in shells only
  Related PR: dolthub/go-mysql-server#1278
  Since the SQL methods for Point, LineString, Polygon, and Geometry no longer return the hex formatted string, need to do a little extra work for prettier shell output.
• 4394: sql-server: Implement dolt_cluster_role, dolt_cluster_role_epoch variables, dolt_assume_cluster_role procedure.
  This is just the variable persistence and some validation and rules around how they can transition.
• 4389: Fix race for concurrent dolt_commit calls
  This fixes #4361
• 4387: remotestorage,remotesrv: Be less restrictive in repository paths for remotesapi repos that are not served through dolthub.
  This uses the repo_path parameter to allow single and 3+ path component paths for repository names.
• 4386: go/cmd/dolt/commands/sqlserver: yaml_config.go: Parse and validate cluster: config.
• 4385: Updated bats README and a test to make it easier to get started
  Also improved parquet tests
• 4384: Implement merge stats for the new format
• 4383: go/libraries/doltcore/sqle: ReadReplicaDatabase: Add variable for force pulling remote branches.
  This adds @@global.dolt_read_replica_force_pull, which will turn read replica fetches into force pulls, allowing a read replica to true-up with a remote which has experienced a force push.
• 4363: support dolt_diff_summary_table_function
  dolt_diff_summary table function is the same as CLI dolt diff --summary command. It takes fromCommit, toCommit and tableName, which is optional.
  If tableName is not defined, the table function will output all tables with data diff.
  If the table does not have data diff, it will output empty result.
  If the table is keyless, like CLI command, only row added and row deleted will have appropriate information, and the rest is nil.
  Current issues:
  • drop-column causes row values to be modified into NULL, it should be different from user-defined NULL or NULL added from add-column.
  • for keyless table, update table causes delete old row and insert new row, which affects the row added and deleted result to be incorrect.
• 4268: Feat/support aliyun oss store
  support aliyun oss as backend store

go-mysql-server

• 1281: Alias resolution updates
  This change moves our alias identification and column name qualification code closer to MySQL's behavior. The major changes are:
  • identify available alias, table, and column names on a per-scope level (instead of the previous "nestingLevel" that was based off of the node depth of the execution plan tree)
  • supply additional context to the qualifyExpression method so that the containing node can be used to switch on different alias visibility rules
    There are still more nuanced edge cases in MySQL's alias behavior that we should continue working on matching (e.g. dolthub/go-mysql-server#1285, dolthub/go-mysql-server#1286), but this change moves us a nice step forward.
    Dolt CI Tests:
  • #4410
    Fixes:
  • dolthub/go-mysql-server#525
  • #4344
• 1280: More join types
  Add FullOuterJoin, SemiJoin, and AntiJoin.
  None of these new join nodes are safe for join ordering transformations yet. They are explicitly excluded from join planning.
  The getField indexes for these three nodes' join conditions deserve more consideration. I excluded them from auto-fixup after manually correcting join condition get fields for the appropriate schemas.
  FullOuterJoin uses a union distinct execution operator, which is correct but a lot slower than a merge join-esque operator.
  SemiJoin and AntiJoin rearrange subquery expression scopes. I separate resolve and finalizeSubqueryExpressions to perform decorrelation before predicate pushdown (where we were panicking on FixUpExpressions) and join ordering (we want to decorrelate scopes before join planning).
  Other:
  • query plan tests added for exist hoisting edge cases i did not catch on first pass
  • fixed bug with CTE stars
• 1279: Andy/mysql validator
• 1278: should not be returning hex on the wire for geometry types
  Fix for: #4390
  For improved display purposes, I changed the SQL method to convert the raw binary of spatial types to their hex equivalent (dolthub/go-mysql-server#1068).
  DBeaver expects the binary in WKB format to generate their maps, so our SQL method must return them in binary
• 1276: Apply sql_select_limit to first scope only
  re: #4391 and #4353
• 1275: Throw error in alter table set default if default fails rule(s)
  This PR makes alter column set default error when its default expression fails the default expression rules. Previously the DDL succeeded and when an insert or any other query was performed the default would cause a continuous error.

  alter table t alter column col1 set default '{\"bye\":1}'
  -- errors with: TEXT, BLOB, GEOMETRY, and JSON types may only have expression default values

• 1272: Removed exponential time complexity for foreign key analysis
  This fixes dolthub/go-mysql-server#1268
  Foreign key analysis created acyclical trees that were traversed during query execution to emulate cascade operations. This meant that cyclical foreign keys were converted to an acyclical tree. Normally this isn't possible as cyclical trees are infinitely traversable, but MySQL has a depth limit of 15, which allowed us to materialize an acyclic tree with a maximum height of 15 nodes. This, however, lead to trees with an exponential number of nodes: roughly (number_of_fks)¹⁵ × 1.5 nodes in the tree. With just 3 foreign keys, we'd get a tree with roughly 22 million nodes, which would take forever to process.
  This PR completely changes the analysis step to now generate cyclical trees. In addition, depth checks are now properly implemented (during query execution rather than during analysis), being represented by a returned error once the depth limit has been reached. Interestingly, MySQL is supposed to process up to 15 operations (returning an error on the 16th), but cyclical foreign keys will error on the 15th operation. I think this is a bug in MySQL, but nonetheless the behavior has been duplicated here.
  I also updated the timestamp_test.go file to grab an unused port. This prevents test failures due to requesting an already-in-use port. Not related to this PR in particular, but it was annoying to deal with so I fixed it.

vitess

• 194: adding missing keywords to token.go
  I forgot to add some keywords used in ROW_FMT table option, which caused a bats test to fail
• 193: adding support for PARTITION syntax for table creation
  Parsing table_options for statements like CREATE TABLE (column_definitions) table_options better matches MySQL
  Can parse partition_options based off of https://dev.mysql.com/doc/refman/8.0/en/create-table.html
  Fix for: #4358
• 192: add full join parsing
• 191: go/mysql: query.go: Put strings into BindVars as copies, not slices of the recyclable wire packet buffers.
  Cherry picks vitessio/vitess#5562.
• 190: Adding back support for using reserved keywords unquoted in InsertInto
  I previously excluded reserved_sql_id and only included sql_id because I didn't think reserved keywords should be allowed as identifiers in insert into columns without being quoted, but it appears we have tests in Dolt that rely on that (e.g. sql-diff.bats calls: INSERT INTO test (pk, int, string, boolean, float, uint, uuid) values ...) and I figured if we can support those without the quoting, it seems like a nice feature for customers, so I switched back to reserved_sql_id.
  It is confusing that we refer to identifiers as "reserved" keywords when they don't require backtick quoting though and would be nice to tidy that up in our grammar in the future.

Closed Issues

• 4390: POLYGON column not working/decoding with DBeaver
• 4321: Bad error message on DROP DATABASE when called on a database that initialized the sql-server
• 4319: Can't drop a database with a - in it
• 4404: Dolt doesn't respect the original order of CHECK, UNIQUE constraints in schemas
• 2673: Reorder of columns in UNIQUE INDEX when re-create table
• 4043: Add DOLT_DIFF_SUMMARY table function
• 2636: dolt diff --summary with cell-wise statistics
• 4270: Diff shows rows that have no diffs when a column has been added in the past.
• 4392: Bindvars not filled in for procedure calls
• 4391: sql_select_limit should not apply to subqueries
• 4361: concurrency problems with dolt_commit()
• 4373: dolt fetch upstream main fails on VPS with 1GB RAM
• 4220: Merge not showing correct statistics for rows merged
• 4329: Doltpy v2: Lib to can use Dolt from python
• 4328: Publish libs that can use Dolt
• 4340: Feature Request: Generate JSON_TABLE from table column reference
• 525: Column alias in where clause should be an error
• 1268: Analyzing self-referential foreign keys triggering an infinite loop

Latency

Current Default Format (__LD_1__)

New Format (__DOLT__)