This release contains backwards incompatible changes:

• MySQL 8.0 Server Connection Metadata – When connecting to a Dolt sql-server, Dolt now advertises that it is a MySQL 8.0 server, instead of a MySQL 5.7 server as in previous Dolt versions. This is a backwards incompatible change, because some clients (e.g. Laravel) use this metadata to determine which SQL features and syntax to use.
• Recovery of Dropped Databases – Dropped databases can now be restored by using the dolt_undrop() stored procedure. This is a backwards incompatible change, because disk drive space is no longer automatically reclaimed after dropping a database; instead, it can be manually purged with the new dolt_purge_dropped_databases(); procedure.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 6795: /.github/scripts/sql-correctness/get-dolt-correctness-job-json.sh: change correctness precision
• 6784: go: sqle: cluster: Make users and grants replication respect dolt_cluster_ack_writes_timeout_secs.
• 6783: go: sqle: cluster: Fix a bug when replicating users and grants and branch control where updates two updates which came together quickly could fail to replicate the second update.
• 6782: go: sqle: cluster: Make branch control replication respect dolt_cluster_ack_writes_timeout_secs
• 6771: integration-tests/go-sql-server-driver: Add a test for undropping a cluster replicated database.
• 6762: Feature: New stored procedure dolt_undrop() to restore dropped databases
  This change introduces a new stored procedure, dolt_undrop(<database name>) that can be used to restore a dropped database. Dropped databases are now moved to a new dolt_dropped_databases directory until they are either restored or permanently purged with the new dolt_purge_dropped_databases stored procedure.
  Usage:

  # Use dolt_undrop() with the database name to restore a dropped database
  drop database mydb;
  call dolt_undrop('mydb');
  ...
  # Use dolt_purge_dropped_databases() with no arguments to permanently delete all dropped databases
  drop database mydb;
  call dolt_purge_dropped_databases;

  Required Privileges: Calling dolt_undrop() does not require any special privileges. Calling dolt_purge_dropped_databases() requires SUPER privileges.
  Testing Notes: Tests are in BATS, since Dolt enginetests use InMemoryBlobStore instead of writing db data to a filesystem, so there isn't currently a good way to undrop databases in enginetests. If the noms data files were stored through the InMemFS, then the same code would work in our tests to undrop databases. Currently the repo state files are able to be undropped, but not the actual database data, since it disappears with the InMemoryBlobStore reference when we drop the database.
  Fixes: #6484
  Documentation Updates: dolthub/docs#1768

go-mysql-server

• 2065: Remove redundant information_schema creation in the driver example
  In the driver's example, information_schema is duplicated.
  It is created in two places.
  1. factory.Resolve()
  2. Catalog in Analyzer in Driver.OpenConnector()
     There is no need to create it in the factory.Resolve().
     I checked databases using this:

  diff --git a/driver/_example/main.go b/driver/_example/main.go
  index 34e0580ed..adbc1a249 100644
  --- a/driver/_example/main.go
  +++ b/driver/_example/main.go
  @@ -35,6 +35,14 @@ func main() {
  rows, err := db.Query("SELECT * FROM mytable")
  must(err)
  dump(rows)
  +
  +       rows, err = db.Query("SHOW DATABASES")
  +       must(err)
  +       for rows.Next() {
  +               var db string
  +               must(rows.Scan(&db))
  +               fmt.Println("db:", db)
  +       }
  }
  func must(err error) {

• 2064: Implement fast join ordering for large joins that can be implemented as a series of lookups.
  Fixes #6713
  This PR is built on top of dolthub/go-mysql-server#2063 and includes its changes, plus two additional commits.
  The purpose of this PR is to attempt to construct a left-deep join ordering where every join is possible to optimize into a lookup. It uses Functional Dependencies to identify the column set that determines every other set in the final join. If a "lookup only join order" exists, then these columns must be in the innermost join. We use that as the basis and pick join filters one at a time to create a new join plan.
  Currently this only handles the case where a single column determines every other column, and only the case where at each juncture, only a single possible choice can be made for the next table in the join. This is sufficient to handle the queries used in sqllogictests
  Right now, this is intentionally limited in scope: it's designed to only affect the extremely large (up to 64 tables) joins used in sqllogictests, and won't currently be used outside of large joins. But there's no reason that this can't be used for other joins. In fact, in some cases I observed this generate correct join plans that our current brute-force reordering misses. (edge.applicable claims that these ordering violate a conflict rule even though they should be correct. We may be overly conservative in some places.)
• 2063: Improve functional dependency analysis for joins.
  This is a prerequisite for fixing #6713.
  Basically, this PR does two things:
  • Adds additional bookkeeping to FDS in order to track "partial FDS keys", that is, column sets that determine some (but not all) of the other columns in a relation. Before this PR, we attempted to only track keys that determined the entire relation. This would cause us to lose some information and prohibit some optimizations. (There were also a couple of cases in nested joins where we accidentally added partial keys to FDS anyway (by adding a key from a child table to the FDS of the parent.) If we ever used these keys it could have caused correctness issues, but it doesn't look like we ever did.
  • Improves the simplifyCols method in FDS to use partial keys in order to improve analysis.
    Overall, these changes allow us to compute much better FDS keys for complicated joins, by allowing us to remember and reuse keys derived from child tables to improve the computed key for tables later in the join.
• 2062: "CREATE TABLE" fails with a "database not found" error when using the driver
  In the main branch (commit 4cc2f2c), executing the following SQL in the driver's example (driver/_example) failes with the error database not found: mydb:

  diff --git a/driver/_example/main.go b/driver/_example/main.go
  index 34e0580ed..64fe11a2b 100644
  --- a/driver/_example/main.go
  +++ b/driver/_example/main.go
  @@ -35,6 +35,9 @@ func main() {
  rows, err := db.Query("SELECT * FROM mytable")
  must(err)
  dump(rows)
  +
  +       _, err = db.Exec("CREATE TABLE table2 (id integer, primary key (id))")
  +       must(err)
  }
  func must(err error) {

  output:

  John Doe john@doe.com ["555-555-555"] 2023-10-09 21:29:48.750044594 +0900 JST m=+0.016306123
  John Doe johnalt@doe.com [] 2023-10-09 21:29:48.750060666 +0900 JST m=+0.016322195
  Jane Doe jane@doe.com [] 2023-10-09 21:29:48.750067418 +0900 JST m=+0.016328947
  Evil Bob evilbob@gmail.com ["555-666-555", "666-666-666"] 2023-10-09 21:29:48.750073628 +0900 JST m=+0.016335158
  2023/10/09 21:29:48 database not found: mydb
  exit status 1
  

  This is because the DefaultSessionBuilder of the driver is initialized with an empty DBProvider.
  driver/session.go:22
  In this PR, I have fixed this by passing the DatabaseProvider returned by factory.Resolve() to the SessionBuilder.
• 2061: More histogram support, redo stats table
  Reshape statistics interfaces to better support custom Dolt implementation.
  • Add ANALYZE TABLE <table> [UPDATE/DROP] HISTOGRAM ON <column,...> USING <json blob> support and a few tests
  • Replace use of update information schema cardinality updates with histogram updates
  • Default catalog has an in-memory histogram
  • New memory histogram impl
  • Delete old statistics table/histogram code
    The only prod difference is that the new update path can overwrite the default table count, while it was a testing-only thing before.
    companion: dolthub/vitess#279
    Dolt bump seems OK
• 2060: Fixed various bugs in last_insert_id
  Fixes #6776
  Also adds better test coverage for LAST_INSERT_ID() and the INSERT_ID field in the ok response for updates.

vitess

• 280: Update the default server version to 8.0.33
  The connection handshake was advertising a server version of 5.7.9-Vitess and some clients were using that info and trying and speak MySQL-5.7 to Dolt (example issue)
  This change updates the default advertised server version to 8.0.33-Dolt.
  Dolt CI tests are running at: #6798
• 279: Add ANALYZE HISTOGRAM support
• 277: Allow parsing of CREATE TABLE t AS (...) UNION (...)
  This allows parsing of CREATE TABLE AS statements when the expression being used to create the table is a set operation like UNION, INTERSECT, or EXCEPT.
  The "AS" keyword is typically optional. But this change only allows set ops to be used with CREATE ... AS when the AS is explicit. This is to avoid an ambiguity in the current grammar when attempting to parse CREATE TABLE t (, where what follows could be a set op or a table definition. Fully matching MySQL's spec here would require rewriting our grammar to avoid this ambiguity, which is outside the scope of the PR. However, this PR makes us strictly more correct than we were before.

Closed Issues

• 6484: Drop Database is destructive. Could make it more safe.
• 5820: Dolt works poorly with prepared emulation off in PHP PDO
• 6641: CREATE TABLE {name} SELECT ... UNION SELECT ... fails to parse.
• 6797: Laravel example app requires environment variable Dolt does not support
• 6713: Dolt takes forever to generate join plans for a join of many tables.
• 6776: SELECT LAST_INSERT_ID() returns BIGINT SIGNED for BIGINT UNSIGNED ids

Latency