Auto-refresh is now available (off by default). A controller API for statistics stats, stops, and monitors stats threads. More details here.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 7468: go/libraries/events: Restructure events collection so that we can send events while the dolt process is running.
  We make the GlobalCollector mutable.
  We give the Collector a sendingThread which can have an Emitter. When the Collector collects enough events, it forwards a batch of them to the sendingThread. If the sendingThread has an Emitter, it attempts to send its current batch on that Emitter.
  On shutdown, contexts get canceled, background threads get waited on, and then an accumulated un-emitted events are returned from the Collector, same as previously.
• 7463: SELECT ... INTO OUTFILE ... tests
  This PR adds BATS tests for a SELECT ... INTO OUTFILE ... feature we're adding.
  Companion pr: dolthub/go-mysql-server#2317
• 7457: Added storage capabilities for GMS Extended Types
  This adds the necessary scaffolding to support the new ExtendedType interface within GMS. This interface is primarily used by DoltgreSQL to implement PostgreSQL types.
  Related PRs:
  • dolthub/go-mysql-server#2312
  • dolthub/doltgresql#120
• 7424: Stats auto refresh
  Three new system variables to configure stats auto refresh, one for turning on, one for a timer of how often to check whether stats are fresh, and a third for a freshness threshold to trigger writing new stats. The system variables are global, and will apply to all databases in a server context. Adding or dropping databases, tables, and indexes are reflected by the statistics provider.
  Statistic updates now funnel through a process where we preserve buckets for pre-existing chunks and create new buckets specifically for new chunk ordinals. The auto-refresh threshold compares the number of new and deleted buckets compared the the existing bucket count. So inserts, updates, and deletes all create new chunks or delete chunks relative to the previously saved statistics, and the fraction of (new+deleted)/previous count is compared against the threshold.
  TODO:
  • how much logging is appropriate
  • go server tests
  • is stats dataset replication blocked by default?
    other:
  • ANALYZE table and auto-refresh concurrency, last write wins? as long as auto-refresh thread doesn't die
  • drop database/ANALYZE concurrency, error for writing stats to a deleted folder

go-mysql-server

• 2323: SQLEXCEPTION proc handler
  We originally supported one type of procedure handler, NOT FOUND, which explicitly checked for an error when fetching from a cursor io.EOFs. The implementation for that handler would walk the entire BEGIN/END scope stack inside the Fetch call looking for a handler, execute the handler body, and then embed the scope height into a special return error. The error walked back up the callstack looking for the BEGIN/END block embedded in the error message.
  This PR:
  1. Refactors handlers to coordinate message passing in a centralized way. No metadata is embedded in FetchEOF, because each scope will explicitly compare its handlers to errors raised during execution within its BEGIN/END bounds. (FetchEOF is important because we differentiate between 3 different types of io.EOF in procedure loops).
  2. Add type/object support for representing specific signal handers.
  3. Add SQLEXCEPTION signal handling, which will trigger for any error type (other than io.EOF) that bubbles up during a BeginEndIter's execution.
     re: #7454
     Another thing I noticed is that some of our tests return nil for empty stored procedures when mysql returns ERROR 1329 (02000): No data - zero rows fetched, selected, or processed. https://github.com/dolthub/dolt/issues/new
• 2322: Allow InjectedExpr to handle name resolution
  Modifies the InjectedExpr handling to support additional expressions, such as those requiring name resolution.
  Related PRs:
  • dolthub/vitess#312
  • dolthub/doltgresql#121
• 2317: support SELECT ... INTO OUTFILE/DUMPFILE ...
  This adds support for MySQL's SELECT ... INTO OUTFILE/DUMPFILE ... feature.
  It is the complement to LOAD DATA. There is no LOCAL option, so files created using this feature are on the server.
  This PR adds a custom TestSuite for testing these files, as it needs to write, read, and delete files.
  syntax: dolthub/vitess#311
  fixes #7453
• 2316: Allow primary keys with auto_increment columns to be dropped when an appropriate index exists
  MySQL allows a primary key with an auto_increment column to be dropped as long as there is a secondary index that includes the auto_increment column as the first column in the index. (MySQL Reference)
  This PR also enables dropping a primary key by referencing it by it's ID (PRIMARY), in order to match MySQL's behavior, e.g.

  DROP INDEX `PRIMARY` ON t;
  

  Related to #7456
• 2312: Added the ExtendedType interface
  This adds the ExtendedType interface, which is used within DoltgreSQL to implement PostgreSQL types, as well as in Dolt to properly handle the new type and value serialization.
  Related PRs:
  • #7457
  • dolthub/doltgresql#120
• 2300: Fix External Index Creation
  Fixes regression introduced by 9b9301f8c4709d2d32982068322662643e02f231
  Currently

  CREATE INDEX foo USING driver table (column)

  creates a table scan if driver is not "", "btree" or "hash".
  The regression is caused by the change:

  -	outScope.node = plan.NewCreateIndex(
  +	createIndex := plan.NewCreateIndex(
  ddl.IndexSpec.ToName.String(),
  table,
  cols,
  ddl.IndexSpec.Using.Lowered(),
  config,
  )
  +	createIndex.Catalog = b.cat
  return

vitess

• 312: Allow InjectedExpr to handle name resolution
  Adds the capability for additional expressions that need to be resolved to InjectedExpr.
  Related PRs:
  • dolthub/go-mysql-server#2322
  • dolthub/doltgresql#121
• 311: add syntax support for SELECT ... INTO OUTFILE ... options
  We had parser support for SELECT ... INTO OUTFILE, but were lacking parser support for the variety of OUTFILE options.
  This PR adds support for all the options and certain combinations of them.
  However, we are missing support for duplicate options and certain orderings of these options; tests for these are added as error tests with a TODO.
  MySQL Docs:
  https://dev.mysql.com/doc/refman/8.0/en/select-into.html
  MariaDB Docs (this has the actual grammar):
  https://mariadb.com/kb/en/select-into-outfile/
• 309: Added a way for Doltgres to inject expressions
  This primarily adds a new AST node that DoltgreSQL takes advantage of. All currently-existing nodes are specifically designed for MySQL transformations. In many cases, Doltgres will have different transformations that need to take place, and this provides a way for Doltgres to implement those transformations without having to modify other packages whenever a new transformation is added or edited.
  Related PRs:
  • dolthub/go-mysql-server#2312
  • dolthub/doltgresql#120
• 308: Change Handler.ComResetConnection to return an error
  Related to #3921

Closed Issues

• 7453: Add support for "SELECT INTO" for files
• 7456: Can't drop primary index.
• 3921: Implement COM_RESET_CONNECTION

Latency