Merged PRs

dolt

• 3698: Fix Typo
• 3692: typo
• 3687: Fix internal merges not rolling back on a constraint violation
  Bug fix for transactions that create constraint violations.
  Prior to this PR, an internal merge (a merge called inside a transaction) or a transaction merge that produced constraint violations would not rollback the transaction.
  The desired behavior is that if dolt_force_transaction_commit is false, then rollback the transaction if any constraint violations are encountered.
• 3686: Add remote tracking info to dolt status
  Added remote tracking information for dolt status. If the local branch has upstream remote branch, dolt status outputs whether the local branch is ahead of, behind, diverged from or up-to-date with the tracked remote branch.
• 3685: Speed up client tests by installing pre-compiled R packages
  https://packagemanager.rstudio.com maintains binary versions of R packages so one doesn't have to compile them all to install on Linux
• 3682: persistable vars
• 3678: Fix broken link for hosted dolt in readme.md
  Without the HTTPS prefix, the link went to https://github.com/dolthub/dolt/blob/main/hosted.doltdb.com
• 3676: update and fix cherry-pick docs
• 3673: Update README.md to include hosted
• 3670: Remove SELECT CONSTRAINTS_VERIFY_ALL and DOLT_VERIFY_CONSTRAINTS_ALL
  Removes the ALL versions of CONSTRAINTS_VERIFY() and DOLT_VERIFY_CONSTRAINTS(). --all has been added to the remaining SQL function and procedure.
  Also adds --output-only support for SQL
• 3667: Clean up revision database metadata when renaming/deleting branches
  When a Dolt sql-server connects to a branch-qualified database (e.g. <database>/<branch>) either through a connection or use statement, it tracks metadata for that database+branch and shows the entry as a database in show databases. If that branch is deleted or renamed, the metadata isn't cleaned up and can cause the server to stop working correctly in any query that causes all databases to be searched (e.g. using information_schema, calling a stored procedure, calling `show procedure status).
  This PR updates the branch delete/rename logic to clean up this database metadata and also fixes the safegaurd for deleting/renaming a branch in use by another session, since it didn't take revision databases into account previously.
  Fixes: #3636
  Depends on: dolthub/vitess#170
• 3665: Fix cherry-pick flaky test
  This PR fixes cherry-pick: commit with ALTER TABLE rename table name flaky bats test.
  doltdb.UnionTableNames() method returns inconsistent/randomized ordered array of table names. Renaming table is not tracked when merging, so that it becomes the old name table being dropping and the new table being added. If the tableNames array has old table name first, then the error case is caught, but if the tableNames array has the new table name first, then MergeRoots try to add the exact same table, which returns error of same column tags.
  To avoid this, doltdb.UnionTableNames() method now returns unique table names present in order of RootValues passed in.
• 3654: Update dolt merge unique key violation behavior. Implement unique key violations for new format.

  Both format changes

  CALL DOLT_MERGE conflicts column returns 1 if the post-merge root contains conflicts or violations:
  Previously, the conflicts column would only return 1 if the merge produced new conflicts. Now it returns 1 regardless if whether the conflicts in the root are new or from a previous merge.
  It also returns 1 if there are any constraint violations.

  Current format changes

  1. Unique key constraints during merge are documented for both the existing row and violating row:
  Sometimes a dolt merge or the stored-procedure equivalent CALL DOLT_MERGE may produce unique key violations even when those violations did not exist in either the target or source branches.
  Consider the following:

  -- in the ancestor
  CREATE TABLE t (pk int PRIMARY KEY, col1 int UNIQUE);
  INSERT INTO t VALUES (1, 1);
  -- on the right, this is valid
  INSERT INTO t VALUES (2, 2);
  /*
  Query OK, 1 row affected
  */
  -- on the left, this is valid
  UPDATE t set col1 = 2 where pk = 1;
  /*
  Query OK, 1 row affected
  Rows matched: 1  Changed: 1  Warnings: 0
  */
  -- in the post merge state col1 has the duplicate value 2
  SELECT * from t;
  /*
  +----+------+
  | pk | col1 |
  +----+------+
  | 1  | 2    |
  | 2  | 2    |
  +----+------+
  */

  Prior to this PR only PK 2 would be returned as a constraint violations:

  SELECT * from dolt_constraint_violations_t;
  /*
  +----------------+----+------+---------------------------------------+
  | violation_type | pk | col1 | violation_info                        |
  +----------------+----+------+---------------------------------------+
  | unique index   | 2  | 2    | {"Columns": ["col1"], "Name": "col1"} |
  +----------------+----+------+---------------------------------------+
  */

  After this PR both PK 1 and 2 are returned as constraint violations:

  SELECT * from dolt_constriant_violations_t;
  /*
  +----------------+----+------+---------------------------------------+
  | violation_type | pk | col1 | violation_info                        |
  +----------------+----+------+---------------------------------------+
  | unique index   | 1  | 2    | {"Columns": ["col1"], "Name": "col1"} |
  | unique index   | 2  | 2    | {"Columns": ["col1"], "Name": "col1"} |
  +----------------+----+------+---------------------------------------+
  */

  2. Rows that violate a unique key constraint during merge are no longer filtered. They are allowed to persist into the left:
  In the above example, prior to this PR we would not see PK 2 in our post-merge state:

  -- post-merge state
  SELECT * from t;
  /*
  +----+------+
  | pk | col1 |
  +----+------+
  | 1  | 2    |
  +----+------+
  */

  After this PR, we do see PK 2:

  -- post-merge state
  SELECT * from t;
  /*
  +----+------+
  | pk | col1 |
  +----+------+
  | 1  | 2    |
  | 2  | 2    |
  +----+------+
  */

  New format changes

  The new format matches the current format behavior except for the following.
  1. Merge errors if a single row violates multiple UK or FK violations:
  The PKs of the artifact table are: PKs of the source table, a root-ish hash, and then the artifact type. If a row violates multiple UK or FK constraints in the same merge, there is a PK key collision and the previous CV entry is overwritten with the new one. In the future, we can store both constraint violations in the json metadata.
  This is a problem in the current format as well but we don't produce errors.
  2. If the ancestor has duplicate values for a column and the right de-dupes them and adds a UK constraint, the merge will not abort:
  In the current format this case aborts the merge because on the left we have constraint violations. If the left is missing a unique index, it is built prior to the merge. During this process if a duplicate value is found the entire merge is aborted.
  In the new format, if a unique index is missing in either the left, right, or ancestor we build the merged index with the post-merge primary row data. If we encounter duplicate values here we document them instead of aborting.
  Due to the way the current format updates indexes, even if we documented the constraint violations in the left we would not achieve consistent behavior. For example, we would document the violations that the right might have already resolved.
• 3642: Update Describe tests in Dolt due to bug in GMS
  This pr modifies bats and enginetests due to some missing functionality in GMS.
• 3365: TableStatistics implementation for dolt side
  Implements the sql.StatisticsTable interface

go-mysql-server

• 1073: Fixed returning null type instead of null value
  A few places, such as NULLIF(), were returning the null type (sql.Null) when they should have been returning a null value (nil in Go). This has been fixed.
• 1065: Support describe for views
  This pr:
  1. Introduces a new analyze rules to ensure describe works on views
  2. Fixes a bug in the return of SHOW COLUMNS
  3. Makes sure describe works with expressions
• 1061: blob enginetests
• 998: Column Statistics
  Gathers the following column stats:
  • mean
  • min
  • max
  • count
  • null count
  • distinct count
    and stores them in a modified information_schema.column_statistics table.
    There is now a new sql.Histogram struct that contains the column stats; the contents and format of sql.Histogram come from a mixture of Cockroach and MySQL.
    Additionally, there is also a TableStatistics interface, which is meant to be implemented if people want statistics on that table.

vitess

• 170: Adding support for use db/branch syntax
  Currently, use <database>/<branch> works only from the mysql client, because it parses the use command on the client side and sends it to the server as a different command (not a query). This means this syntax does not work with dolt sql or with other clients that don't parse this command client side and translate it (e.g. go-sql-driver/mysql).
  This PR adds support to our parser for this syntax, so that use <database>/<branch> can be used consistently.
• 169: Adding support for specifying a primary key name
  MySQL allows you to specify an optional name following a primary key definition when creating or altering a table. This name is always ignored though – all primary keys in MySQL are always named PRIMARY.
  Fixes: #3674
• 168: Remove glog dependency, reduce log levels, remove all logutilpb usage and support.

Closed Issues

• 3636: Broken doltdb after deletion of the branch which had DB alias
• 3599: Have 'dolt status' also show how local repo compares with remote
• 3649: dolt sql --result-format csv returns empty values for bit columns
• 3207: ORDER BY with several primary keys on a very large table takes too long ( almost does not load )
• 3143: Dolt table import OOM panic
• 2978: Bad error message when using unsupported FULLTEXT KEY
• 1104: SQL Alchemy cannot parse CREATE TABLE statement with foreign key constraint
• 3257: Panic with querying geometry types
• 3258: MySQL database is only available in sql-server mode
• 3658: NULLIF doesn't work properly with LONGTEXT
• 787: Add support for describing views