Merged PRs

dolt

• 4285: trim byte order mark present in UTF-8 encoded files
  Closes: #3790
• 4284: add nil check for string pointers during dolt table import
  Closes: #4209
• 4283: fix enum panics in history table
  Closes: #4279
  sql.enumtype contains a map, which in golang cannot be used in == operator
• 4278: Escape columns and table names in conflict commands
  Fixes #4274
• 4277: Adds dolt_merge_status system table
  Closes #3504
  The dolt_merge_status system table tells a user if a merge is active. It has the following schema:

  CREATE TABLE `dolt_merge_status` (
  -- Whether a merge is currently active or not
  `is_merging` tinyint NOT NULL,
  -- The commit spec that was used to initiate the merge
  `source` text,
  -- The commit that the commit spec resolved to at the time of merge
  `source_commit` text,
  -- The target destination working set
  `target` text,
  -- A list of tables that have conflicts or constraint violations
  `unmerged_tables` text
  )

  For example, lets create a simple conflict:

  dolt sql -q "CREATE TABLE t (a INT PRIMARY KEY, b INT);"
  dolt add .
  dolt commit -am "base"
  dolt checkout -b right
  dolt sql <<SQL
  ALTER TABLE t ADD c INT;
  INSERT INTO t VALUES (1, 2, 1);
  SQL
  dolt commit -am "right"
  dolt checkout main
  dolt sql -q "INSERT INTO t values (1, 3);"
  dolt commit -am "left"
  dolt merge right

  Output of SELECT * from dolt_merge_status;:

  +------------+--------+----------------------------------+-----------------+-----------------+
  | is_merging | source | source_commit                    | target          | unmerged_tables |
  +------------+--------+----------------------------------+-----------------+-----------------+
  | true       | right  | fbghslue1k9cfgbi00ti4r8417frgbca | refs/heads/main | t               |
  +------------+--------+----------------------------------+-----------------+-----------------+
  

• 4276: Automatically push and pull new databases in replicated servers
• 4273: go/utils/remotesrv: Add a -read-only flag for exposing a remote endpoint that rejects write requests.
• 4271: rCTEs GMS bump
• 4264: go/utils/remotesrv: Add -repo-mode to run a remote server for the primary database in a dolt cli working directory.
• 4262: PartitionRows lookupBuilder access race
  del extra line
• 4259: [store/prolly] Add key range machinery
  Adds functions to support operating on a physical key range.
  Continuation of #4247 and #4258
• 4257: unskipping commit DiffSystemTableTests
• 4255: allow init with empty working set
  In sql session, if the current head on CLI is deleted, the repostate head is empty branch ref.
  If the current head on CLI is renamed, the repostate head is updated with new branch ref.
  dolt_branches table is read-only now.
• 4252: unskip tests for bindvars in filters for history tables and table functions
• 4251: go/utils/remotesrv: Improve memory safety and scalability of remotesrv.
  Fixes a race condition in the way the gRPC implementation stores data for the HTTP server to see on future requests.
  Fixes a memory utilization issue where the full contents of a written table file were read into memory before being persisted to disk.
• 4250: go/doltcore/migrate: Migrate oldgen manifest in dolt migrate
  fixes #4234
• 4243: use commitItr instead of commitwalk for querying dolt_diff table
  This improves select query on unscoped dolt_diff table in old format for commit_hash column filter
• 4242: unskipping tests for prepareds on dolt diff tables
  Unskipping:
  • TestDiffSystemTablePrepared
    Companion PR:
  • dolthub/go-mysql-server#1229
• 4210: Fix dolt version error when repo state is missing
  Fixes #4154
  Not sure if tests are required for this?
• 4179: moving prepared history table test out of broken
  Companion PR:
  • #4179

go-mysql-server

• 1240: Additional fixes to collation coercion
  Coercion rules specify that Unicode and non-Unicode characters may the Unicode collation applied in the event that they're being compared, concatenated, etc. I'm not exactly sure how to determine what a "Unicode" collation is, so I'm assuming that all collations that use more than one byte are Unicode. I can almost guarantee that this is incorrect, however it's a better approximation of behavior than not considering it at all.
• 1238: Added limited support for collation coercibility
  This adds a sort of "fix" for the first issue identified in dolthub/go-mysql-server#1232. This is basically a hack for collation coercibility, which is a significant undertaking. This should suffice in the meantime.
• 1231: Return errors instead of asserting inside runQueryPrepared
  Test PR:
  • #4257
• 1230: fix applying bindvars to filters and table functions
  Changes:
  • Allow bindvars to be applied to FilteredTables using a new DeferredFilteredTable node
  • No longer pushdown filters in post prepared analysis
    Test PR:
  • #4252
• 1229: fix filters being dropped from prepareds on dolt_diff tables
  TransformProjections was not transferring filters from certain prepared plans that did not contain projections.
  Fix for
  • #3916
    Tests:
  • #4242
• 1228: Resolve CTEs ignored recursive changes
  fixes regression where we do not propagate nested changes in cte resolve
• 1227: Fixed collation error message, added more collations
  Fixes #4236, in addition to adding two new collations:
  • utf8mb3_unicode_ci
  • utf8mb4_unicode_520_ci
• 1226: recursive ctes resolve limit filter sort
  General outline:
  1. Union nodes absorb LIMIT, ORDERBY, and DISTINCT logic
  2. Recursive CTEs embed a UNION node, and the analyzer treats them the same for the most part
  3. Rewrite resolve_cte to bridge the gap between WITH -> rCTE -> UNION nodes
     Re: (1), Union is basically collapsed into a single node. I think all scopes should probably look like this longterm for a few reasons. Adding an attribute grammar for resolving tables and columns becomes a lot easier. Transformation rules are simpler because they do not have to see through the tree within a scope. Logical nodes can be separate from physical nodes. Moving from a tree to a hash map of logical nodes is the main way of doing costed planning.
     Re: (2), the easiest way to support arbitrary left/right nodes in recursive ctes is to try to run the regular analyzer on them. Notably, MySQL prevents many types of top-level nodes in rCTEs, which we have not prevented yet. For example, GROUPYBY, ORDERBY, LIMIT are not permitted in the middle of an rCTE (I think because the results are ambiguous). We do not error on those right now, but maybe should in a follow-up PR.
     Re: (3), rCTEs are structurally parsed as Unions, and can be resolved like unions in most of the analyzer, but we need several manual steps to address the ways rCTEs are functionally different. rCTEs should be separated into recursive and non-recursive chains of UNIONs (refer to Aaron's comment). If a rCTE has no recursive portions, we convert to a regular CTE. We manually resolve references to the recursive binding, converting them from UnresolvedTable to RecursiveTable. There is a chicken and egg problem where we have to resolve the left-side of the union before we can construct and apply a schema'd RecursiveTable to the right.
• 1212: fix more prepared enginetests
  Moved these rules to run during preprepared, as the explain queries for history table were missing Exchange node
  • inSubqueryIndexesId
  • AutocommitId
  • TrackProcessId
  • parallelizeId
  • clearWarningsId
    Tests:
  • #4179
• 1202: moveJoinConditionsToFilter bug
  The move join condition rule selects filters not needed as a join condition, identify the target parent for the filter, and then moves the plucked filters to EVERY child of the parent of the join node. The correct behavior is to move the filter immediately above the join whose condition we plucked the filter from.

vitess

• 187: CTEs on unions parse
• 186: add implicit from dual to select x in recursive cte statements

Closed Issues

• 4209: panic on import
• 4274: Can't dolt conflict on a database with a column with a : in it
• 4125: Automatically replicate newly created databases
• 3504: dolt_merge_state system table
• 3790: Dolt table importing csv using UTF-8 with BOM encoding prepends BOM (U+FEFF) to first column's name
• 4279: dolt_history_<table> panics on enum types
• 4232: Optimize dolt_diff system table access when scoped to a commit
• 4248: Deleting the branch you have checked out on the CLI in sql-server mode makes database unusable
• 4234: oldgen and newgen chunkstore versions vary error after dolt migrate
• 4169: Django App Fails to Migrate with Global AUTO_INCREMENT Keys
• 3916: Prepared statement Issue with dolt_diff_<table>
• 4154: dolt version errors when run from directory containing DOLT_ROOT_PATH
• 4175: dolt panics on importing parquet file
• 4109: unexpected "table does not have column" error
• 3995: System Table Performance Investigation
• 3976: sql_mode is always strict
• 4211: Inserting large JSON object in a prepared statement causes panic
• 4007: Make Dolt work with Fivetran through its MySQL connector
• 3800: nodejs concurrency issue
• 3858: Connecting to a dolt sql-server with InnoDb MySQL client may fail if a table uses special characters like hyphen
• 3836: INSERT IGNORE INTO throws duplicate unique key given for keyless tables
• 3811: PHP PDO issue with float data type
• 3650: Dolt Diff across all tables
• 3566: Panic accessing information_schema when mixing old and new storage formats
• 4230: cte table not found when using limit
• 4222: Can't build dolt with LTO
• 3468: Dolt diff does not show diff inside JSON
• 3405: (sql-client) shortcut to navigate cursor to next word
• 3222: Newlines in data exported from sql query
• 3198: [REQUEST] Option to output sql generated by dolt diff
• 3150: Dolt may fail to push on slow internet connections
• 1219: How can i change the log level?