Merged PRs

dolt

• 5350: Dolt fixes for empty column projections
• 5348: fix --track for dolt_checkout
  --track is not working with call dolt_checkout as expected. This change updates dolt_checkout to mirror the before of dolt checkout in the CLI.
  fixes: #5189
• 5340: go/store/nbs: Optimize memory and disk utilization during GC.
• 5331: Add child refs to memtable only when a chunk was added.
  Fixes #5301
• 5330: Fixed race when multiple clients attempt to connect to the same branc…
  …h the first time a replica fetches it
• 5326: fixes timing for multi statement queries
  This change fixes timing for multi statement queries to print the time to execute the query instead of the time to print the results or the time since the first query started.
• 5320: store/datas: fix dropped errors
  This fixes three dropped err variables in the store/datas package.
• 5319: don't create new index if matching on primary key
• 5315: slightly better prefix compare
  A slight performance improvement for findPrefix when looking through table indexes that avoids a make() call and bytes.Compare
• 5312: add --track flag for dolt_branch() procedure
  It is equivalent as using --track flag of CLI dolt branch command.
• 5311: makes -f default behavior for dolt fetch
  This change makes force fetch the default behavior for dolt fetch and removes the -f force flag to prevent errors when fetching after a force push.
  fixes: #4943
• 5308: Allow the pprof server to handle connections from non-localhost
  Allow the pprof server to handle connections from non-localhost machines.
  This change is necessary to reach a Docker pprof server from host machine.
• 5306: dolt_remote_branches table
  New system table shows remote branches with same schema as existing dolt_branches table
• 5302: fix panic for unexpected json file format
  returns error message instead of panic
• 5295: Trim compare string prefix
  this improves a low-selectivity index scan for a customer query.
• 5294: add bats test for binary string encoded to utf8
  Depends on dolthub/go-mysql-server#1583
  DataGrip sets the character_set_results system variable to utf8, which raw binary strings cannot be encoded into. MySQL returns binary type strings in hex format.
• 5279: reuse cursor when verifying fks using secondary indexes
• 5252: Fix int overflow on 32-bit architectures

  $  GOARCH=386 GOOS=linux go build ./...
  # github.com/dolthub/dolt/go/store/skip
  store/skip/list.go:152:21: maxCount (untyped int constant 4294967294) overflows int

  See also dolthub/vitess#216
• 5220: Dolt Binlog Replication
  Initial support for Dolt binlog replication from a MySQL server.
  Includes support for:
  • Configuring a replica's source – CHANGE REPLICATION SOURCE TO ...
  • Configuring optional replication filtering – CHANGE REPLICATION FILTER ...
  • Showing a replica's status – SHOW REPLICA STATUS;
  • Starting / stopping replication – START REPLICA;, STOP REPLICA;
  • Reseting a replica – RESET REPLICA [ALL];
    Notable limitations/restrictions:
  • Only the default ("") replication channel is supported
  • Requires row-based replication with GTID-based auto-positioning is supported
  • A subset of filtering options are present (i.e. REPLICATE_DO_TABLE, REPLICATE_IGNORE_TABLE); more can be added upon customer request
  • Replication privileges are not checked yet; I will follow up with this shortly
    Testing:
  • Integration tests currently live alongside the code, but since they take ~70s to run all of them, I'm thinking about moving them to the integration-tests directory and setting up a separate GitHub actions job to run them. It is convenient that they currently get run with all the storage format variants though.
    Documentation:
  • dolthub/docs#1177
• 5126: Bump knex from 1.0.7 to 2.4.0 in /integration-tests/mysql-client-tests/node
  Bumps knex from 1.0.7 to 2.4.0.
  Release notes

  Sourced from knex's releases.

  2.4.0

  New features:

  • Support partial unique indexes #5316
  • Make compiling SQL in error message optional #5282

  Bug fixes

  • Insert array into json column #5321
  • Fix unexpected max acquire-timeout #5377
  • Fix: orWhereJson #5361
  • MySQL: Add assertion for basic where clause not to be object or array #1227
  • SQLite: Fix changing the default value of a boolean column in SQLite #5319

  Typings:

  • add missing type for 'expirationChecker' on PgConnectionConfig #5334

  2.3.0

  New features:

  • PostgreSQL: Explicit jsonb support for custom pg clients #5201
  • SQLite: Support returning with sqlite3 and better-sqlite3 #5285
  • MSSQL: Implement mapBinding mssql dialect option #5292

  Typings:

  • Update types for TS 4.8 #5279
  • Fix typo #5267
  • Fix WhereJsonObject withCompositeTableType #5306
  • Fix AnalyticFunction type #5304
  • Infer specific column value type in aggregations #5297

  2.2.0

  New features:

  • Inline primary key creation for postgres flavours #5233
  • SQLite: Add warning for undefined connection file #5223
  • MSSQL: Add JSON parameter support for connection #5200

  Bug fixes:

  • PostgreSQL: add primaryKey option for uuid #5212

  Typings:

  • Add promisable and better types #5222
  • Update raw query bind parameter type #5208

  2.1.0 - 26 May, 2022

  ... (truncated)

  Changelog

  Sourced from knex's changelog.

  2.4.0 - 06 January, 2022

  New features:

  • Support partial unique indexes #5316
  • Make compiling SQL in error message optional #5282

  Bug fixes

  • Insert array into json column #5321
  • Fix unexpected max acquire-timeout #5377
  • Fix: orWhereJson #5361
  • MySQL: Add assertion for basic where clause not to be object or array #1227
  • SQLite: Fix changing the default value of a boolean column in SQLite #5319

  Typings:

  • add missing type for 'expirationChecker' on PgConnectionConfig #5334

  2.3.0 - 31 August, 2022

  New features:

  • PostgreSQL: Explicit jsonb support for custom pg clients #5201
  • SQLite: Support returning with sqlite3 and better-sqlite3 #5285
  • MSSQL: Implement mapBinding mssql dialect option #5292

  Typings:

  • Update types for TS 4.8 #5279
  • Fix typo #5267
  • Fix WhereJsonObject withCompositeTableType #5306
  • Fix AnalyticFunction type #5304
  • Infer specific column value type in aggregations #5297

  2.2.0 - 19 July, 2022

  New features:

  • Inline primary key creation for postgres flavours #5233
  • SQLite: Add warning for undefined connection file #5223
  • MSSQL: Add JSON parameter support for connection #5200

  Bug fixes:

  • PostgreSQL: add primaryKey option for uuid #5212

  Typings:

  • Add promisable and better types #5222

  ... (truncated)

  Commits

  • 3475d81 Prepare to release 2.4.0
  • e97f922 Bump tsd from 0.24.1 to 0.25.0 (#5396)
  • e145322 1227: add assertion for basic where clause values (#5417)
  • 962bb0a Bump sinon from 14.0.2 to 15.0.1 (#5413)
  • ab45314 Add JSDoc (TS Flavour) to mjs stub file (#5390)
  • 72bd1f7 Fix: orWhereJson (#5361)
  • 4fc939a Fixes unexpected max acquire-timeout (#5377)
  • 5c4837c Fix lib/.gitignore path separator on Windows. (#5325)
  • 7dbbd00 Bump actions/setup-node from 3.4.1 to 3.5.1 (#5356)
  • d39051f fix: add missing type for 'expirationChecker' on PgConnectionConfig (#5334)
  • Additional commits viewable in compare view
  
  [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
  Dependabot commands and options

  
  You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/dolt/network/alerts).

go-mysql-server

• 1592: name unnamed values properly
  Fixes #1591
  Signed-off-by: Dave Henderson dhenderson@gmail.com
• 1588: partially validate and create underlying indexes even if foreign_key_checks=0
  When FOREIGN_KEY_CHECKS is disabled, we should still create the underlying secondary index or assign the primary key indexes.
  Currently, we don't do that, but the moment any query is run with FOREIGN_KEY_CHECKS enabled, we create the indexes and assign any index references.
  Additionally, we need to do some of the validation according to MySQL Docs
  Related: #5316
• 1587: Changed how empty set of projected columns is handled
  Started with tightening the semantics of the ProjectedTable interface as it relates to an empty projection (nil v. empty slice). Then made changes to printing of ResolvedTable and IndexedTableAccess. This revealed problems in the prune columns rule when all columns were being pruned. Fixed those, which had been masking other bugs, where we hadn't been pruning when we could have been. This was in turn caused by other bugs in the prune rule dealing with case sensitivity.
  We should now be able to prune more columns than before, and we can see an empty column projection in plans.
• 1586: Unnest IN_SUBQUERY GROUP_BY into partial join
  Resolving group by expressions used to happen in two phases, and now happens in one so that we can detect cacheable groupby subqueries at applyJoin time.
  Fix a miscellaneous bug in RightSemiLookupJoin. The right left node has to return distinct results, otherwise we will duplicate results.
  Dolt tests pass locally.
• 1584: not perform optimization_rule on like clause for utf8mb3 charset
  optimization rule uses ÿ last character in the charset utf8mb4, which does not work for utf8mb3 charset values.
• 1583: literal.String() returns hex string for binary type data
  DataGrip sets the character_set_results system variable to utf8, which raw binary strings cannot be encoded into. MySQL returns binary type strings in hex format.
• 1563: Fix unsafe table pointer access in applyJoin

vitess

• 219: Backporting client support for caching_sha2_password auth
  The MySQL client API in our fork of vitess is limited to the mysql_native_password auth plugin, but the default auth plugin as of MySQL 8.0 is caching_sha2_password. This means that to use Dolt binlog replication, customers either have to change the default auth plugin in MySQL or they have to create a user that is identified with the older mysql_native_password auth plugin. This additional configuration step adds friction for customers wanting to try out Dolt binlog replication.
  This PR pulls in selected changes from the tip of vitessio/vitess to pick up client support for connecting to MySQL servers that default to caching_sha2_password auth plugin (and will still fall back to mysql_native_password). All of these changes are pulled directly from vitessio/vitess, without any other changes mixed in.
  I'm not aware of anywhere else in our codebase where we're using the MySQL client API from vitess. As far as I know, it's only used by the binlog implementation. That means this change shouldn't have any effect on other parts of Dolt/GMS, and I've gone ahead and run the Dolt CI tests against this change as a sanity check:
  • #5339
    I've pulled over additional tests from the tip of vitess for this logic. I've also tested locally and confirmed that my binlog replication integ tests all pass with this change, and no longer require overriding the default auth plugin for the MySQL server. I've also started on some additional binlog replication integ tests to make sure we cover password/no-password and ssl/no-ssl.
• 217: Adding implicit string concatenation for a variable assignment needed for binlog replication
  When a mysqldump is taken for a database tracking GTIDs, it outputs GTID tracking information via a line like:

  SET @@GLOBAL.GTID_PURGED= /*!80000 '+'*/ 'beabe64c-9dc6-11ed-8021-a0f9021e8e70:1-126';

  In MySQL 8.0+, the MySQL comment inserts a '+' between the = and the assignment value, which gets implicitly concatenated with the adjacent quoted string in MySQL.
  GMS/Dolt doesn't support this implicit string concatenation, but I wanted to get this query to work – otherwise customers will have to manually edit their mysql dump files to remove this syntax before it can be imported into Dolt. I didn't think it was worth taking on the larger change right now to support implicit quoted string concatenation everywhere (#5232), so instead, I added special handling for this case since it's important for the binlog setup experience.

Closed Issues

• 5189: --track does not work in call dolt_checkout
• 4245: Tables with hyphen character cause mysql client to drop connection
• 3770: Support DELETE JOINs
• 1660: Server logging for inserts is deceptive
• 5344: Cloning a dolt repo onto a virtual machine
• 3637: USE db/branch works differently than call dolt_checkout('branch')
• 4503: Strange behavior with enum in composite unique key in a table without primary key
• 4957: Support call procedure as of <commit>
• 4853: Sporadic slowness in GCS read replica
• 5148: High memory usage and long start up time for a database with thousands of branches
• 5019: Feature request: Optimize initialization for large databases
• 3296: Incorrect result for some cases of 'explicit join tree syntax for joins'
• 4494: COUNT(*) select start impedance mismatch
• 4482: Support JSON extract operators
• 5301: Memory leak in latest dolt
• 5237: dolt panic: invalid memory address or nil pointer dereference
• 5287: EXPLAIN with uuid_to_bin is not working in IntelliJ IDEA (like DataGrip)
• 5143: Attempting to Import Nested JSON Objects to Dolt can Cause Dolt to Crash
• 5304: Need a system table to view all branches equivalent to dolt branch -va
• 1592: dolt_branches table should contain remotes on clone
• 1590: dolt checkout to most recently updated branch
• 384: Dolt login opens browser for incorrect account
• 935: Msi installer add option to install to user folder
• 1300: dolt gc breaks sql-server inserts until restarted
• 4943: Make -f the default behavior for dolt fetch
• 1591: unbound variable "v1" in query

Latency