Merged PRs

dolt

• 5818: Transaction isolation for branch-qualified databases
  As well as substantially address the issues about concurrent access to dolt_branches called out in #5784
  As a consequence, the dolt_branch procedure now implicitly commits any current transaction.

• 5815: Prevent merges when either side has multiple indexes covering the same set of columns
  When merging multiple indexes that cover the same set of columns (even if they are different types, such as a unique index and a prefix index), there is a race condition because of how our index support assumes the column tag set will be unique for every index. This can cause the wrong index to be matched and the wrong merge decisions to be made.
  I tried to come up with another way to opportunistically match indexes across the sides of a merge in PR #5797, but I still found more edge cases and the extra complexity didn't seem worth it. It seemed better to have Dolt be vocal about the issue and let users resolve it, instead of trying to be clever when we can't merge that situation correctly every time.

• 5814: go/libraries/doltcore/env/multi_repo_env.go: Always use the GRPCDialProvider from the bootstrap env for the MultiRepoEnv RemoteDialProvider.
  Fixes #5811.

• 5806: Report schema conflicts for merges with column type changes
  As a first step towards being able to auto-resolve column type changes during a merge, I added a check for type compatibility (which always returns incompatible for DOLT storage format, unless the TypeInfos are exactly the same). This also results in schema conflict metadata being reported.
  I also added some tests for a few classes of type changes and reorganized the existing tests a bit.

• 5803: Fix dolt creds check with specified endpoint
  Using dolt creds check --endpoint <endpoint> --creds <creds> was failing with an audience error because we were using the wrong parsing logic to get the host for the endpoint. This fixes that and adds a log if the endpoint cannot be parsed

  % dolt creds check --endpoint doltremoteapi.awsdev.ld-corp.com:443 --creds b33diq4ht3t73hnp5bofekvsg9lk36ou2gto74f7ot02g38r70mg
  Calling...
  Endpoint: doltremoteapi.awsdev.ld-corp.com:443
  Key: b33diq4ht3t73hnp5bofekvsg9lk36ou2gto74f7ot02g38r70mg
  error: calling doltremoteapi with credentials.
  cause: rpc error: code = Unauthenticated desc = jwt_token validation failed: square/go-jose/jwt: validation failed, invalid audience claim (aud)
  

• 5800: go/doltcore/{sqle, merge}: Update sqlutil.ParseCreateTable to fix test gaps in schema merge suite

• 5798: --data-dir flag enabled before the sql subcommand
  The complexity of creating global arguments is being broken into multiple pieces. This step includes the breaking up of the top level dolt command set to enable global argument parsing and initialization of the CliContext. At present, the sql subcommand is the only one getting this treatment, but it will be extended with additional commands in the future.
  This change doesn't change much that is visible to the user, with the exception of the --data-dir flag being enabled before the sql subcommand.
  Step 4 of N for: #3922

• 5795: Correct busted interface change for CliContext

  TIL that interface pointers in golang are not useful. This changes that, and add in a couple calls in command.go which were neglected in the first refactor.
  Step 3 of N: #3922

• 5790: Bug fix and test for index merge bug that didn't detect incompatible index types
  The equality functions on Index weren't considering prefixLengths or isSpatial, which resulted in a merge bug where dolt would get confused and match two indexes as identical when they really weren't.

• 5788: Adding basic support for conflict validation in new schema merge test framework
  This also pointed out an issue in the framework where any secondary index and check constraint definitions are silently swallowed.

• 5787: Run schema merge tests in both directions
  Running the ThreeWayMergeWithSchemaChangeTestScripts tests in both directions has been pretty helpful to catch a few bugs, so I took a stab at making testSchemaMerge run merge tests in both directions.
  There shouldn't be any changes to the test cases, but it does help remove duplication in the test cases so that they are a little easier to maintain.
  (I see the Check Formatting CI check is failing here, too... I'll push another commit to reformat those and see if that makes it happy.)

• 5783: dolt show supports displaying non-commit objects.
  PR is still a draft: it needs additional documentation and tests.
  dolt show now accepts a hash of any object in the datastore, not just commits. It will also accept the special strings WORKING and STAGED.
  Some object types (currently just commits) have two representations: a "pretty" format which is more human readable, and a "non-pretty" format which displayed as JSON-like structured data. dolt show prefers the pretty format when possible, but this can be suppressed with --no-pretty.
  Further work: add support for --raw, which outputs the raw binary object. (noms show supports this but I don't know how much use it gets.)

• 5779: Add the CliContext parameter to Command.Exec
  Empty interface passed as last argument to Exec. Refactor by goland.
  NoOp change.

• 5777: go.mod: Bump go-mysql-server, migrate to github.com/dolthub/flatbuffers/v23/go.

• 5773: Fixed a bug where foreign key changes in the working set weren't carried forward during dolt checkout
  Fixes #1310

• 5772: Schema conflict detection edge cases
  #2973 pointed out an edge case where a schema conflict wasn't getting reported. That uncovered some other edge cases. Adding those the current code was feeling messy, so I refactored our column merge logic a bit to try and better organize it. We now have mostly separate steps for: 1) calculating the mapping between left/right/anc columns, 2) detecting conflicts based on that mapping, and 3) iterating through the mapping and deciding which column to accept in the merge.
  I'm still not super happy with the conflictKind and the error messages that describe the schema conflicts, but I thought it would make more sense to see the schema merge conflict resolution UX you're working on and then clean those up in a separate pass.
  Fixes: #2973

• 5771: Follow-up to validating extra command line arguments.
  A follow-up to #5762 addressing comments in the original PR.

• 5768: Add Server Runtime details to the sql-server.lock file
  Add runtime information for the server to the sql-server.lock file. We track three pieces of information:

  1. The PID. As before.
  2. The port of the server, if there is one. You can grab the lock as a cli process, which has no port.
  3. A secret. Currently implemented as a UUID. This will be used to ensure that the connecting through the server is only performed by user's and processes which have read access to the server.
     In addition, this change modifies the sql-server.lock file to be only readable by the owner.
     Step (1) of: #3922.

• 5767: go/doltcore/{doltdb, sqle}: dolt_schema_conflicts system table
  …m table

• 5764: Update setup instructions for bats tests
  Bats tests also require that remotesrv be installed.
  I added instructions for that to the README.

• 5762: Add error when commands have extra arguments.
  Rolling this "Add error when commands have extra arguments." again.
  dolt sql-client inherits its arg parser from dolt sql-server, which didn't play nice with the original PR's restriction on sql-server's positional arguments. I relaxed the arg parser and added an extra verification check into sql-server itself.

• 5761: Revert "Add error when commands have extra arguments."
  Reverts #5753
  I mistakenly thought that a failing test was flaky. Time to revert the change, investigate, and make sure I don't do anything like this again.

• 5760: Fix error message in remote command

• 5753: Add error when commands have extra arguments.
  Multiple commands would previously silently ignore extra positional arguments. This happened most frequently with commands that aren't supposed to take any positional arguments, but also occurred with some others, most notably merge, which caused #5729.
  This fixes that by requiring users of ArgParser to specify how many positional arguments they expect by calling either NewArgParserWithMaxArguments(int) or NewArgParserWithVariableArguments(). The error message can be customized by assigning to ArgParser.TooManyArgsError. The goal is that by requiring developers to be explicit about this, it will help avoid similar issues in the future.
  This adds BATS tests for dolt merge in order to test that the new functionality works. I'm not going to add tests for every other command because there's like 80, but new commands should probably have a test for too many positional args.

• 5732: Fixed bug in dolt checkout with uncommitted changes on another branch
  Fixes #5725

• 5655: Add BATS test that dolt log correctly tags HEAD commits.
  Regression test for #5558

• 5626: Schema merge support for adding/dropping/renaming/reordering columns
  First pass at getting schema merge working for column adds/drops/renames/reorders. Includes support for:

  • migrating existing data in a primary index to the merged schema
  • mapping changes from the right side, to the merged schema and merging them into primary and secondary indexes, as well as conflict and constraint violation detection.

go-mysql-server

• 1731: sql/parse: expose utility functions to convert parsed index and check constraint definitions
• 1728: hashjoin indexing
  When we have plan of the following format:

  InSubquery
  ...
  CrossJoin
  Left: Table
  Right: SubqueryAlias
  OuterScopeVisibility: true
  ...
  HashJoin
  HashLookup
  source: ...
  target: TableAlias
  ...
  

  The indexes we assign to GetFields during analysis don't align with the indexes of the actual columns in each row during execution time. This is a result of StripNode, PrependNode, and the nested Joins with SubqueryAlias.
  This error wasn't caught sooner as the incorrect indexes are too low, so they never threw IndexOutOfBounds errors and just returned potentially incorrect results instead.
  The fix was to correct these indexes at analysis time.
  Firstly, SubqueryAlias nodes with OuterScopeVisibilty = true inside joins need to see the left sibling node (in addition to the parent nodes). So Scope was modified to include some new fields, specifically for sibling nodes. Additionally, the file finalizeSubquery was changed to track the parent as well, so we could detect when we're analyzing a SubqueryAlias on the right side of a join, and add the left child to the scope.
  Additionally, pushdownFilters was modified to not undo all the changes to the Analyzer for HashLookups.
  At runtime, the PrependRow nodes cache the rows outside the InSubquery, while the buildJoinIter for CrossJoin would include both the outside and the left row. This meant that some parts of the inner HashJoin would receive extra columns while others didn't. The fix here was to alter the scope.InJoin depending on which parts of HashJoin we were building.
  Lastly, to have these changes not break for PreparedStatements, we just needed to not redo finalizeUnions in postPrepared, as we don't replan joins in postPrepared, so we don't know if we're in a join or not, and the correct indexes are set in prePrepared.
  Along the way, we discovered a query that panics, but the cause is different than the purpose of this fix, and it panicked the same way before these changes, so it is left as a skipped test.
  Fix for: #5714
• 1727: implement json_set function
  implements the json_set function with a few edge cases outstanding
  fixes: #5680
• 1726: subquery indexing tests
• 1724: Fix README.md
  Replace spaces with tabs in code indentation.
• 1722: go.mod: Use dolthub/flatbuffers/v23 instead of google/flatbuffers.
• 1721: go.mod: Move oliveagle/jsonpath -> dolthub/jsonpath.
• 1720: memory: extract rangeFilterExpr into expression package
  rangeFilterExpr contains a complex set of logic to build a sql expression given the list of sql ranges and the list of expressions on the index.
  Extract the majority of this function into NewRangeFilterExpr in the expression package. Replace the Or and And helper functions with JoinOr and JoinAnd.
  Update the call in the memory/ package to call the new expression function.
  Fix JoinOr and JoinAnd to check if the expressions are nil.
  Remove ineffective nil check in JoinAnd.
  Move nil checks into NewOr and NewAnd.
  Add comment explaining parameters to NewFilterRangeExpr.
• 1719: Added serving tray and bowtie
• 1717: Added a mascot image
• 1715: implement find_in_set
  MySQL docs: https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_find-in-set
  Note: Collations priority isn't exactly the same as MySQL
  fix for: #5659
• 1713: Changing MaxTextResponseByteLength() to respect character_set_results
  The Type.MaxTextResponseByteLength() function was not honoring the character_set_results system variable, which resulted in Dolt sending different response length metadata from MySQL when the value of character_set_results is not the same as the type's charset.
  This change adds a *sql.Context parameter to the MaxTextResponseByteLength(ctx) function, so that the correct value can be pulled out of the session when the response metadata is sent through the handler.
  Fixes: #5735
  Related Dolt change: #5752

Closed Issues

• 5714: dolt results not equal to MySQL results with multiple nested CTEs
• 5793: Improve error messaging when multiple processes connect to db concurrently.
• 5811: Using the dolt_clone sql procedure to clone a private database has weird behavior
• 3820: N-gram databases for other-language Wikipedias
• 5549: Getting "out of range for tinyint" error from dolt
• 5680: Add JSON_SET support
• 4547: Dolt says constraint violation on merge when one does not exist
• 5683: Unexpected NULL return values with false filter or empty set
• 2973: Merging Modified and Deleted Column should be a Conflict
• 5723: "Format PR" CI Workflow fails on PRs that come from forks.
• 1310: Foreign Keys don't travel with working set on dolt checkout
• 5729: call dolt_merge('branch1', 'branch2') should either merge branch1 and branch2 or fail not silently ignore the branch2 argument.
• 5725: Changes on multiple branches from sql-server has bad interaction with the CLI
• 5558: dolt log incorrectly identifies HEAD
• 5758: Typo in error msg
• 5755: Getting Started Docs sql-client missing user parameter
• 5735: Pasting into Datagrip TEXT field puts null
• 1533: convert_tz is not working as expected with @@GLOBAL.time_zone