github dolthub/dolt v0.34.9
0.34.9
on GitHub

latest releases: v1.43.13, v1.43.12, v1.43.11...
2 years ago

Merged PRs

dolt

  • 2507: Bump GMS and Vitess
  • 2504: /go/libraries/doltcore/doltdb: Deleted key itr
  • 2503: Clone sets upstream for main
  • 2500: /go/libraries/doltcore/table/noms: Removed Noms Map Creator and Updater
  • 2497: /go/libraries/doltcore/doltdb: Extracted Persistent Object from doltdb.Table
    This refactor is the first step in supporting a new storage format. The primary focus of this change is to modify doltdb.Table to extract its backing types.Struct.
    Throughout the doltdb package, in-memory objects are coupled with the persistent types objects that are used to de/serialize them to/from disk. The intention of this refactor and subsequent work is to decouple storage details from persistent objects. For example, schema.Schema is an in-memory abstraction that has a separate utility in the encoding package to read/write schemas to/from disk. This has created a clean separation of concerns for schema.Schema, and makes it easy to swap out serialization formats.
  • 2495: Refactor the Export path to read from the engine
    This pr removes the need of TableDataLoc from export and moves it towards an engine based table reader. It also modifies the writer implementations to work with sql.Row
    TODO: I don't think export should support either --continue or --mapping. They were never tested nor are they relevant. If there is an error reading a row from a table, you should fail hard.
  • 2494: bats test for failed window agg query
    bats test and GMS bump for #2448
    dolthub/go-mysql-server#680
  • 2490: Fix auto fwt transform to not drop a row on flush
  • 2488: Fixed bug with incorrectly applied index lookups for multi column indexes
  • 2486: Improve 'Data Mover' code on import path
  • 2485: Skipped test for Index overuse issue
    Added skipped bats test for the issue where an index is used inappropriately when a function modifies the column producing incorrect results.
  • 2482: add pager to dolt log
    dolt log output is paginated
  • 2472: Modified bats for SQL checkout to assert that the branch should not change on the CLI.
    SQL Checkout should not change the branch in command line context. It only changes the branch for the session which ends after the statement. Resolves #2468.
  • 2457: Add import benchmarks and add in import performance fix
    Two noticeable problems.
    1. No longer using BulkImportTableEditor. Needs to be threaded through in case the batch mode is set in the session.
    2. Performance overhead from DoltRow to SqlRow conversion throughout the insert process. This should be reduced in the future as row.Row is deprecated
      Current results 
    generating test file
Running the current version of import
Successfully initialized dolt data repository.
Rows Processed: 1000000, Additions: 1000000, Modifications: 0, Had No Effect: 0
Import completed successfully.
real	0m47.759s
user	1m16.795s
sys	0m25.848s
Running version 0.34.5
Rows Processed: 1000000, Additions: 1000000, Modifications: 0, Had No Effect: 0
Import completed successfully.
real	0m37.736s
user	1m12.005s
sys	0m30.281s
  • 2369: Async push for replication
    Async hook overview:
    • Calling the commit hook (Execute) pushes a (dataset, database) tuple into a channel
    • go routine 1 is responsible for reading from this channel, and setting the dataset id -> pull args in a shared map. Multiple commits to the same dataset override old values in this map.
    • goroutine 2 has two phases that are run iteratively. Phase 1 copies the current state of the commit map into a local value. Phase 2 iterates through daatsets in the local commit map, pushing the most recent dataset commit if its hash is not the deduplication map's value. The deduplication map is private to gouroutine 2, and saved the pushed commit hash for each dataset.
      lifecycle management:
    • new engine interface maintains BackgroundThreads, finalizing when we are done with the engine
      Initial perf comparison:
    • PushOnWrite: 7.9 sec / op
    • AsyncPushOnWrite: 6.5 sec / op
    • diff ~ 20% speedup
      The perf comparison depends on many factors (the transaction throughput, commit size, network latency, etc), but this isolated test supports the idea that amortizing commits is better than individually pushing every commit. In practice, the time/memory overhead of coordinating the amortization with threads should always be smaller that the round trip of pushing every individual dataset.
      edge cases:
    • commits to a dataset move backwards in time or delete values. The older commits contains chunks that fail to push (? but push synchronizes ancestors commits? so this should be OK unless we reset hard?)
    • every commit creates a unique dataset id, and the amortization map blows up

go-mysql-server

  • 689: Added test utility method for IndexTableAccess
  • 684: sql/plan: load_data: Move LoadInfile logic from server/handler to the plan node.
    This puts the LoadInfile functionality as a Service on the sql.Context and has
    the plan node call into it. This is a better place for it in general, rather
    than inspecting the resulting plan tree in server/handler and calling adhoc
    methods on the *mysql.Conn based on what we find.
    Also addresses a number of (potential) issues with the existing load data local
    functionality including:
    • Race-y file writes and reads from a single file (.LOADDATA) for concurrent
      requests.
    • Incorrect handling of scanner.Err() checking; we now correctly return the
      errors when they occur, instead of deferring them to the Close() call.
    • Avoids sending LoadInfilePacket for a non-LOCAL LOAD DATA call.
      The most complected part of the new implementation is the ordering constraint
      around RowUpdateAccumulator iterator calling Close() on the row source before
      it returns an Ok result from the Next() call. An alternative approach would be
      to copy the contents to a temporary file in LoadData.RowIter().
  • 683: Changed Range overlap algorithm
    Implemented a tree structure for ranges based on https://en.wikipedia.org/wiki/Interval_tree#Augmented_tree. I tested this against a benchmark that replicated the slowdown seen in Dolt release 0.34.5 and saw linear scaling, just as we'd hope. There are some cases that are handled very inefficiently, and I left some strategies on how to solve them listed in the comments, which I will probably implement at a later time.
  • 681: Return Uint64 instead of int64 in charsets table
  • 680: Window aggregation buffer sharing test
    Re: #2448
    Dolt diff table aggregation queries were bleeding results. We still use buffer sharing, but this test should prevent regressions for the same issue.
  • 679: server/handler.go: Migrate doQuery rows loop to errgroup.
    Previously this used a |quit| signaling channel and an |errChan| to
    communicate any errors. We didn't get context cancelation in all cases and our
    call to |rows.Close()| raced with the background thread's call to
    |rows.Next()|.
  • 678: server,sql/p{arse,lan}: Move handling of KILL statements to plan nodes instead of adhoc regex parsing in server handler.
  • 676: Bug fix for multi-column index matching and test for same
  • 675: Design for RowFrame
  • 673: fix cross join panic
    The cross join expression check would panic for any Comparator outside of the whitelist, instead of returning false to indicate an unmatchable expression. 
    > SELECT a6, x5, b2+a6, e4  FROM t6, t4, t5, t2 WHERE b4 in (434,267,535,407,331,2,921,469)   AND c5=819   AND d6 in (192,129,970,151)   AND (c2=728 OR 806=c2 OR 141=c2)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1e2c6d3]
goroutine 1 [running]:
github.com/dolthub/go-mysql-server/sql/expression.(*GetField).Name(...)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/get_field.go:82
github.com/dolthub/go-mysql-server/sql/analyzer.comparisonSatisfiesJoinCondition({0x2d03fbe8, 0xc0013df5e0}, 0xc0013df580)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:50 +0x2d3
github.com/dolthub/go-mysql-server/sql/analyzer.expressionCoversJoin.func1({0x29e5070, 0xc0013df5e0})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:61 +0x5b
github.com/dolthub/go-mysql-server/sql/expression.InspectUp.func1({0x29e5070, 0xc0013df5e0})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:61 +0x39
github.com/dolthub/go-mysql-server/sql/expression.InspectUp.func2({0x29e5070, 0xc0013df5e0})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:68 +0x27
github.com/dolthub/go-mysql-server/sql/expression.TransformUp({0x29e5070, 0xc0013defc0}, 0xc00016dfc8)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:53 +0x13d
github.com/dolthub/go-mysql-server/sql/expression.InspectUp({0x29e5070, 0xc0013defc0}, 0xc00016e020)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/expression/transform.go:67 +0xad
github.com/dolthub/go-mysql-server/sql/analyzer.expressionCoversJoin({0x29e5070, 0xc0013defc0}, 0x3cd6108)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:58 +0x3f
github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins.func1.1({0x29d3b00, 0xc0013df580})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:93 +0x318
github.com/dolthub/go-mysql-server/sql/plan.TransformUp.func1({{0x29d3b00, 0xc0013df580}, {0x29d3b00, 0xc0013df440}, 0x0, {0x37ff6f8, 0x0, 0x0}})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:98 +0x4f
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df420}, {0x29d3b00, 0xc0013df440}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:91 +0x6ba
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df440}, {0x29d3b00, 0xc0013df460}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3b00, 0xc0013df460}, {0x29d3fc8, 0xc0013df480}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3fc8, 0xc0013df480}, {0x0, 0x0}, 0xffffffffffffffff, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016e778)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
github.com/dolthub/go-mysql-server/sql/plan.TransformUpCtx({0x29d3fc8, 0xc0013df480}, 0x100c00016e7e8, 0x100e674)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:53 +0xbd
github.com/dolthub/go-mysql-server/sql/plan.TransformUp({0x29d3fc8, 0xc0013df480}, 0x203000)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:97 +0x45
github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins.func1({0x29d3fc8, 0xc0013df480})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:85 +0x159
github.com/dolthub/go-mysql-server/sql/plan.TransformUp.func1({{0x29d3fc8, 0xc0013df480}, {0x29d45f8, 0xc0013cd200}, 0x0, {0x37ff6f8, 0x0, 0x0}})
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:98 +0x4f
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d3fc8, 0xc0013df2e0}, {0x29d45f8, 0xc0013cd200}, 0x0, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016eca0)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:91 +0x6ba
github.com/dolthub/go-mysql-server/sql/plan.transformUpCtx({{0x29d45f8, 0xc0013cd200}, {0x0, 0x0}, 0xffffffffffffffff, {0x37ff6f8, 0x0, 0x0}}, 0x0, 0xc00016eca0)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:72 +0x41f
github.com/dolthub/go-mysql-server/sql/plan.TransformUpCtx({0x29d45f8, 0xc0013cd200}, 0x1d9c087, 0xc0013b6ec0)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:53 +0xbd
github.com/dolthub/go-mysql-server/sql/plan.TransformUp({0x29d45f8, 0xc0013cd200}, 0xc0007dbb01)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/plan/transform.go:97 +0x45
github.com/dolthub/go-mysql-server/sql/analyzer.replaceCrossJoins(0x26a3cc2, 0x26a0458, {0x29d45f8, 0xc0013cd200}, 0xc0013cd200)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/replace_cross_joins.go:78 +0x45
github.com/dolthub/go-mysql-server/sql/analyzer.(*Batch).evalOnce(0x29d45f8, 0xc00096d740, 0xc0007dbb60, {0x29d45f8, 0xc0013cc5d0}, 0x104e534)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/batch.go:93 +0x1e7
github.com/dolthub/go-mysql-server/sql/analyzer.(*Batch).Eval(0xc0006b9350, 0x37ff6f8, 0xc0007dbb60, {0x29d45f8, 0xc0013cc5d0}, 0x1)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/batch.go:54 +0x145
github.com/dolthub/go-mysql-server/sql/analyzer.(*Analyzer).analyzeWithSelector(0xc0007dbb60, 0xc00054eb40, {0x29d45f8, 0xc00096d740}, 0x0, 0x271def0)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/analyzer.go:376 +0x314
github.com/dolthub/go-mysql-server/sql/analyzer.(*Analyzer).Analyze(...)
/Users/vinairachakonda/go/pkg/mod/github.com/dolthub/go-mysql-server@v0.11.1-0.20211201172932-4744e4cc5d2d/sql/analyzer/analyzer.go:348
...
  • 672: Automatically Generate README File
    Doesn't include argument types for functions.
    Janky solution to creating Instances of FunctionN.
    Might be difficult to edit README now.
  • 668: fix _example/main.go sql.Schema -> sql.PrimaryKeySchema
  • 667: sql/parse: Remove adhoc parsing of certain structures with regexes and custom lexers.
    Moves the following to vt/sqlparser:
    • SHOW VARIABLES
    • SHOW WARNINGS
      Gets rid of unnecessary special handling for SET and SHOW PROCESSLIST.
  • 665: SqlEngine tracks BackgroundThreads
    SqlEngine manages the lifecycle of async background threads through a
    BackgroundThreads object. Integrators can add goroutines before or
    after SqlEngine initialization, but finalizing the engine will terminate
    the threads.
    Async goroutines are expected to terminate when the context provided by
    the thread manager is cancelled. The BackgroundThreads instance uses a
    wait group to allow threads to drain gracefully before exiting.
  • 663: Restrict HASH IN conversion to supported operators.
    We were converting TupleIn to HashIn for expressions that HashIn could
    not evaluate. This change makes explicit which left and right children
    we permit for HashIn conversion, and expands HashIn.Eval to evaluate
    those expression classes. Added expression unit tests, analyzer unit
    tests, query enginetests, and query planner enginetests for more sets of
    expressions that we support and do not support.
    Additionally, this simplifies the expression evaluation hashing
    technique. The new behavior has not been verified with deeply nested
    tuple expressions.

vitess

  • 103: go/mysql: conn: Add LoadInfile method which returns a ReadCloser for reading the remote file from a client.
    We will migrate the implementation in go-mysql-server from HandleLoadDataLocalQuery to this interface. This keeps the responsibility for writing the file and the likes out of the vitess/conn layer.
  • 102: Added GRANT, REVOKE, and the remainder of CREATE USER
  • 101: go/vt/sqlparser: Parse KILL {CONNECTION,QUERY} statements.
  • 100: go/vt/sqlparser: Extend the parser for correctly parsing some extended SHOW syntax.
    • SHOW COUNT(*) {WARNINGS, ERRORS}
    • SHOW {WARNINGS,ERRORS} LIMIT ...[, ...]
    • SHOW {,SESSION,GLOBAL} VARIABLES {LIKE,WHERE} ...
    • SHOW FULL PROCESSLIST
  • 99: Added more statements for user handling
    Added RENAME USER, CREATE ROLE, DROP ROLE, SHOW GRANTS, and a portion of CREATE USER. The rest of CREATE USER, along with GRANT and REVOKE will come in the next PR.
  • 97: Added proper syntax parsing for account names, along with DROP USER
    This PR primarily implements the changes necessary for account names, which required edits in the parser itself. DROP USER was implemented to test the name variations, so refer to the tests in parse_test.go for how account names work (and to see why the changes were necessary). All parser tests were validated against a MySQL 8.0 instance.
    In a nutshell, account names take the form user@host, and both user and host must be separate strings. The parser treated @ as a standard letter, and we used regex matching in GMS and Dolt to catch "invalid" names (@@system_var is valid but what@@ is invalid). My first pass was to treat the entire account name as a string and split it using a regex, however I could not get it to pass all of the parser tests. This was the only way I could make it work with all tests passing.
    Notes:
    • The other statements will come in a different PR, which will be straightforward in comparison.
    • DROP USER ''; looks like weird syntax, but an empty name means that all names match. In this case, you can assign some dbs/tables to be available to everyone.
    • Following the above, DROP USER ``; is valid, but failed as we returned an error on empty quoted identifiers. MySQL allows empty quoted identifiers to parse, but rejects them as invalid names depending on the query. This has been fixed.
    • Added ast_permissions.go to put all of the statements dealing with users, grants, etc. For me, ast.go operates very slowly and occasionally hangs for a few minutes, so I'm putting my new statements in a new file for now. I doubt you'll have pushback, but if so I can merge the file back into ast.go when I'm done.
  • 96: Better support for MySQL MULTI_STATEMENTS.

Closed Issues

  • 2122: dolt should invoke the pager for long output
  • 2448: Window functions returning incorrect results
  • 2478: Dolt diff misses some rows
  • 2484: Index breaks query when using lower()
  • 2468: dolt sql -q "select dolt_checkout('main')" does not checkout main
  • 664: _example/main.go unable to compile
  • 653: HASH IN fails on arithmetic, function nodes
Check out latest releases or
releases around dolthub/dolt v0.34.9

Don't miss a new dolt release

NewReleases is sending notifications on new releases.

Get notifications