Merged PRs

dolt

• 5398: go/commands/engine: get NBF from database provider
• 5396: go/commands/engine: set analyzer parallelism to 1
• 5391: Fixes comments in entrypoint
  Fixes comments in entrypoint
  Actually server starts at line 151 https://github.com/dolthub/dolt/pull/5391/files#diff-7cfc2aa3796f096105480a1a669658cfc12d2688b6300eef89d9ce94eef44e26L151
• 5386: go/store/nbs: aws_table_persister.go: Fix conjoin code to not error on a single short Read() by using io.ReadFull.
  In #5307 we started returning a real io.Reader for chunkSource.reader(ctx). That means the Read() calls can come back short if they are directly attached to a net.TCPConn, for example. It is not an error for a Read() to come back short. Use io.ReadFull to correctly express the intent in the conjoin code for AWS remotes.
  Fixes a bug where doltremoteapi fails commits sometimes.
• 5383: Emit schema elements from dolt_schemas table as their DDL equivalents during dolt dump
  Fixes #5318
• 5381: Bump golang.org/x/net from 0.2.0 to 0.7.0 in /go
  Bumps golang.org/x/net from 0.2.0 to 0.7.0.
  Commits

  • 8e2b117 http2/hpack: avoid quadratic complexity in hpack decoding
  • 547e7ed http2: avoid referencing ResponseWrite.Write parameter after returning
  • 39940ad html: parse comments per HTML spec
  • 87ce33e go.mod: update golang.org/x dependencies
  • 415cb6d all: fix some comments
  • 7e3c19c all: correct typos in comments
  • 296f09a http2: case insensitive handling for 100-continue
  • f8411da nettest: fix tests on dragonfly and js/wasm
  • 8e0e7d8 go.mod: update golang.org/x dependencies
  • 7805fdc http2: rewrite inbound flow control tracking
  • Additional commits viewable in compare view
  
  [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
  Dependabot commands and options

  
  You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/dolt/network/alerts).
• 5380: dolt_diff respects qualified database
  This change corrects the dolt_diff system table to use the qualified database if provided instead of always defaulting to the session's current database.
  fixes: #5226
• 5374: go/store/nbs: Fix some quota leaks in conjoin, GC.
  Adds a paranoid mode where we noisely detect unclosed table files. The mode can be enabled by setting an environment variable.
  Fixes some unit tests, including all of go/store/... to run cleanly under the paranoid mode.
  Changes the quota interface to:
  • Release |sz int| bytes instead of requiring a []byte with the correct length to show up.
  • Work with |int| instead of |uint64|, since MaxUint64 is never allocatable and MaxInt32+z is only allocatable on 64-bit platforms.
  • Not return an error on Release(). Implementations should not fail to release quota.
• 5373: go/store/nbs: rename chunks journal constants in preparation for jour…
  …nal index
• 5368: Make dropping and adding the same view produce no diff
  Fixes #5313
  Also made dropping all views (and triggers) cause the dolt_schemas table to delete itself, so that you can create views and then drop them without causing a diff.
• 5367: ArtifactEditor now resets after flushing
  ArtifactEditor held on to keys after flushing because it was a value receiver. Under the capacity threshold, constraint violations only flush once and we bypass this bug. For >50k constraint violations, every incremental CV will write the previous duplicate 40k CV's.
• 5365: Handle multiple lines in tabular diffs
  This changes our tabular output so that multiline strings display in a consistent manner. This will allow us to properly display text diffs as previewed in #5335. Modifications to that PR will follow this one.
  This is how diffs will look like with this PR:
  
  Performance-wise, I tested a SELECT over 5000 rows with two text columns containing all multiline strings for every row, and I averaged around 10% faster than the old implementation. We now sample every 100 rows rather than only the first 100, and this makes each local chunk of 100 rows visually consistent. In addition, I've cut down on the amount of formatting that is done to the columns, which is what I believe is the source of the speed increase.
  As a side benefit, dolt sql-client uses the same tabular output logic, so now our SQL client will handle multiline strings from any compatible database, not just Dolt. The default MySQL client behaves like our old implementation, so I think this is another cool win.
• 5362: Tweaks to README for binlog replication launch
• 5353: Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0 in /go
  Bumps github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0.
  Changelog

  Sourced from github.com/aws/aws-sdk-go's changelog.

  Release v1.34.0 (2020-08-07)

  Service Client Updates

  • service/glue: Updates service API and documentation
    • AWS Glue now adds support for Network connection type enabling you to access resources inside your VPC using Glue crawlers and Glue ETL jobs.
  • service/organizations: Updates service API and documentation
    • Documentation updates for some new error reasons.
  • service/s3: Updates service documentation and examples
    • Updates Amazon S3 API reference documentation.
  • service/sms: Updates service API and documentation
    • In this release, AWS Server Migration Service (SMS) has added new features: 1. APIs to work with application and instance level validation 2. Import application catalog from AWS Application Discovery Service 3. For an application you can start on-demand replication

  SDK Features

  • service/s3/s3crypto: Updates to the Amazon S3 Encryption Client - This change includes fixes for issues that were reported by Sophie Schmieg from the Google ISE team, and for issues that were discovered by AWS Cryptography.

  Release v1.33.21 (2020-08-06)

  Service Client Updates

  • service/ec2: Updates service API, documentation, and paginators
    • This release supports Wavelength resources, including carrier gateways, and carrier IP addresses.
  • service/lex-models: Updates service API and documentation
  • service/personalize: Updates service API and documentation
  • service/personalize-events: Updates service API and documentation
  • service/personalize-runtime: Updates service API and documentation
  • service/runtime.lex: Updates service API and documentation

  Release v1.33.20 (2020-08-05)

  Service Client Updates

  • service/appsync: Updates service API and documentation
  • service/fsx: Updates service documentation
  • service/resourcegroupstaggingapi: Updates service documentation
    • Documentation updates for the Resource Group Tagging API namespace.
  • service/sns: Updates service documentation
    • Documentation updates for SNS.
  • service/transcribe: Updates service API, documentation, and paginators

  Release v1.33.19 (2020-08-04)

  Service Client Updates

  • service/health: Updates service documentation
    • Documentation updates for health

  Release v1.33.18 (2020-08-03)

  ... (truncated)

  Commits

  • ae9b9fd Release v1.34.0 (2020-08-07)
  • 1e84382 Merge commit '12ff57a16373dda5a0c22eafdf0fa1c4c224f7c4' into release
  • b811ea8 Release v1.33.21 (2020-08-06) (#3462)
  • 12ff57a Updates to the Amazon S3 Encryption Client - This change includes fixes for i...
  • 2007a98 Release v1.33.20 (2020-08-05) (#3460)
  • 39b4438 Release v1.33.19 (2020-08-04) (#3458)
  • e14cc11 Merge pull request #3432 from diehlaws/common-files-standardization
  • 9a13de7 Release v1.33.18 (2020-08-03) (#3456)
  • 41f3140 Add reference links to readme
  • 29d57fc Implementing suggested changes
  • Additional commits viewable in compare view
  
  [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
  Dependabot commands and options

  
  You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/dolt/network/alerts).
• 5346: Block schema import when table has rows
  Fixes #1850
• 5327: print something for FK diffs
  It is possible to commit foreign keys that haven't been resolved yet.
  If there are differences in the column tags referenced in a foreign key, but not in the tables' schemas, our diffs are blank (even the hashes are the same).
  This just prints that something was modified for the affected foreign keys.
• 5164: implement spatial indexes
  Changes:
  • Add feature flag to enabled spatial indexes: DOLT_ENABLE_SPATIAL_INDEX
  • Changes to flatbuffer stuff to read/write spatial indexes
  • First pass at naive and unoptimized implementation of Spatial Indexes
  • Tests for Spatial Indexes, Spatial Index Plans, and their Prepared counterparts (some are skipped)
  • Contains some fixes for typeinfo not retaining SRID information for multi-geometries
  • Changes to fix some diff table prepared issues
    Companion PR: dolthub/go-mysql-server#1558
    Note: prepared test that I changed here had a column anmed commit, which got confused with to_commit in the diff #5188
    TODOs
  • Proper benchmarks
  • Potential Optimizations:
  • Level Pruning
  • Z-BBox skipping
  • Subdividing queries
  • Make Spatial Indexes Covering
  • Multi-Threading Level Searches

go-mysql-server

• 1600: enginetest: Small cleanup for DROP DATABASE and DROP TABLE tests to be a little more hygienic in their use of state across the tests.
• 1598: refactor(driver): adjust comments and interfaces to clarify behavior
  Improves the readability of the code w.r.t what purposes the Provider handles and points to the fact that the Provider can implement ContextBuilder.
  Adds a new DSN function to access the original dsn on the Connector.
  Fixes #1596
• 1597: Restore driver tests
  This PR restores and updates the driver package's tests and example. I included tests and an example when I submitted my original PR but since then they were removed.
• 1594: Fix join hints for scope unnesting
  Fixes join hints for recent subquery unnesting and semiJoin -> innerJoin rules, and fixes invalid testing setup issues.
  Tracking Dolt bump: #5361
• 1593: driver/conn: implement conn, execer, and queryer interfaces
  Implement more of the driver.Conn interfaces:
  • driver.Conn
  • driver.Execer
  • driver.ExecerContext
  • driver.Queryer
  • driver.QueryerContext
• 1592: name unnamed values properly
  Fixes #1591
  Signed-off-by: Dave Henderson dhenderson@gmail.com
• 1589: Separate cardinality from costing, convert semi to inner join
  This started out as a semiJoin optimization: semiJoin(xy ab) => project(ab) -> innerJoin(xy, distinct(ab)).
  Adding project and distinct to memo groups was hard because we did not differentiate between operator cost (estimation of compute time) and cardinality (number of rows returned by a relational expression). Refactoring costing to distinguish the two caused a lot of bugs. As a result, I converted many of the join tests into a format that will run each with a biased coster. All of those join op tests should be run with every join operator possible in its expression tree.
  The biggest effect is that we now choose merge and hash joins a lot more often for small tables, which are most of our preexisting tests.
  We also change a lot of integration test plans. The increase in testing makes me less worried about correctness and more worried about perf. But the current changes are necessary to bring us closer to histogram-based costing. I would rathe do this now, when we have more leeway for increasing perf compared to a month ago.
  There is still room for improvements organizing costing, merge join, and join tests as a whole.
• 1558: apply spatial index lookup
  Changes:
  • Add support for displaing SPATIAL KEY for SHOW CREATE TABLE.
  • Allow IndexedTableAccess for in-memory tables that have spatial indexes
  • dolt pr with interface changes: #5164
  • added new testing suite for Spatial Indexes
  • some fixes to prepared queries for filtered tables

vitess

• 220: Exposing SQL Option and SQL Mode flags for Query events
  Query binlog events are sent from the primary when the replica needs to execute a SQL statement directly. This is only for DDL operations, since DML operations are handled with a structured data message that indicates exactly which rows changed.
  There's extra metadata passed in Query binlog messages that wasn't getting exposed by Vitess that we need to execute the statements correctly (e.g. foreign_key_checks disabled). This PR exposes the SQL Options and SQL Mode metadata so that we can access them from Dolt.
• 219: Backporting client support for caching_sha2_password auth
  The MySQL client API in our fork of vitess is limited to the mysql_native_password auth plugin, but the default auth plugin as of MySQL 8.0 is caching_sha2_password. This means that to use Dolt binlog replication, customers either have to change the default auth plugin in MySQL or they have to create a user that is identified with the older mysql_native_password auth plugin. This additional configuration step adds friction for customers wanting to try out Dolt binlog replication.
  This PR pulls in selected changes from the tip of vitessio/vitess to pick up client support for connecting to MySQL servers that default to caching_sha2_password auth plugin (and will still fall back to mysql_native_password). All of these changes are pulled directly from vitessio/vitess, without any other changes mixed in.
  I'm not aware of anywhere else in our codebase where we're using the MySQL client API from vitess. As far as I know, it's only used by the binlog implementation. That means this change shouldn't have any effect on other parts of Dolt/GMS, and I've gone ahead and run the Dolt CI tests against this change as a sanity check:
  • #5339
    I've pulled over additional tests from the tip of vitess for this logic. I've also tested locally and confirmed that my binlog replication integ tests all pass with this change, and no longer require overriding the default auth plugin for the MySQL server. I've also started on some additional binlog replication integ tests to make sure we cover password/no-password and ssl/no-ssl.

Closed Issues

• 984: Dolt SQL select order is inconsistent on larger tables
• 5292: Importing a dump using dolt sql creates an unexpected new database instead of using cwd
• 5318: Dump of dolt_schemas table not compatible with MySQL. Maybe need a views option with dolt, none, or mysql as options?
• 5226: dolt_diff system table doesn't respect qualified database
• 1850: dolt removes data on incremental schema update
• 4333: Support --user and --password sql-server args even if their is a config.yaml
• 5313: Deleting and re-inserting the same view results in a dolt diff
• 5316: Dolt creates secondary indexes of columns in a foreign when you run a delete query and there are foreign key constraints
• 5372: A dolt gc'd repo is 5x the size of push/clone
• 5366: Panic runtime error: index out of range [1] with length 1
• 5370: Weird dropped final decimal query results
• 5287: EXPLAIN with uuid_to_bin is not working in IntelliJ IDEA (like DataGrip)
• 251: LastInsertId always returns 0
• 1596: driver/: dsn database name appears to be ignored
• 1591: unbound variable "v1" in query

Latency