Release of Vitess v12.0.0

Major Changes

This release includes the following major changes or new features.

Inclusive Naming

A number of commands and RPCs have been deprecated as we move from master to primary.
All functionality is backward compatible except as noted under Incompatible Changes.

Gen4 Planner

The newest version of the query planner, Gen4, becomes an experimental feature as part of this release.
While Gen4 has been under development for a few release cycles, we have now reached parity with its predecessor, v3.

To use Gen4, VTGate's -planner_version flag needs to be set to gen4.

Minor Changes

Query Serving

• Add SHOW VITESS_REPLICATION_STATUS Query Support #8900
  • The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment.

Incompatible Changes

vtctl command output

Wherever vtctl commands produced master or MASTER for tablet type, they now produce primary or PRIMARY.
Scripts and tools that depend on parsing command output will need to change.

Example:

$ vtctlclient -server localhost:15999 ListAllTablets 
zone1-0000000100 sourcekeyspace 0 primary 192.168.0.134:15100 192.168.0.134:17100 [] 2021-09-24T01:12:00Z
zone1-0000000101 sourcekeyspace 0 rdonly 192.168.0.134:15101 192.168.0.134:17101 [] <null>
zone1-0000000102 sourcekeyspace 0 replica 192.168.0.134:15102 192.168.0.134:17102 [] <null>
zone1-0000000103 sourcekeyspace 0 rdonly 192.168.0.134:15103 192.168.0.134:17103 [] <null>

Governance

• Added @GuptaManan100 to maintainers #8833

Changelog

Bug fixes

Build/CI

• Update golang/x dependencies #8688
• tests: use AtomicInt32 instead of int to fix races #8696

Cluster management

• Use consul lock properly #8310
• Trivial: mysqlctl reinit_config -h was not showing help #8369
• Port #8422 to main branch #8745
• [vtctl] Add missing call to flag.Parse in commandGetRoutingRules #8795
• Fix for padding in OrderAndCheckPartitions #8873

Observability

• schema engine: reset table size stats when a table is dropped #8634

Query Serving

• Fix for function calls in DEFAULT value of CREATE TABLE statement in main #8477
• Fixing multiple issues related to onlineddl/lifecycle #8500
• boolean values should not be parenthesised in default clause #8502
• mysql/server: abort connection if secure transport is required #8504
• Clear only the current shard's schemacopy rows during schema reload #8519
• go/mysql/auth: fix error message for wrong auth method. #8525
• Fix handshake protocol with MySQL client #8537
• Copy parameter data from byte buffer in COM_STMT_SEND_LONG_DATA #8562
• Negative float default #8587
• [vtexplain] prevent panic on show tables/show full tables #8601
• OnlineDDL: better scheduling/cancellation logic #8603
• Fix parsing of FIRST clause in ALTER TABLE #8614
• onlineddl Executor: build schema with DBA user #8624
• Handle subquery merging with references correctly #8662
• Fixing error on deletes from owning table for a lookup being populated by CreateLookupVindex after a SwitchWrites #8701
• Fixing a panic in vtgate with OLAP mode #8722
• default to primary tablet if not set in VStream api #8755
• Fix bug in parsing of version comments #8770
• Proper merge of the SysTableTableName fields when joining two routes #8771
• fix parsing of MySQL Server Version flag #8791
• Fix typo in pool config syntax. Fixes #8819 #8820
• VtGate schema tracker to use provided user #8857
• fix merging dba query as a subquery both in v3 and gen4 #8871

VReplication

• MoveTables: don't create unnecessary streams on the target for non-intersecting sources and targets #8090
• VDiff: Add BIT datatype to list of byte comparable types #8401
• Fix vreplication error metric #8483
• Return from throttler goroutine if context is cancelled to prevent goroutine leaks #8489
• Fix VReplication logging to file and db #8521
• Fix passing the wrong cell/cells variable to CreateLookupVindex #8590
• Refresh SrvVSchema after an ExternalizeVindex: was missing #8670
• Fixing missing argument in function call #8692
• Fix how we identify MySQL generated columns #8763

VTAdmin

• [vtadmin-web] Set Backup status indicators to the correct colours #8410
• [vtadmin] Fix bad copy-paste in pool config flag parsing #8518

vtctl

• Print actual current datetime for vtctl dry run commands #8778

vttestserver

• Remove lingering mysqld sock files on vttestserver container startup #8463
• Fixed filename for the configuration file in Vttestserver #8809

CI/Build

Build/CI

• Update release process #7759
• [ci] Add errcheck to golangci-lint #7996
• gomod: do not replace GRPC #8416
• consolidation: Fix flaky test #8417
• Small build improvements #8418
• proto: upgrade vtprotobuf version #8454
• Automatically use the latest tag of Vitess for cluster upgrade E2E test #8471
• Addition of known issues to release notes #8482
• Fix Cluster 14 flakiness #8494
• add question on backporting back to PR template #8541
• Enable GitHub Actions concurrency feature #8546
• Updated Makefile do_release script to include godoc steps #8550
• Changing codahale/hdrhistogram import path #8637
• Upgrade to Go 1.17 #8640
• Fixes for reparent endtoend test flakiness #8642
• hooks: remove govet because it already runs as part of golangci-lint #8674
• Enhancement of the release instructions #8790
• Remove consul-api usage in favor of official consul/api #8794
• Enhancement of the release notes generation #8877

Governance

• update MAINTAINERS and CODEOWNERS for deepthi, pH14 and vmg #8675

Query Serving

• Cleanup: import of gh-ost test suite is complete #8459

VReplication

• VReplication support for non-PRIMARY KEY (Online DDL context) #8364

Documentation

Build/CI

• maintainers: add Messaging as area of expertise #8578

Cluster management

• Enhance k8stopo flag documentation #8464
• Update some vtctl/vtctlclient command help #8560
• Trivial: fix the GenerateShardRanges vtctl[client] help #8586

Examples

• examples: update README to use new vreplication command syntax #8825

Governance

• update governance #8609

Query Serving

• vtexplain examples #8652

VReplication

• vtctl help/usage update #8879

VTAdmin

• [vtadmin] Remove outdated commands from vtadmin-web README #8496
• Initial pass at some vtadmin docs #8526

Enhancement

Backup and Restore

• Add Support for Restoring Specific Backups #8824

Build/CI

• Initial support for stress testing in end-to-end tests #8406
• CI: Remove mariadb101 which is well past eol and no longer available in distributions #8446
• Adding GitHub Self Hosted Runner tests #8721

Cluster management

• Vitess mixin improvements #7499
• Expose topo_consul_lock_session_checks to allow customized consul session check #8298
• Handle lock release with SIGHUP in VTGR #8472
• Enhance PRS error message #8529
• srvtopo: resilient watcher #8583
• Add -restart_before_backup parameter for vtbackup #8608
• srvtopo: allow unwatching from watch callbacks #8633
• introduced cluster_operation as a new error code in vtrpc #8646
• servenv: add --onclose_timeout flag #8651
• Improve determinism of bootstrap #8840

Observability

• [vtadmin] cluster debug pages #8427
• Export memstats for statsd backend #8777

Query Serving

• [tablet, queryrules] Extend query rules to check comments #8233
• Add "show global status" support #8344
• added batch lookup param to lookup vindex #8398
• Online DDL/VReplication: able to read from replica #8405
• multishard autocommit should work for bypass routing #8428
• Add "no-scatter" flag to prohibit the use of scatter queries #8439
• OnlineDDL: -skip-topo is always 'true' #8450
• Add rows as keyword in sqlparser #8467
• Periodic update of gh-ost binary #8470
• fix when show tables record too much error log "Got unhandled packet.." #8478
• SHOW VITESS_MIGRATION '...' LOGS, retain logs for 24 hours #8493
• NativeDDL: analyzing added&removed unique keys #8495
• Refactor authentication server plugin mechanism #8503
• Update to planetscale/tengo v0.10.1-ps.v4 #8516
• query serving to continue when topo server restarts - main #8534
• support grpc reflection on APIs #8551
• Customized CreateLookUpVindex to be Compatible with non-consistent Lookup Vindex #8570
• Add additional options for configuring SSL modes as a client #8588
• Fail plan for unsupported aggregate function #8593
• gateway: use keyspace events when buffering requests #8700
• Add optional query annotations, i.e. prefix SQL comments indicating the #8783
• Slight improvement of the ACL error message #8805
• Changed parsing for unions #8821
• Use vt_dba user for online schema migration cutover phase #8836
• Add parsing support for column list with subquery alias #8884

TabletManager

• Allow min TLS version for tablet to mysqld conns #8757

VReplication

• VReplication: Add ability to tag workflows #8388
• Ignore SBR statements from pt-table-checksum #8396
• Tablet Picker: add metric to record lack of available tablets #8403
• VReplication: support different keys on source and target tables (different covered columns) #8423
• VStream API: handle reparenting and unhealthy tablets #8445
• Change local example to use v2 vreplication flows and change default to v2 from v1 #8553
• Customized CreateLookUpVindex to Include a Flag to Continue Vreplication After Backfill with an Owner Provided #8572
• Update MoveTables CLI help to reflect change we made at the beginning of #8597
• VStreamer Field/Row Events: add Keyspace/Shard #8598
• VStream API: Add flag to stop streaming on a reshard #8628

VTAdmin

• [vtadmin] cluster rpc pools #8421
• Add support for passing custom interceptors to vtadmin grpcserver #8507

Feature Request

Cluster management

• [vtctldserver] Migrate remaining ServingGraph rpcs to VtctldServer #8249
• [grpctmclient] Add support for (bounded) per-host connection reuse #8368
• VTGR: Vitess + MySQL group replication #8387
• Add isActive flag to vtgr to support multi-cell topology #8780
• [vtctld] Add SetWritable, StartReplication and StopReplication rpcs #8816
• [vtctld] sleep/ping tablets #8826
• [vtctld] migrate more util rpcs #8843
• [vtctld] migrate validator rpcs #8849
• [vtctld] localvtctldclient #8882
• [vtctldserver] Migrate RunHealthCheck #8892
• [vtctl] run new commands as standalone binary #8893

VTAdmin

• [vtadmin] [experimental] add per-api RBAC implementation #8515
• [vtadmin] shard replication positions #8775
• [vtadmin] GetVtctlds #8792

Internal Cleanup

Build/CI

• Bump aws-sdk-go to v1.34.2 #8632
• Upgrade consul api: go get github.com/hashicorp/consul/api@v1.10.1 #8784

Cluster management

• topodata: remove deprecated field 'ServedTypes' #8566
• Upgrade to k8s 1.18 client #8762

Deployments

• Remove the deprecated Helm charts and related code #8868

Examples

• Bump ini from 1.3.5 to 1.3.8 in /vitess-mixin/e2e #8823

Governance

• fix copyright #8611

Query Serving

• Adds flag to vttablet to disallow online DDL statements #8433
• Remove vtrpc.LegacyErrorCode #8456
• Allow for configuration of the minimal TLS version #8460
• engine: allow retrying partial primitives #8727
• srvtopo: expose WatchSrvKeyspace #8752
• Clean up Primitive interface implementations #8901

VReplication

• LegacySplitCloneWorker is no longer used in any code paths #8867

VTAdmin

• [vtadmin] Add Options struct to wrap grpc/http options #8461
• Update vtadmin local scripts to enable basic rbac #8801

vtctl

• Correctly identify backup timestamp var as a string #8891

Other

Examples

• Improve the Docker local and compose examples #8685

Performance

Cluster management

• throttler: don't allocate any resources unless it is actually enabled #8643

Query Serving

• cache: track memory usage more closely #8804

VTAdmin

• Use fmt.Fprintf instead of fmt.Sprintf #8922

Testing

Cluster management

• Alter tests to reuse cluster components #8276
• Adds some more orchestrator tests to vtorc #8535

Query Serving

• increase conn killer check to double the tx timeout value #8649
• Added UNION testcases and auxilary code for running tests #8797
• fixed regression in v3 for grouping by integer functions #8856

VTAdmin

• [vtadmin] Add a vtctld Dialer unit test #8455

The release includes 1281 commits (excluding merges)

Thanks to all our contributors: @Anders-PlanetScale, @GuptaManan100, @Johnny-Three, @Juneezee, @ajm188, @aquarapid, @askdba, @bnu0, @choo-stripe, @dbussink, @dctrwatson, @deepthi, @demmer, @dependabot[bot], @derekperkins, @doeg, @eeSeeGee, @falun, @fatih, @frouioui, @hallaroo, @harshit-gangal, @ilikeorangutans, @mattlord, @rafael, @ritwizsinha, @rohit-nayak-ps, @sahutd, @shlomi-noach, @sonne5, @systay, @tdakkota, @tokikanno, @utk9, @vmg