Release of Vitess v23.0.0

Summary

Table of Contents

• Major Changes
  • Breaking Changes
  • Flag Naming Convention Migration
  • New default versions
    • Upgrade to MySQL 8.4
  • New Support
    • Multi-Query Execution
    • Transaction Timeout Session Variable
    • Experimental: Query Throttler
    • Multiple Lookup Vindexes Support
    • Reference Tables in Materialize Workflows
    • Online DDL Shard-Specific Completion
    • WITH RECURSIVE CTEs
    • CREATE TABLE ... SELECT Support
  • Deprecations
    • Metrics
    • CLI Flags
  • Deletions
    • Metrics
  • New Metrics
    • VTGate
    • VTTablet
    • VTOrc
• Minor Changes
  • New CLI Flags
    • VReplication/Materialize
    • Observability
    • VTOrc
    • Backup/Restore
    • CLI Tools
    • VTAdmin
    • VTGate
  • Modified Metrics
  • Parser/SQL Enhancements
  • Query Planning Improvements
  • Topology
    • --consul-auth-static-file requires 1 or more credentials
  • VTOrc
    • Aggregated Discovery Metrics HTTP API removed
    • Dynamic control of EmergencyReparentShard-based recoveries
    • Recovery stats to include keyspace/shard
    • /api/replication-analysis HTTP API deprecation
  • VTTablet
    • API Changes
    • CLI Flags
    • Managed MySQL configuration defaults to caching-sha2-password
    • MySQL timezone environment propagation
    • gRPC tabletmanager client error changes
  • Docker

Major Changes

Breaking Changes

Deleted VTGate Metrics

Four deprecated VTGate metrics have been completely removed in v23.0.0. These metrics were deprecated in v22.0.0:

Impact: Any monitoring dashboards or alerting systems using these metrics must be updated to use the replacement metrics introduced in v22.0.0:

• Use QueryExecutions instead of QueriesProcessed
• Use QueryRoutes instead of QueriesRouted
• Use QueryExecutionsByTable instead of QueriesProcessedByTable and QueriesRoutedByTable

See the v22.0.0 release notes for details on the new metrics.

ExecuteFetchAsDba No Longer Accepts Multi-Statement SQL

The ExecuteFetchAsDba RPC method in TabletManager now explicitly rejects SQL queries containing multiple statements (as of PR #18183).

Impact: Code or automation that previously passed multiple semicolon-separated SQL statements to ExecuteFetchAsDba will now receive an error. Each SQL statement must be sent in a separate RPC call.

Migration: Split multi-statement SQL into individual RPC calls:

// Before (no longer works):
ExecuteFetchAsDba("CREATE TABLE t1 (id INT); CREATE TABLE t2 (id INT);")

// After (required in v23+):
ExecuteFetchAsDba("CREATE TABLE t1 (id INT);")
ExecuteFetchAsDba("CREATE TABLE t2 (id INT);")

gRPC TabletManager Error Code Changes

The vttablet gRPC tabletmanager client now returns errors wrapped by the internal go/vt/vterrors package (PR #18565).

Impact: External automation relying on google-gRPC error codes must be updated to use vterrors.Code(err) to inspect error codes, which returns vtrpcpb.Codes defined in proto/vtrpc.proto.

Migration:

// Before:
if status.Code(err) == codes.NotFound { ... }

// After:
if vterrors.Code(err) == vtrpcpb.Code_NOT_FOUND { ... }

GTID API Signature Changes

Several GTID-related API signatures changed in PR #18196 as part of GTID performance optimizations:

Changed: BinlogEvent.GTID() method signature
Impact: Code directly using the GTID parsing APIs may need updates. Most users are unaffected as these are internal APIs.

GenerateShardRanges API Signature Change

The key.GenerateShardRanges() function signature changed in PR #18633 to add a new hexChars int parameter controlling the hex width of generated shard names.

Impact: Code calling GenerateShardRanges() directly must be updated to pass the new parameter.

The corresponding vtctldclient command gained a new --chars flag to control this behavior.

Flag Naming Convention Migration

Vitess v23.0.0 includes a major standardization of CLI flag naming conventions across all binaries. 989 flags have been migrated from underscore notation (flag_name) to dash notation (flag-name) in PR #18280 and related PRs.

Backward Compatibility

• v23.0.0 and v24.0.0: Both underscore and dash formats are supported. Underscore format is deprecated but functional.
• v25.0.0: Underscore format will be removed. Only dash format will be accepted.

Automatic Normalization

Flag normalization happens automatically at the pflag level (PR #18642), so both formats are accepted without requiring code changes in v23/v24.

Example Flag Renames

Common flags affected (full list of 989 flags available in PR #18280):

Backup flags:

• --azblob_backup_account_name → --azblob-backup-account-name
• --s3_backup_storage_bucket → --s3-backup-storage-bucket
• --xtrabackup_root_path → --xtrabackup-root-path

Replication flags:

• --heartbeat_enable → --heartbeat-enable
• --replication_connect_retry → --replication-connect-retry

gRPC flags (PR #18009):

• All gRPC-related flags standardized (30+ flags)

Action Required

Users should update configuration files, scripts, and automation to use dash-based flag names before upgrading to v25.0.0. The migration is backward compatible in v23 and v24, allowing gradual updates.

New default versions

Upgrade to MySQL 8.4

The default major MySQL version used by our vitess/lite:latest image is going from 8.0.40 to 8.4.6.
This change was merged in #18569.

VTGate also advertises MySQL version 8.4.6 by default instead of 8.0.40. If that is not what you are running, you can set the mysql_server_version flag to advertise the desired version.

⚠️ Upgrading to this release with vitess-operator:

If you are using the vitess-operator, considering that we are bumping the MySQL version from 8.0.40 to 8.4.6, you will have to manually upgrade:

1. Add innodb_fast_shutdown=0 to your extra cnf in your YAML file.
2. Apply this file.
3. Wait for all the pods to be healthy.
4. Then change your YAML file to use the new Docker Images (vitess/lite:v23.0.0).
5. Remove innodb_fast_shutdown=0 from your extra cnf in your YAML file.
6. Apply this file.

This is only needed once when going from the latest 8.0.x to 8.4.x. Once you're on 8.4.x, it is possible to upgrade and downgrade between 8.4.x versions without needing to run innodb_fast_shutdown=0.

New Support

Multi-Query Execution

Vitess v23.0.0 introduces native support for executing multiple queries in a single RPC call through new ExecuteMulti and StreamExecuteMulti APIs (PR #18059).

This feature provides more efficient batch query execution without requiring manual query splitting or multiple round trips.

Usage Example:

queries := []string{
    "SELECT * FROM users WHERE id = 1",
    "SELECT * FROM orders WHERE user_id = 1",
    "SELECT * FROM payments WHERE user_id = 1",
}
results, err := vtgateConn.ExecuteMulti(ctx, queries)

Configuration: Enable with the --mysql-server-multi-query-protocol flag on VTGate.

Transaction Timeout Session Variable

A new transaction_timeout session variable has been added (PR #18560), allowing per-session control over transaction timeout duration.

Usage:

-- Set transaction timeout to 30 seconds for this session
SET transaction_timeout = 30;

-- Begin a transaction that will automatically rollback if not committed within 30s
BEGIN;
-- ... perform operations ...
COMMIT;

This provides more granular timeout control compared to global server settings, useful for:

• Long-running batch operations that need extended timeouts
• Interactive sessions that should fail fast
• Different timeout requirements per application workload

Experimental: Query Throttler

Vitess v23.0.0 introduces a new, experimental Query Throttler framework for rate-limiting incoming queries (RFC issue #18412, PR #18449, PR #18657). Work on this new throttler is ongoing with the potential for breaking changes in the future.

Feedback on this experimental feature is appreciated in GitHub issues or the #feat-handling-overload channel of the Vitess Community Slack.

Features:

• File-based configuration for throttling rules
• Dry-run mode for testing throttling without enforcement
• Dynamic rule reloading

Configuration:

• --query-throttler-config-refresh-interval - How often to reload throttler configuration

Dry-run Mode: Test throttling rules without actually blocking queries, useful for validating configuration before enforcement.

Multiple Lookup Vindexes Support

Creating multiple lookup vindexes in a single workflow is now supported through the --params-file flag (PR #17566).

Usage:

# Create multiple lookup vindexes from JSON configuration
vtctldclient LookupVindexCreate \
  --workflow my_lookup_workflow \
  --params-file /path/to/params.json \
  commerce

params.json example:

{
  "vindexes": [
    {
      "name": "user_email_lookup",
      "type": "consistent_lookup_unique",
      "table_owner": "users",
      "table_owner_columns": ["email"]
    },
    {
      "name": "user_name_lookup",
      "type": "consistent_lookup",
      "table_owner": "users",
      "table_owner_columns": ["name"]
    }
  ]
}

This significantly improves workflow efficiency when setting up multiple vindexes, reducing the number of separate operations required.

Reference Tables in Materialize Workflows

Reference tables can now be added to existing materialize workflows using the new Materialize ... update sub-command (PR #17804).

Usage:

# Add reference tables to an existing workflow
vtctldclient Materialize --workflow my_workflow update \
  --add-reference-tables ref_table1,ref_table2 \
  --target-keyspace my_keyspace

Use Case: Incrementally add reference tables to running materialize workflows without recreating the entire workflow, improving operational flexibility.

Online DDL Shard-Specific Completion

Online DDL migrations can now be completed on a per-shard basis using the new COMPLETE VITESS_SHARDS syntax (PR #18331).

Usage:

-- Complete migration on specific shards only
ALTER VITESS_MIGRATION '9e8a9249_3976_11ed_9442_0a43f95f28a3'
  COMPLETE VITESS_SHARDS '-80,80-';

-- Complete migration on all remaining shards
ALTER VITESS_MIGRATION '9e8a9249_3976_11ed_9442_0a43f95f28a3'
  COMPLETE;

Benefits:

• Gradual rollout of schema changes across shards
• Ability to validate changes on subset of shards before full rollout
• Better control over migration timing and impact

WITH RECURSIVE CTEs

Vitess now supports WITH RECURSIVE common table expressions (PR #18590), enabling recursive queries for hierarchical data.

Example:

-- Find all employees in a management hierarchy
WITH RECURSIVE employee_hierarchy AS (
    SELECT id, name, manager_id, 1 as level
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    SELECT e.id, e.name, e.manager_id, eh.level + 1
    FROM employees e
    INNER JOIN employee_hierarchy eh ON e.manager_id = eh.id
)
SELECT * FROM employee_hierarchy ORDER BY level, name;

This is a major SQL compatibility enhancement for applications with hierarchical or graph-like data structures.

CREATE TABLE ... SELECT Support

The SQL parser now supports CREATE TABLE ... SELECT statements (PR #18443), improving MySQL compatibility.

Example:

-- Create a table from a query result
CREATE TABLE recent_orders
SELECT * FROM orders
WHERE order_date > DATE_SUB(NOW(), INTERVAL 30 DAY);

Deprecations

Metrics

CLI Flags

As part of the Flag Naming Convention Migration, 989 CLI flags across all Vitess binaries have been deprecated in their underscore format. The dash format should be used going forward.

Deprecation Timeline:

• v23.0.0 and v24.0.0: Underscore format deprecated but functional
• v25.0.0: Underscore format will be removed

Action Required: Migrate to dash-based flag names before v25.0.0. See Flag Naming Convention Migration for details.

Deletions

Metrics

See Breaking Changes for migration guidance.

New Metrics

VTGate

Transaction Types in TransactionsProcessed:

• Single - Single-shard transactions
• Multi - Multi-shard transactions
• TwoPC - Two-phase commit transactions

Use Case: The TransactionsProcessed metric helps identify transaction patterns and shard distribution, useful for:

• Monitoring transaction types across your deployment
• Identifying queries that could be optimized to single-shard
• Tracking two-phase commit usage and potential bottlenecks

VTTablet

Use Case for OnlineDDLStaleMigrationMinutes: Alert on stalled Online DDL migrations that haven't progressed for an extended period, helping identify migrations that may need intervention.

VTOrc

Use Case for EmergencyReparentShardDisabled: Create alerts to ensure EmergencyReparentShard-based recoveries are not disabled for an undesired period, maintaining high availability posture.

SkippedRecoveries Reasons: The Reason dimension tracks why recoveries were skipped (e.g., "ERSDisabled", "ReplicationLagHigh", "NotEnoughReplicas"), providing actionable insights for operational troubleshooting.

Minor Changes

New CLI Flags

VReplication/Materialize

Observability

VTOrc

Use Case: The --allow-recovery=false flag is useful for:

• Testing VTOrc capacity/discovery performance ahead of a rollout, for example: migrating to VTOrc from another solution.
• Maintenance windows where automatic failovers should be prevented
• Debugging VTOrc behavior without triggering recoveries
• Running VTOrc in observation-only mode

Backup/Restore

CLI Tools

Example:

# Generate shard ranges with 4 hex characters
vtctldclient GenerateShardRanges --shards 16 --chars 4
# Output: -1000,1000-2000,2000-3000,...,f000-

VTAdmin

Five new TLS-related flags were added for secure vtctld connections (PR #18556):

These flags enable mTLS (mutual TLS) authentication between VTAdmin and vtctld for enhanced security.

VTGate

Modified Metrics

VTOrc Recovery Metrics Enhanced with Keyspace/Shard Labels

The following VTOrc recovery metrics now include Keyspace and Shard labels in addition to the existing RecoveryType label (PR #18304):

1. FailedRecoveries
2. PendingRecoveries
3. RecoveriesCount
4. SuccessfulRecoveries

Impact: Monitoring queries and dashboards using these metrics may need updates to account for the additional label dimensions.

Benefits:

• More granular observability into which keyspaces/shards are experiencing recovery issues
• Ability to alert on recovery patterns per keyspace/shard
• Better troubleshooting of cluster-specific issues

Example PromQL Query:

# Before (v22): Recovery count by type only
sum(rate(RecoveriesCount[5m])) by (RecoveryType)

# After (v23): Recovery count by type, keyspace, and shard
sum(rate(RecoveriesCount[5m])) by (RecoveryType, Keyspace, Shard)

VTGate QueryExecutionsByTable Behavior Change

The QueryExecutionsByTable metric now only counts successful query executions (PR #18584). Previously, it counted all query attempts regardless of success/failure.

Impact:

• Metric values may decrease if your workload had significant query failures
• More accurate representation of successful query volume
• Failed queries are tracked separately via error metrics

Parser/SQL Enhancements

Vitess v23.0.0 includes significant SQL parser improvements for better MySQL compatibility:

New SQL Syntax Support

CREATE PROCEDURE Improvements

Enhanced CREATE PROCEDURE statement parsing (PR #18142, PR #18279):

• Better handling of DEFINER clauses with various formats
• Differentiation between BEGIN...END blocks and START TRANSACTION statements
• Support for SET statements within procedure bodies
• Improved handling of semicolons within procedure definitions

Operator Precedence Fixes

• Fixed MEMBER OF operator precedence with AND (PR #18237)
• Ensures correct query evaluation when combining JSON operations with boolean logic

Query Planning Improvements

Window Functions in Single-Shard Queries

Window functions can now be pushed down to single-shard queries (PR #18103), improving performance for analytics workloads.

Before v23: Window functions were always executed at VTGate level, even for single-shard queries.

After v23: Window functions are pushed down when the query targets a single shard, reducing data transfer and improving performance.

Example:

-- This query now executes entirely on the target shard
SELECT
    user_id,
    order_date,
    amount,
    SUM(amount) OVER (PARTITION BY user_id ORDER BY order_date) as running_total
FROM orders
WHERE user_id = 12345;  -- Single-shard query

UNION Query Merging Improvements

UNION query optimization has been significantly enhanced (PR #18289, PR #18393):

Extended UNION Merging: UNION queries with Equal and IN opcodes can now be merged more aggressively, generating simpler SQL.

Derived Table Elimination: Unnecessary derived table wrapping is avoided for UNION queries, producing cleaner and more efficient SQL.

Example:

-- Query:
SELECT * FROM t1 WHERE id = 1
UNION
SELECT * FROM t1 WHERE id = 2;

-- Before v23: Wrapped in derived table
SELECT * FROM (
    SELECT * FROM t1 WHERE id = 1
    UNION
    SELECT * FROM t1 WHERE id = 2
) AS dt;

-- After v23: Direct UNION (simpler, more efficient)
SELECT * FROM t1 WHERE id = 1
UNION
SELECT * FROM t1 WHERE id = 2;

Multi-Shard Read-Only Transactions in SINGLE Mode

Read-only transactions can now span multiple shards when using SINGLE transaction mode (PR #18173).

Before v23: SINGLE mode restricted all transactions to single shards, even read-only ones.

After v23: Read-only transactions can access multiple shards in SINGLE mode, improving flexibility without sacrificing consistency guarantees.

Impact: Applications using SINGLE transaction mode can now perform multi-shard read queries within transactions without needing to upgrade to MULTI or TWOPC modes.

Deferred Optimization for Prepared Statements

Prepared statements now support deferred optimization (PR #18126), allowing preparation to succeed even when plan generation requires runtime values.

Benefits:

• More prepared statements succeed at preparation time
• Better support for queries with parameter-dependent optimization
• Reduced application errors from failed PREPARE statements

Behavior: When a prepared statement cannot be fully optimized at preparation time, optimization is deferred to execution time when bind variable values are available.

Query Buffering for INSTANT DDL

Query buffering has been implemented for INSTANT DDL operations (PR #17945), reducing query failures during schema changes.

Features:

• Automatic buffering of queries during INSTANT DDL execution
• Forced termination of blocking transactions
• Transparent to applications

Impact: Applications experience fewer query errors during schema changes, improving availability during DDL operations.

Topology

--consul-auth-static-file requires 1 or more credentials

The --consul-auth-static-file flag used in several components now requires that 1 or more credentials can be loaded from the provided json file (PR #18152).

Impact: Configurations with empty or invalid credential files will now fail at startup rather than silently continuing with no authentication.

VTOrc

Aggregated Discovery Metrics HTTP API removed

VTOrc's undocumented /api/aggregated-discovery-metrics HTTP API endpoint was removed (PR #18672). The list of documented VTOrc APIs can be found here.

We recommend using the standard VTOrc metrics to gather the same metrics. If you find that a metric is missing in standard metrics, please open an issue or PR to address this.

Dynamic control of EmergencyReparentShard-based recoveries

Note: disabling EmergencyReparentShard-based recoveries introduces availability risks; please use with extreme caution! If you rely on this functionality often, for example in automation, this may be signs of an anti-pattern. If so, please open an issue to discuss supporting your use case natively in VTOrc.

The new vtctldclient RPC SetVtorcEmergencyReparent was introduced (PR #17985) to allow VTOrc recoveries involving EmergencyReparentShard actions to be disabled on a per-keyspace and/or per-shard basis. Previous to this version, disabling EmergencyReparentShard-based recoveries was only possible globally/per-VTOrc-instance. VTOrc will now consider this keyspace/shard-level setting that is refreshed from the topo on each recovery. The disabled state is determined by first checking if the keyspace, and then the shard state. Removing a keyspace-level override does not remove per-shard overrides.

To provide observability of keyspaces/shards with EmergencyReparentShard-based VTOrc recoveries disabled, the EmergencyReparentShardDisabled metric was added. This metric label can be used to create alerting to ensure EmergencyReparentShard-based recoveries are not disabled for an undesired period of time.

Example:

# Disable ERS recoveries for a keyspace
vtctldclient SetVtorcEmergencyReparent --keyspace commerce --enabled=false

# Disable ERS recoveries for a specific shard
vtctldclient SetVtorcEmergencyReparent --keyspace commerce --shard 80- --enabled=false

# Re-enable ERS recoveries
vtctldclient SetVtorcEmergencyReparent --keyspace commerce --enabled=true

Recovery stats to include keyspace/shard

The following recovery-related stats now include labels for keyspaces and shards (PR #18304):

1. FailedRecoveries
2. PendingRecoveries
3. RecoveriesCount
4. SuccessfulRecoveries

Previous to this release, only the recovery "type" was included in labels. See Modified Metrics for more details.

/api/replication-analysis HTTP API deprecation

The /api/replication-analysis HTTP API endpoint is now deprecated and is replaced with /api/detection-analysis (PR #18615), which currently returns the same response format.

Timeline: The /api/replication-analysis endpoint will be removed in a future version. Users should migrate to /api/detection-analysis.

VTTablet

API Changes

• Added RestartReplication method to TabletManagerClient interface (PR #18628). This new RPC allows stopping and restarting MySQL replication with semi-sync configuration in a single call, providing a convenient alternative to separate StopReplication and StartReplication calls.

• Added GetMaxValueForSequences and UpdateSequenceTables gRPC RPCs (PR #18172) for VReplication sequence management during SwitchWrites operations.

CLI Flags

• --skip-user-metrics flag if enabled, replaces the username label with "UserLabelDisabled" to prevent metric explosion in environments with many unique users (PR #18085).

See New CLI Flags for complete list of new flags.

Managed MySQL configuration defaults to caching-sha2-password

The default authentication plugin for MySQL 8.0.26 and later is now caching_sha2_password instead of mysql_native_password (PR #18010). This change is made because mysql_native_password is deprecated and removed in future MySQL versions. mysql_native_password is still enabled for backwards compatibility.

This change specifically affects the replication user. If you have a user configured with an explicit password, it is recommended to make sure to upgrade this user after upgrading to v23 with a statement like the following:

ALTER USER 'vt_repl'@'%' IDENTIFIED WITH caching_sha2_password BY 'your-existing-password';

In future Vitess versions, the mysql_native_password authentication plugin will be disabled for managed MySQL instances.

MySQL timezone environment propagation

Fixed a bug where environment variables like TZ were not propagated from mysqlctl to the mysqld process (PR #18561).
As a result, timezone settings from the environment were previously ignored. Now mysqld correctly inherits environment variables.

⚠️ Deployment Impact: Deployments that relied on the old behavior and explicitly set a non-UTC timezone may see changes in how DATETIME values are interpreted. To preserve compatibility, set TZ=UTC explicitly in MySQL pods.

gRPC tabletmanager client error changes

The vttablet gRPC tabletmanager client now returns errors wrapped by the internal go/vt/vterrors package (PR #18565). External automation relying on google-gRPC error codes should now use vterrors.Code(err) to inspect the code of an error, which returns vtrpcpb.Codes defined in the proto/vtrpc.proto protobuf.

See Breaking Changes for migration guidance.

Docker

Bullseye went EOL 1 year ago, so starting from v23, we will no longer build or publish images based on debian:bullseye (PR #18609).

Builds will continue for Debian Bookworm, and add the recently released Debian Trixie. v23 explicitly does not change the default Debian tag to Trixie.

Additional Information

The entire changelog for this release can be found here.

The release includes 246 merged Pull Requests.

Thanks to all our contributors: @Arshdeep54, @BenjaminLockhart, @GrahamCampbell, @GuptaManan100, @HenryCaiHaiying, @app/dependabot, @app/vitess-bot, @arthurschreiber, @bantyK, @beingnoble03, @canoriz, @chapsuk, @chrisplim, @corbantek, @davidpiegza, @dbussink, @deepthi, @demmer, @derekperkins, @frouioui, @harshit-gangal, @jdoupe, @jeefy, @leejones, @mattlord, @maxenglander, @mdlayher, @mhamza15, @morgo, @mounicasruthi, @nickvanw, @notfelineit, @rohit-nayak-ps, @rvrangel, @shlomi-noach, @siddharth16396, @stankevich, @stutibiyani, @systay, @timvaillancourt, @twthorn, @vitess-bot, @wukuai, @yoheimuta