Release of Vitess v24.0.0

Summary

Table of Contents

• Major Changes
  • New Support
    • Window function pushdown for sharded keyspaces
    • View Routing Rules
    • Tablet targeting via USE statement
  • Breaking Changes
    • External Decompressor No Longer Read from Backup MANIFEST by Default
• Minor Changes
  • Logging
    • Structured logging
  • VReplication
    • --shards flag for MoveTables/Reshard start and stop
    • Automatic tablet retry for tablet-specific errors
  • VTGate
    • Removed --grpc-send-session-in-streaming flag
    • New default for --legacy-replication-lag-algorithm flag
    • New "session" mode for --vtgate-balancer-mode flag
    • Binlog Streaming Support
  • Query Serving
    • JSON_EXTRACT now supports dynamic path arguments
  • VTTablet
    • New Experimental flag --init-tablet-type-lookup
    • QueryThrottler Observability Metrics
    • QueryThrottler Event-Driven Configuration Updates
    • New in_order_completion_pending_count field in OnlineDDL outputs
    • Tablet Shutdown Tracking and Connection Validation
    • Connection Pool Waiter Cap
  • Tracing
    • OpenTelemetry tracing support
    • Deprecation of OpenTracing-based tracing backends
  • VTOrc
    • New --cell Flag
    • Improved VTOrc Discovery Logging
    • Ordered Recovery Execution and Semi-Sync Rollout
    • Deprecated VTOrc Metric Removed
    • Deprecation of Snapshot Topology feature
    • Deprecated /api/replication-analysis Endpoint Removed
  • Metrics
    • Extended Go Runtime Metrics via Prometheus
  • Backup and Restore
    • MySQL CLONE Support for Replica Provisioning
    • Restore Hook Improvements

Major Changes

New Support

Window function pushdown for sharded keyspaces

This release introduces an optimization that allows window functions to be pushed down to individual shards when they are partitioned by a column that matches a unique vindex.

Previously, all window function queries required single-shard routing, which limited their applicability on sharded tables. With this change, queries where the PARTITION BY clause aligns with a unique vindex can now be pushed down and executed on each shard.

For examples and more details, see the documentation.

View Routing Rules

Vitess now supports routing rules for views, and can be applied the same as tables with vtctldclient ApplyRoutingRules. When a view routing rule is active, VTGate rewrites queries that reference the source view to use the target view's definition instead. For example, given this routing rule:

{
  "rules": [
    {
      "from_table": "source_ks.my_view",
      "to_tables": ["target_ks.my_view"]
    }
  ]
}

And this view definition:

CREATE VIEW target_ks.my_view AS SELECT id, name FROM user;

A query like SELECT * FROM source_ks.my_view would be internally rewritten to:

SELECT * FROM (SELECT id, name FROM target_ks.user) AS my_view;

View routing rules require the schema tracker to monitor views, which means VTGate must be started with the --enable-views flag and VTTablet with the --queryserver-enable-views flag. The target view must exist in the specified keyspace for the routing rule to function correctly. For more details, see the Schema Routing Rules documentation.

Tablet targeting via USE statement

VTGate now supports routing queries to a specific tablet by alias using an extended USE statement syntax:

USE keyspace:shard@tablet_type|tablet_alias;

For example, to target a specific replica tablet:

USE commerce:-80@replica|zone1-0000000100;

Once set, all subsequent queries in the session route to the specified tablet until cleared with a standard USE keyspace or USE keyspace@tablet_type statement. This is useful for debugging, per-tablet monitoring, cache warming, and other operational tasks where targeting a specific tablet is required.

Note: A shard must be specified when using tablet targeting. Like shard targeting, this bypasses vindex-based routing, so use with care.

Breaking Changes

External Decompressor No Longer Read from Backup MANIFEST by Default

The external decompressor command stored in a backup's MANIFEST file is no longer used at restore time by default. Previously, when no --external-decompressor flag was provided, VTTablet would fall back to the command specified in the MANIFEST. This posed a security risk: an attacker with write access to backup storage could modify the MANIFEST to execute arbitrary commands on the tablet.

Starting in v24, the MANIFEST-based decompressor is ignored unless you explicitly opt in with the new --external-decompressor-use-manifest flag. If you rely on this behavior, add the flag to your VTTablet configuration, but be aware of the security implications.

See #19460 for details.

Minor Changes

Logging

Structured logging

Vitess now uses structured JSON logging by default. Log output is emitted as JSON to stderr. To configure the minimum log level, pass --log-level (one of debug, info, warn, error; default info). For a human-readable format with automatic color detection, pass --log-format=text. To revert to the previous glog backend, pass --log-structured=false.

glog is deprecated as of v24 and will be removed in v25.

VReplication

--shards flag for MoveTables/Reshard start and stop

The start and stop commands for MoveTables and Reshard workflows now support the --shards flag, allowing users to start or stop workflows on a specific subset of shards rather than all shards at once.

Example usage:

# Start workflow on specific shards only
vtctldclient MoveTables --target-keyspace customer --workflow commerce2customer start --shards="-80,80-"

# Stop workflow on specific shards only
vtctldclient Reshard --target-keyspace customer --workflow cust2cust stop --shards="80-"

Automatic tablet retry for tablet-specific errors

VReplication workflows now automatically retry with different tablets when encountering tablet-specific errors. Previously, workflows without a cell preference would default to the local cell and could get stuck retrying the same failing tablet indefinitely.

When a tablet encounters errors like binary log purging (MySQL error 1236 or 1789) or GTID set mismatches, VReplication adds that tablet to an ignore list and tries other tablets across all cells. Once all matching tablets have been tried, the ignore list is cleared and the workflow retries from scratch.

This is particularly useful in multi-cell deployments where a tablet in the local cell may lack the required binary logs, but tablets in other cells still have them.

VTGate

Removed --grpc-send-session-in-streaming flag

The VTGate flag --grpc-send-session-in-streaming has been removed. This flag was deprecated in v22 via #17907 and defaulted to true.

The session is now always sent as the last packet in the streaming response for StreamExecute and StreamExecuteMulti RPCs. This behavior is required to support transactions in streaming and cannot be disabled.

Impact: Remove any usage of the --grpc-send-session-in-streaming flag from VTGate startup scripts or configuration.

New default for --legacy-replication-lag-algorithm flag

The VTGate flag --legacy-replication-lag-algorithm now defaults to false, disabling the legacy approach to handling replication lag by default.

Instead, a simpler algorithm purely based on low lag, high lag and minimum number of tablets is used, which has proven to be more stable in many production environments. A detailed explanation of the two approaches is explained in this code comment.

In v25 this flag will become deprecated and in the following release it will be removed. In the meantime, the legacy behaviour can be used by setting --legacy-replication-lag-algorithm=true. This deprecation is tracked in #18914.

New "session" mode for --vtgate-balancer-mode flag

The VTGate flag --vtgate-balancer-mode now supports a new "session" mode in addition to the existing "cell", "prefer-cell", and "random" modes. Session mode routes each session consistently to the same tablet for the session's duration.

To enable session mode, set the flag when starting VTGate:

--vtgate-balancer-mode=session

Binlog Streaming Support

VTGate now supports GTID-based binlog streaming through two protocols:

• MySQL protocol: Clients can connect using the standard MySQL COM_BINLOG_DUMP_GTID replication command—no special VStream-aware adapters or direct MySQL access required.
• gRPC: The new BinlogDumpGTID streaming RPC in vtgateservice.proto provides native gRPC access for custom clients without the MySQL protocol dependency.

Note: Only GTID-based streaming is supported. File/position-based streaming is not available through either COM_BINLOG_DUMP or COM_BINLOG_DUMP_GTID and returns an error.

This feature is disabled by default. Enable it with --enable-binlog-dump.

New flags:

• --enable-binlog-dump: Enables binlog dump support. Without this flag, binlog dump requests return an error.
• --binlog-dump-authorized-users: Comma-separated list of users authorized to execute binlog dump operations, or % to allow all users.

Requirements:

When initiating a binlog dump connection, clients must specify:

• An empty filename
• A file position (filepos) of 4
• A GTID position

For gRPC clients, specify the keyspace, shard, and optionally the tablet type or tablet alias directly in the BinlogDumpGTIDRequest.

Limitations:

• Each stream operates on a single tablet—no data aggregation across shards.
• No automatic failover—if the targeted tablet becomes unavailable, the stream fails and the client must reconnect to a different tablet.
• Not compatible with MoveTables or Reshard operations. Use the VStream API for those use cases.

Query Serving

JSON_EXTRACT now supports dynamic path arguments

The JSON_EXTRACT function now supports dynamic path arguments like bind variables or results from other function calls. Previously, JSON_EXTRACT only worked with static string literals for path arguments.

Null handling now matches MySQL behavior. The function returns NULL when either the document or path argument is NULL.

Static path arguments are still optimized, even when mixed with dynamic arguments, so existing queries won't see any performance regression.

VTTablet

New Experimental flag --init-tablet-type-lookup

The new experimental flag --init-tablet-type-lookup for VTTablet allows tablets to automatically restore their previous tablet type on restart by looking up the existing topology record, rather than always using the static --init-tablet-type value.

When enabled, the tablet uses its alias to look up the tablet type from the existing topology record on restart. This allows tablets to maintain their changed roles (e.g., RDONLY/DRAINED) across restarts without manual reconfiguration. If disabled or if no topology record exists, the standard --init-tablet-type value will be used instead.

Note: Vitess Operator–managed deployments generally do not keep tablet records in the topo between restarts, so this feature will not take effect in those environments.

QueryThrottler Observability Metrics

VTTablet now exposes new metrics to track QueryThrottler behavior.

Four new metrics have been added:

• QueryThrottlerRequests: Total number of requests evaluated by the query throttler
• QueryThrottlerThrottled: Number of requests that were throttled
• QueryThrottlerTotalLatencyNs: Total time each request takes in query throttling, including evaluation, metric checks, and other overhead (nanoseconds)
• QueryThrottlerEvaluateLatencyNs: Time taken to make the throttling decision (nanoseconds)

All metrics include labels for Strategy, Workload, and Priority. The QueryThrottlerThrottled metric has additional labels for MetricName, MetricValue, and DryRun to identify which metric triggered the throttling and whether it occurred in dry-run mode.

These metrics help monitor throttling patterns, identify which workloads are throttled, measure performance overhead, and validate behavior in dry-run mode before configuration changes.

QueryThrottler Event-Driven Configuration Updates

QueryThrottler configuration is now stored in SrvKeyspace within the topology server and managed using standard topology tools. Previously, tablets polled for configuration changes every 60 seconds. Tablets now use event-driven watches (WatchSrvKeyspace) to receive updates immediately when throttling configuration changes. All tablets in a keyspace see configuration changes at roughly the same time, and topology server changes are versioned and auditable.

This change replaces the previous file-based configuration loader with a protobuf-defined configuration structure stored in the topology. The new configuration includes fields for enabling/disabling throttling, selecting the throttling strategy, and configuring strategy-specific rules.

New in_order_completion_pending_count field in OnlineDDL outputs

OnlineDDL migration outputs now include a new in_order_completion_pending_count field. When using the --in-order-completion flag, this field shows how many migrations must complete before the current migration. The field is visible in SHOW vitess_migrations queries and vtctldclient OnlineDDL <db> show outputs.

This provides better visibility into migration queue dependencies, making it easier to understand why a migration might be postponed. The count is automatically updated during the scheduler loop and cleared when migrations complete, fail, or are cancelled.

Tablet Shutdown Tracking and Connection Validation

Vitess now tracks when tablets cleanly shut down and validates tablet records before attempting connections, reducing unnecessary connection attempts and log noise.

New Field: A new tablet_shutdown_time field has been added to the Tablet protobuf. This field is set to the current timestamp when a tablet cleanly shuts down and is cleared (set to nil) when the tablet starts. This allows other Vitess components to detect when a tablet is intentionally offline.

Connection Validation: When a tablet record has tablet_shutdown_time set, Vitess components will skip connection attempts and return an error indicating the tablet is shutdown. VTOrc will now skip polling tablets that have tablet_shutdown_time set. For tablets that shutdown uncleanly (crashed, killed, etc.), the field remains nil and the pre-v24 behavior is preserved (connection attempt with error logging).

Note: This is a best-effort mechanism. Tablets that are killed or crash may not have the opportunity to set this field, in which case components will continue to attempt connections as they did in v23 and earlier.

Tablet Connection Pool Waiter Cap

VTTablet now allows to set a limit on the number of requests waiting to get a connection from the connection pool, for
the query, stream and transaction connection pools. The limits are set with the following flags:

• --queryserver-config-query-pool-waiter-cap.
• --queryserver-config-stream-pool-waiter-cap.
• --queryserver-config-txpool-waiter-cap.

All of the above have a default value of 0, meaning no limit, thus preserving the behavior of the previous version.

Tracing

OpenTelemetry tracing support

Vitess now supports OpenTelemetry as a tracing backend. To use it, set --tracer opentelemetry on any Vitess binary. Traces are exported via OTLP/gRPC, configurable with the following flags:

• --otel-endpoint: OpenTelemetry collector endpoint. If empty, the OTEL_EXPORTER_OTLP_ENDPOINT env var is used; if that is also unset, the OTel SDK defaults to localhost:4317.
• --otel-insecure (default false): use insecure connection to the collector.
• --tracing-sampling-rate (default 0.1): sampling rate for traces (shared across all tracing backends).

Any OTLP-compatible backend (Jaeger v1.35+, Grafana Tempo, Datadog Agent, etc.) can receive these traces.

Deprecation of OpenTracing-based tracing backends

The following tracing backends are deprecated as of v24 and will be removed in v25:

• opentracing-jaeger — Uses the Jaeger client-go library, which has been archived. The Jaeger project recommends migrating to OpenTelemetry. Users should migrate to --tracer opentelemetry with an OTLP-compatible Jaeger endpoint (v1.35+).
• opentracing-datadog — Uses the OpenTracing bridge in dd-trace-go. Users should migrate to --tracer opentelemetry with the Datadog Agent's OTLP ingestion endpoint.

The --tracer opentracing-jaeger and --tracer opentracing-datadog options continue to work in v24 but will log a deprecation warning at startup. The following Jaeger-specific flags are also deprecated and will be removed in v25:

• --jaeger-agent-host
• --tracing-sampling-type

Migration: Replace --tracer opentracing-jaeger with --tracer opentelemetry and --jaeger-agent-host host:port with --otel-endpoint host:4317. Ensure your Jaeger deployment accepts OTLP (Jaeger v1.35+ listens on port 4317 by default).

VTOrc

New --cell Flag

VTOrc now supports a --cell flag that specifies which Vitess cell the VTOrc process is running in. The flag is optional in v24 but will be required in v25+, similar to VTGate's --cell flag.

When provided, VTOrc validates that the cell exists in the topology service on startup. Without the flag, VTOrc logs a warning about the v25+ flag requirement.

This enables future cross-cell problem validation, where VTOrc will be able to ask another cell to validate detected problems before taking recovery actions. The flag is currently validated but not yet used in VTOrc recovery logic.

Note: If you're running VTOrc in a multi-cell deployment, start using the --cell flag now to prepare for the v25 requirement.

Improved VTOrc Discovery Logging

VTOrc's DiscoverInstance function now includes the tablet alias in all log messages and uses the correct log level when errors occur. Previously, error messages did not indicate which tablet failed discovery, and errors were logged at INFO level instead of ERROR level.

This improvement makes it easier to identify and debug issues with specific tablets when discovery operations fail.

Ordered Recovery Execution and Semi-Sync Rollout

VTOrc now executes recoveries per-shard with defined ordering, rather than per-tablet in isolation. Problems that have ordering dependencies (e.g., semi-sync configuration) are executed serially first, while independent problems are executed concurrently. This ensures that dependent recoveries happen in the correct sequence within a shard.

The main user-facing improvement is to semi-sync rollouts: VTOrc now ensures replicas have semi-sync enabled before updating the primary. Previously, enabling semi-sync on the primary before enough replicas were ready could stall writes while the primary waited for semi-sync acknowledgements that no replica was prepared to send.

See #19427 for details.

Deprecated VTOrc Metric Removed

The DiscoverInstanceTimings metric has been removed from VTOrc in v24. This metric was deprecated in v23.

Migration: Use DiscoveryInstanceTimings instead, which provides the same timing information for instance discovery actions (Backend, Instance, Other).

Impact: Monitoring dashboards or alerting systems using DiscoverInstanceTimings must be updated to use DiscoveryInstanceTimings.

Deprecation of Snapshot Topology feature

VTOrc's Snapshot Topology feature, which is enabled by setting --snapshot-topology-interval to a non-zero-value is deprecated as of v24 and the logic is planned for removal in v25.

The lack of facilities to read the snapshots created by this feature coupled with the in-memory nature of VTOrc's backend means this logic has limited usefulness. This deprecation is explained and tracked in detail in #18691.

Migration: remove the VTOrc flag --snapshot-topology-interval before v25.

Impact: VTOrc can no longer create snapshots of the topology in it's backend database.

Deprecated /api/replication-analysis Endpoint Removed

The /api/replication-analysis endpoint has been removed from VTOrc in v24. Use /api/detection-analysis instead, which provides the same functionality.

Migration: Update any scripts, monitoring systems, or automation that calls /api/replication-analysis to use /api/detection-analysis instead. The replacement endpoint accepts the same query parameters (keyspace, shard) and returns the same JSON response format.

Impact: HTTP requests to /api/replication-analysis will return a 404 Not Found error.

Metrics

Extended Go Runtime Metrics via Prometheus

Vitess now exposes the full set of Go runtime metrics via Prometheus. The default Prometheus GoCollector only exposes three runtime/metrics (/gc/gogc:percent, /gc/gomemlimit:bytes, /sched/gomaxprocs:threads) plus the legacy go_memstats_* set. Starting in v24, all Vitess components expose approximately 150 additional metrics from Go's runtime/metrics package.

New metrics include:

• Heap allocation histograms
• GC cycle counts and pause histograms
• Memory class breakdowns
• Goroutine state breakdowns
• Scheduler latency histograms
• CPU time breakdowns by class (user, GC, scavenge, idle)

A new go_info_ext gauge is also added with compiler, GOARCH, and GOOS labels, providing extended build environment information beyond the standard go_info metric.

Affected components: vtgate, vttablet, vtctld, vtorc, vtbackup, mysqlctld

No configuration required — the metrics appear automatically on the /metrics endpoint for all components using the Prometheus backend.

Backup and Restore

MySQL CLONE Support for Replica Provisioning

VTTablet and VTBackup now support using MySQL's native CLONE plugin to provision new replicas by copying data directly from a donor tablet over the network. Physical-level data copying is significantly faster than logical backup and restore, especially for large datasets. Requires MySQL 8.0.17+ and InnoDB-only tables.

New Flags:

Clone User Configuration:

Example Usage:

Clone from the shard's primary:

vttablet \
  --mysql-clone-enabled \
  --restore-with-clone \
  --clone-from-primary \
  ...

Clone from a specific tablet:

vtbackup \
  --mysql-clone-enabled \
  --restore-with-clone \
  --clone-from-tablet=zone1-0000000100 \
  ...

Note: All tablets participating in CLONE operations (both donors and recipients) must have --mysql-clone-enabled set during MySQL initialization to ensure the CLONE plugin is loaded and the clone user exists.

Restore Hook Improvements

Extended Hook Coverage: The vttablet_restore_done hook now fires when restores are triggered via vtctldclient RestoreFromBackup. Previously, this hook only ran during tablet startup or clone operations.

New Environment Variable: The hook now sets TM_RESTORE_DATA_BACKUP_ENGINE to indicate which backup engine was used. The value comes from the backup manifest's BackupMethod field.

TM_RESTORE_DATA_BACKUP_ENGINE is only set when a restore reads from an actual backup—not for clone-based restores or when no backup is used. Hook scripts can use this to perform engine-specific actions based on whether the restore used builtin, xtrabackup, or another engine.

The entire changelog for this release can be found here.

The release includes 436 merged Pull Requests.

Thanks to all our contributors: @ChaitanyaD48, @Devanshusharma2005, @MargaretMorehead, @anujagrawal380, @aparajon, @app/dependabot, @app/promptless, @app/vitess-bot, @arthurschreiber, @c-r-dev, @chengyuan, @connorolaya, @dasl-, @dbussink, @demmer, @derekperkins, @ejortegau, @esignorelli, @farhann-saleem, @frouioui, @ghostframe, @harshit-gangal, @jdoupe, @khkim6040, @lizztheblizz, @maksimov, @mattlord, @maxenglander, @mcpherrinm, @mcrauwel, @mhamza15, @nickvanw, @pourtorabehsan, @rjlaine, @rvrangel, @sbaker617, @shlomi-noach, @siddharth16396, @stefanb, @stutibiyani, @systay, @tanjinx, @tetsuro-ohyama, @timvaillancourt, @ttran397, @twthorn, @varundeepsaini, @vitess-bot, @yushuqin