supabase/supavisor v2.10.0

Release v2.10.0

on GitHub

Overview

Supavisor 2.10 contains many new features and enhancements, including:

• Message-level tracking of backend connections in transaction mode, which improves the handling of pipelining, extended query protocol errors and COPY.
• Optional detection of statements that leave session state behind in transaction mode.
• Pool draining on node shutdown, and pool rebalancing across the cluster.
• Retries waiting for clients over the connection limit
• Handling of password changes during authentication, and support for custom SCRAM iteration counts.

The above items and other new features of Supavisor 2.10 are explained in more detail in the sections below.

Changes

Connection Pooling

• Track backend connections message by message in transaction mode (#1079, #1176, #1205, #1207). A backend is returned to the pool only after it has answered everything forwarded to it. Previously pipelined batches, a Sync sent in a separate write, or a failed COPY could release the backend too early or leave the client hanging.
• Add tenant setting txn_mode_leak_action to detect statements that leave session state behind in transaction mode (#1113). Statements such as session-level SET, LISTEN, session advisory locks and temporary tables can be logged or rejected. The default, ignore, keeps the previous behavior.
• Fix tenant setting client_idle_timeout, which had no effect (#1068). Idle client connections are now closed after the configured time.
• Add tenant setting server_reset_query (#1238). It sets the query run before a session mode connection returns to the pool, as in pgbouncer. The default remains DISCARD ALL.
• Hold clients over the connection limit until a slot frees up, instead of rejecting them with EMAXCONN immediately (#1179). The wait, about 1.5 seconds by default, is set at build time by
  ADMISSION_RETRIES and ADMISSION_BACKOFF.
• Allow least-recently-used eviction of backend prepared statements (#980). Set the backend_prepared_statements_storage feature flag to lru to enable it. Random eviction remains the default.
• Fix several issues with named prepared statements (#1185, #1207). A Describe on a backend that had not prepared the statement no longer fails with 26000, named portals are kept, and statements a backend failed or skipped are prepared again on next use.

Authentication

• Handle password changes during client authentication (#1213). With cleartext authentication, the password is checked again against the new secrets. SCRAM clients receive a 28P01 error asking them to reconnect.
• Support SCRAM secrets with iteration counts other than 4096 (#1222). Counts up to 32768 are accepted.
• Apply SASLprep to passwords as PostgreSQL does (#960). Some non-ASCII passwords previously failed to authenticate.
• Add feature flag cleartext_auth_over_tls to use cleartext password authentication for TLS clients (#1224, #1249). This saves a round trip, but clients that require SCRAM, such as libpq withrequire_auth=scram-sha-256, can't connect, so it is off by default.
• Count only client-caused failures towards the auth_error circuit breaker (#1240). Previously an upstream outage, such as a failing auth query or JIT API, could block client addresses across the cluster.
• Invalidate cached validation secrets when the last pool for a tenant and user stops (#1208)
• Forward the client's IP address when proxying JIT connections to another node (#1186). Network restriction checks previously saw the proxying node's address.
• Fix a crash with JIT connections to tenants without upstream_verify (#1038)

Client Connections

• Accept PostgreSQL Boolean spellings, such as on and yes, in connection startup options (#1201). Invalid values are now reported with a NOTICE instead of being ignored silently.
• Allow serving separate TLS certificates to clients connecting through specific hostnames (#1244). They are configured with the DOWNSTREAM_V2_* variables.
• Disconnect clients whose socket writes time out (#1060). Previously such clients could linger and hold a backend, possibly mid-transaction.
• Add a 30-second send timeout to backend sockets (#1261). Previously a write to an unresponsive backend blocked until the kernel gave up.
• Report TCP errors and fatal TLS alerts on client connections as ECLIENTSOCKETCLOSED (#1230)
• Fix a crash when proxying a client that did not specify a database (#1267)
• Fix a crash when a client disconnected before the allow_list check (#1231)
• Fix a race that could leave a dead backend connection in the pool (#1206)
• Add tenant setting log_connections (#1236). It controls logging of connections and disconnections, and defaults to true.

Cluster Management

• Drain pools before a node shuts down (#1203, #1204, #1215, #1219, #1226). A node that is shutting down stops receiving new pools, and gives its clients up to 2.5 seconds to finish before closing them.
• Add POST /api/rebalance to redistribute pools across the cluster (#1218). Rebalancing stops pools, and their clients start them on the right node when they reconnect.
• Add GET /api/tenants/:external_id/pools to list a tenant's pools across the cluster (#1237)
• Accept JSON Booleans for banned in PATCH /api/tenants/:external_id (#1050)
• Keep the :syn event handler across hot code upgrades (#1095)

Server Configuration

• Add SUPAVISOR_LOG_BURST_LIMIT_ENABLE, SUPAVISOR_LOG_BURST_LIMIT_MAX_COUNT and SUPAVISOR_LOG_BURST_LIMIT_WINDOW_TIME to configure the logger's burst limit (#1214)
• Detect the IP version of the metadata database when SUPAVISOR_DB_IP_VERSION is unset (#1027). Previously IPv4 was assumed, and IPv6-only hosts failed.

Monitoring

• Add label app_name to connection metrics (#891). It requires the app_name_metric feature flag.
• Add histogram supavisor_client_connection_request_duration (#1241, #1253). It records the duration of each connection request, labeled by outcome and by who was at fault
• Add counter supavisor_client_errors for errors that terminated client connections (#1230)
• Add counters supavisor_client_admission_admitted and supavisor_client_admission_rejected (#1179)
• Add scheduler utilization metrics (#1200)
• Add cgroup v2 memory metrics (#1233)
• Add disk space metrics (#1070)
• Add TCP active and passive open counters (#1161)
• Add EC2 ENA network allowance metrics (#1265). They are exported when ENA_INTERFACE is set.
• Add metrics for the node's IPv6 address and AMI version (#1077, #1146)
• Add counters for log lines and for lines dropped by the logger's burst limit (#1220)
• Remove the tenant label from supavisor_client_handler_state_duration (#1168). This reduces the metric's cardinality.
• Fix deletion of a tenant's metrics when only one of its pools stopped (#1221)
• Report unset upgrade history labels of supavisor_prom_ex_application_version_info as nil (#1138). Prometheus dropped them when they were empty strings.
• Add tls to client connection log metadata and db_host to backend connection log metadata (#1227, #1228)
• Add tenant metadata to pool conflict and cache invalidation logs (#1139, #1259)
• Always log the payload of invalid startup messages (#1075)

Performance

• Reduce CPU usage when forwarding large result sets (#1207)
• Enable SO_REUSEPORT on client listeners (#1155)
• Collect tenant metrics in a single pass (#1199). With thousands of tenants, collection could previously take longer than its interval.
• Reduce memory usage of TLS client connections (#1223)

Special thanks

We thank the community contributors who helped shape this release:

Snehil Shah, for working on server_reset_query, startup option validation, client_idle_timeout , and for contributing to the message-level tracking of backend connections and to application_name handling, along with several smaller fixes and test improvements.

Dhruv Vaishnav, for fixing Describe of named prepared statements on backends that had not prepared them yet.

MorganaFuture, for their work on least-recently-used eviction of prepared statements.

breken-ai, for fixing named portals handling when prepared statements are enabled.