pg-sharding/spqr 3.1.0

on GitHub

We are excited to announce SPQR 3.1.0!

This release focuses on production operations and reliability: improved redistribution tooling and recovery, connection pinning and PostgreSQL advisory lock support, stronger two-phase commit recovery, new key range management commands, extended protocol improvements, and better observability.

This release covers all changes from 3.0.0 to 3.1.0.

What's Changed

Data Redistribution & Recovery

• Add spqr-redistributor by @EinKrebs #2648
• Add checks to spqr-redistributor by @EinKrebs #2675
• Add spqr-redistributor generate-task run command by @EinKrebs #2717
• Add commands to verify locked key ranges and recover failed task groups to spqr-monitor by @EinKrebs #2638
• Disable recovering key range moves by default by @EinKrebs #2634
• Fix unlocking key range in spqr-monitor verify by @EinKrebs #2651
• Fix spqr-monitor recover deletion of key range moves by @EinKrebs #2688
• Fix task group restart when transferring a key range in one go by @EinKrebs #2718
• Add DROP REDISTRIBUTE TASK ALL command by @EinKrebs #2786
• Check that key range is locked before inserting/deleting data during redistribution by @EinKrebs #2813
• Make recoverable transfer errors SPQR-specific by @reshke #2787
• Make most insert-time redistribution errors recoverable by @reshke #2788
• Improve key range move and data transfer error reporting by @reshke #2714 #2727 #2728 #2750 #2783
• Split operation lock and write lock for key ranges by @EinKrebs #2871
• Await PIDs before locking key range during move by @EinKrebs #2722

Key Range & Shard Management

• Introduce CREATE KEY RANGES FOR DISTRIBUTION by @diPhantxm #2604
• Support SQL for CREATE KEY RANGES FOR DISTRIBUTION by @diPhantxm #2639
• Add documentation for CREATE KEY RANGES FOR DISTRIBUTION by @diPhantxm #2739
• Support CREATE SHARD ... FORCE to ignore shard checks by @diPhantxm #2635
• Check shard migrations on CREATE SHARD by @diPhantxm #2595
• Add MOVE KEY RANGE ... META ONLY command by @EinKrebs #2825
• Update MOVE KEY RANGE documentation by @EinKrebs #2846
• Drop key range move status from QDB by @diPhantxm #2830
• Fix key range left locked after RENAME failure by @diPhantxm #2867
• Update key range metadata on shards when deleting all key ranges by @diPhantxm #2883
• Delete tables from spqrguard metadata on DROP TABLE by @EinKrebs #2740

Advisory Locks & Connection Pinning

• Add initial pg_advisory_lock support by @reshke #2661
• Create common infrastructure for connection pinning by @reshke #2751
• Implement connection pinning by @reshke #2768
• Automatically pin connections for advisory locks by @reshke #2775
• Add tests for auto-pinning by @reshke #2801
• Add advisory lock stress testing by @reshke #2802

Two-Phase Commit (2PC)

• Add __spqr__allow_autoprotect_2pc GUC by @reshke #2641
• Improve transaction deployment error reporting by @reshke #2671
• Add deployment context to transaction utilities by @reshke #2694
• Improve 2PC recovery logic by @reshke #2865 #2866
• Remove two-phase metadata for successfully reverted transactions by @reshke #2909
• Remove transaction metadata for successfully committed 2PC transactions with eager cleanup by @reshke #2921
• Add eager 2PC cleanup GUC and tests by @reshke #2920
• Add transaction listing methods and extended two-phase transaction SHOW target by @reshke #2922
• Optionally persist only 2PC transaction data in MemQDB by @EinKrebs #2924
• Allow the 2PC watchdog to work without a backend rule by @EinKrebs #2925
• Use compare-and-swap for transaction status changes by @EinKrebs #2926

Query Routing & Execution

• Support in-router aggregate functions by @reshke #2754
• Add default read-only transaction support by @reshke #2842
• Add SPQR EXECUTE HOST FILTER by @reshke #2905
• Fix query cancellation in multishard queries by @reshke #2703
• Match catalog queries case-insensitively by @reshke #2732
• Try already acquired routes for route-oblivious plans by @reshke #2930
• Fix command tag handling for reference relation modifications by @reshke #2762

Extended Protocol & Portals

• Invalidate portal description cache on parse by @reshke #2645
• Fix multiple portal execution iterations with non-zero MaxRows by @reshke #2680
• Fix portal suspension hazards for single-shard queries by @reshke #2682
• Fix slice description for multi-slice virtual plans by @reshke #2779
• Support extended protocol during query analysis by @reshke #2833
• Support ParseComplete in xproto message copy by @reshke #2885
• Fix dummy prepared statement parameter substitution by @reshke #2919
• Hash parameter OIDs in prepared statement names by @reshke #2933
• Fix INT2 bind parameter parsing and named portal binds by @reshke #2940

Console & SQL

• Implement generic string GUCs by @reshke #2663
• Add log_query router configuration option by @reshke #2702
• Fix spqr-console handling of empty queries by @reshke #2791
• Reject invalid UNITE KEY RANGE ALL / SPLIT KEY RANGE ALL commands by @reshke #2808
• Improve console grammar and reject additional invalid SQL by @reshke #2798 #2814
• Make SPQR_DEFAULT_ROUTE_BEHAVIOUR a generic string GUC by @reshke #2818
• Make preferred execution engine a generic GUC by @reshke #2820
• Add assign hooks for generic GUCs by @reshke #2826
• Make SPQR_REPLY_NOTICE a generic GUC by @reshke #2836
• Make __spqr__sharding_key a generic GUC by @reshke #2844
• Make maintenance parameters generic GUCs by @reshke #2918
• Make SHOW notice message format configurable through GUC by @reshke #2908

Monitoring & Observability

• Add mean key range lock time statistics by @EinKrebs #2636
• Add counter for authentication and TLS initialization errors by @reshke #2664
• Add structured test reports and automatic flaky-test candidate tracking by @Denchick #2670 #2677 #2687 #2692
• Add instance start time and last reload time to SHOW INSTANCE by @reshke #2683
• Add total non-virtual queries counter by @reshke #2699
• Add Prometheus summary metric type by @xelavopelk #2585
• Add client alive time metric by @reshke #2719
• Add client network address to disconnection messages by @reshke #2720
• Add metrics from ConnectionStatMgr by @xelavopelk #2742
• Improve SPQR-specific error logging and data-transfer logging by @reshke #2674 #2714 #2728
• Fix log message loss during shutdown by @reshke #2906
• Rate-limit cancel request processing by @reshke #2917

Configuration, Security & Networking

• Add gRPC TLS configuration types and credential adapter by @Denchick #2748
• Configure TSA recheck frequency through dbpool_tsa_recheck_period by @reshke #2849
• Properly implement TSA checker timeouts by @reshke #2853
• Do not hold pool lock while checking connections for TSA match by @reshke #2665
• Redact secrets in configuration output by @Denchick #2845
• Remove invalid pool settings from backend rules by @Denchick #2900
• Add validation for empty shard on client connection by @Denchick #2726

Stability & Correctness

• Use graceful shutdown for failed-auth client pools by @reshke #2656
• Fix adapter UNITE implementation by @reshke #2723
• Fix task worker goroutine leaks by @Denchick #2784
• Fix index-out-of-range panic when parsing shard host addresses by @Denchick #2840
• Fix potential goroutine leak in connection acquisition by @reshke #2852
• Fix pool data race during host parsing by @reshke #2857
• Fix panic caused by nil connection on listener error by @diPhantxm #2824
• Cleanup backend connections after search path hazards by @reshke #2899
• Fix malformed DISTRIBUTION REPLICATED handling by @Denchick #2777
• Fix bootstrap error logging by @EinKrebs #2923

Documentation & CI

• Revise Docker image promotion documentation by @Denchick #2628
• Add documentation for enabling spqrguard by @Denchick #2889
• Run benchmarks on schedule and relevant file changes by @diPhantxm #2632 #2662
• Make benchmark CI retryable by @diPhantxm #2676
• Speed up generated-code checks by @Denchick #2730
• Update CodeQL configuration by @Denchick #2746

Full Changelog: 3.0.0...3.1.0