pg-sharding/spqr 2.9.9

SPQR 2.9.9

on GitHub

This is one of final release before 3.0.0 target.

Changes include experimental post processing support, 2pc improvements, QDB in pg initial support, JOIN routing fixes, SQL level prepared queries support and much more.

What's Changed

• Update memqdb.feature by @reshke in #2115
• Key range versioning basics by @EinKrebs in #2072
• Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #2118
• Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #2117
• Bump lyx by @reshke in #2119
• Preparation refactoring for QDB migration. by @reshke in #2116
• Support storing FQN relations in MemQDB. by @reshke in #2121
• Add HELP command PoC by @Denchick in #2057
• Support FQN relations for distribution in ETCD by @reshke in #2124
• Improve visibility for SHOW redistribute_tasks by @reshke in #2123
• Add isolation test for data move with schema-qualified relation by @reshke in #2126
• Fix state conversion bug by @reshke in #2127
• Add isolation test for data move with schema-qualified relation (#2126) by @reshke in #2128
• Fix panic in RedistributeTasksVirtualRelationScan by @EinKrebs in #2129
• Add key range commands MDX generation by @Denchick in #2125
• fixed create reference relation for regress with coordinator by @xelavopelk in #2092
• Automatic TPC-C benchmarks on label performance by @diPhantxm in #2113
• Tests for ./router/server.ShardServer.Name() by @futig in #2132
• Support routing of insert-from-aux-values. by @reshke in #2134
• More tests on CTE+aux values processing by @reshke in #2135
• Add unit tests for StmtLogger.shouldLogStatement by @Minetoff in #2114
• Add unit test for split-VALUES() INSERT logic by @reshke in #2136
• Add rewrite support for aux CTE feature by @reshke in #2137
• Idempotent RedistributeKeyRange call by @EinKrebs in #2133
• Clustered detach relation from sequence support by @xelavopelk in #2122
• converting drop key range command into 2 phase execution by @xelavopelk in #2120
• Setup FDW to a master host by @EinKrebs in #2138
• Bump yc-actions/yc-cr-login from 2 to 3 by @dependabot[bot] in #2139
• Bump peter-evans/create-or-update-comment from 4 to 5 by @dependabot[bot] in #2142
• Bump hashicorp/setup-terraform from 3 to 4 by @dependabot[bot] in #2140
• Bump actions/checkout from 2 to 6 by @dependabot[bot] in #2141
• Fix log message by @EinKrebs in #2143
• Reorder & improve operations in RunCoordiantor by @EinKrebs in #2144
• GetSequenceRelations implemented in adapter by @xelavopelk in #2145
• fix error message for regress in clustered mode by @xelavopelk in #2146
• Drop task group lock after its exit by @EinKrebs in #2147
• Support CTE rewrite by @reshke in #2148
• Bump docker/setup-qemu-action from 3 to 4 by @dependabot[bot] in #2150
• Bump docker/login-action from 3 to 4 by @dependabot[bot] in #2149
• Support transitive CTE for aux values. by @reshke in #2152
• Bump docker/setup-buildx-action from 3 to 4 by @dependabot[bot] in #2155
• Fix panic in RuleRouterImpl.PreRoute by @EinKrebs in #2154
• Fix support for insert from aux values by @reshke in #2156
• Support QdbAddrs with multiple choice in Coordinator config by @diPhantxm in #2153
• Send proper SQL state for erroneous begin/rollback statements. by @reshke in #2158
• Small cleanup patch by @reshke in #2159
• Hold redistribute task lock with its execCtx + cancel it by @EinKrebs in #2157
• Bump lyx by @reshke in #2162
• Bump google.golang.org/grpc from 1.79.1 to 1.79.2 by @dependabot[bot] in #2160
• Bump docker/build-push-action from 6 to 7 by @dependabot[bot] in #2161
• Fix CTE routing support for hashed distributions by @reshke in #2163
• Add limit processing for multi-shard queries by @reshke in #2165
• Bump golang.org/x/sync from 0.19.0 to 0.20.0 by @dependabot[bot] in #2168
• Fix routing for aux-values-insert with hashed distribution by @reshke in #2166
• Add regression suite for clustered installation. by @xelavopelk in #2151
• Use lyx types for console sort support by @reshke in #2170
• move retries "keyrange locks" BL to meta layer by @xelavopelk in #2164
• Experimental support for cluster-wide ORDER BY by @reshke in #2171
• tests: add tests on DropDistribution by @Daewoow in #2172
• Add ALTER REFERENCE TABLE SET STORAGE method by @EinKrebs in #2173
• Bump golang.org/x/crypto from 0.48.0 to 0.49.0 by @dependabot[bot] in #2176
• Set up spqrguard to forbid direct queries to shards by @EinKrebs in #2174
• fix EtcdQDB.GetRelationSequence by @xelavopelk in #2179
• Add cancel_pid column to show clients by @reshke in #2180
• Support extended protocol for virtual functions calls. by @reshke in #2185
• Add experimental support for pg_isolation_test_session_is_blocked by @reshke in #2187
• Release transaction in case of out error by @reshke in #2188
• Fixes for 890ebf8 by @reshke in #2189
• Fix expression routing data move by @EinKrebs in #2190
• Check extension versions by @EinKrebs in #2191
• MemQDB.packMemqdbCommands is beautified by @xelavopelk in #2182
• move retries "keyrange locks" BL to meta layer (step 2) by @xelavopelk in #2183
• fix: deep-copy pgproto3 message byte slices to prevent cross-client data contamination by @Ledatu in #2177
• Proper support for frontend prepared statement close by @reshke in #2192
• Add extended protocol test for Flush messages. by @reshke in #2193
• Bump google.golang.org/grpc from 1.79.2 to 1.79.3 by @dependabot[bot] in #2196
• Add simple tests for protocol Close {statement, portal} by @reshke in #2194
• Bump github.com/go-ldap/ldap/v3 from 3.4.12 to 3.4.13 by @dependabot[bot] in #2195
• CreateSequence in tranMngr by @xelavopelk in #2181
• Support pg isolation tester util function in xproto. by @reshke in #2197
• Support ICP 'wait' action. by @reshke in #2200
• Cleanup & refactoring in docker configs by @EinKrebs in #2199
• Fix panic in MemQDB.CloseRouter method by @EinKrebs in #2201
• Support ICP-lock in pg_isolation_test_session_is_blocked by @reshke in #2202
• Support SPQR virtual function calls in legacy planner by @reshke in #2203
• Bump github.com/lib/pq from 1.11.2 to 1.12.0 by @dependabot[bot] in #2204
• Support empty from-list in pure virtual selects by @reshke in #2205
• Introduce show two_phase_tx for handy 2pc feature testing & visibility by @reshke in #2206
• Docs for __spqr__maintain_params hint by @Denchick in #2208
• Fix extensions setup on shards by @EinKrebs in #2209
• docs: create relation by @Denchick in #2210
• Fix wrong next_lower_bound calc in show key_ranges_extended by @Denchick in #2212
• Add isolation test for 2pc commit feature. by @reshke in #2213
• Enable recently-fixed steps in clustered regress suite by @xelavopelk in #2184
• Remove debug sleep in isolation tests by @EinKrebs in #2219
• fix issue #1321 by @futig in #2217
• Include Distribution.Relations to spqrguard setup by @EinKrebs in #2218
• Bump go.etcd.io/etcd/client/v3 from 3.6.8 to 3.6.9 by @dependabot[bot] in #2220
• Standardise key range validation error messages for clustered setup. by @xelavopelk in #2215
• regress test with noncluster mode became similar like clustered by @xelavopelk in #2214
• Fix qdb config in docker compose setup by @EinKrebs in #2222
• Enable "common" regress test suite in clustered setup by @xelavopelk in #2224
• proto_test.go: test for error in xproto by @rkhapov in #2225
• Support renaming relation distribution keys by @Ledatu in #2186
• proto_test.go: add test to error with flush by @rkhapov in #2226
• Add context and error to DCStateKeeper functions in qdb & refactor by @EinKrebs in #2230
• Fix multi-shard for datarow-returning prepared statements by @reshke in #2229
• Store two phase tx metadata storage shards in etcd by @EinKrebs in #2231
• Allow storing DCState in postgres by @EinKrebs in #2236
• Add SHOW support for router runtime settings by @diPhantxm in #2223
• Bump github.com/lib/pq from 1.12.0 to 1.12.1 by @dependabot[bot] in #2243
• Bump github.com/rs/zerolog from 1.34.0 to 1.35.0 by @dependabot[bot] in #2242
• Fix overwrite query for xproto by @reshke in #2245
• Add test for PR 2245 by @xelavopelk in #2246
• proto_test.go: test for parse error by @rkhapov in #2241
• Test 2pc transactions recovery by @EinKrebs in #2240
• Support UUID type for default shard by @diPhantxm in #2239
• Bump go.etcd.io/etcd/client/v3 from 3.6.9 to 3.6.10 by @dependabot[bot] in #2249
• Bump github.com/lib/pq from 1.12.1 to 1.12.2 by @dependabot[bot] in #2250
• Bump google.golang.org/grpc from 1.79.3 to 1.80.0 by @dependabot[bot] in #2247
• Refactor Session Parameter handling + Tests by @diPhantxm in #2238
• Split RFQ message computation and sending by @reshke in #2253
• Remove few instances of direct client ReplyErr by @reshke in #2254
• Separate error building function from client report utility by @reshke in #2255
• Take client tx status report out of ProcessMessage by @reshke in #2256
• Refactor common xproto test run logic. by @reshke in #2259
• Add more pipeline tests by @reshke in #2260
• Futher refactor ProcessMessage by @reshke in #2261
• Fix error path cleanup after 312a066 by @reshke in #2265
• Enhance testing bash by @reshke in #2263
• Fix expected output for regression tests by @reshke in #2267
• Fix regress test for clustered regress suite by @xelavopelk in #2268
• ci: cancel stale test runs on new push by @tmchow in #2264
• Remove old & unneeded hold route technique by @reshke in #2269
• Add test for pipelining with unnamed statement by @reshke in #2271
• Fix panic on incorrect prepared statement in feature test by @xelavopelk in #2258
• Bump github.com/lib/pq from 1.12.2 to 1.12.3 by @dependabot[bot] in #2273
• proto_test.go: PG-compatible run mode by @rkhapov in #2272
• proto_test.go: test pstmt usage after simple query by @rkhapov in #2275
• Fix executor reset for unavailable shard by @reshke in #2278
• Move common connection acquirition logic to util func by @reshke in #2277
• Use bootstrapConnection in all xproto test by @reshke in #2279
• Revert "ci: cancel stale test runs on new push (#2264)" by @reshke in #2280
• Add proto test for 677b599 by @reshke in #2281
• docs: more accurate time_quantiles setting description by @Denchick in #2282
• Do not require v2 for reference relation routing by @reshke in #2283
• Fix index out of range panic in Update/DeleteKeyRangeMove by @EinKrebs in #2103
• relay.go: fix pstmt not exist error msg by @rkhapov in #2284
• Bump github.com/jackc/pgx/v5 from 5.8.0 to 5.9.1 by @dependabot[bot] in #2221
• Support NO WAIT for RETRY TASK GROUP by @diPhantxm in #2248
• Small refactoring for InitExecutionTargets by @reshke in #2285
• Add reference relation storage declaration to routing unit tests by @reshke in #2286
• Move reference relation combine logic by @reshke in #2287
• Fixed gossip-message processing for meta transaction by @xelavopelk in #2237
• docs: restructure faq.md by @Denchick in #2288
• check-spelling@v0.0.26 by @Denchick in #2290
• Add ALTER SYSTEM sql command. by @reshke in #2294
• Silence linter by @reshke in #2295
• Add basic regression test for schema-qualify router logic by @reshke in #2296
• Do not show SPQR admin console greeting if not configured. by @reshke in #2297
• Collect relations modified by query & small refactor by @reshke in #2300
• Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.22.0 to 1.43.0 by @dependabot[bot] in #2302
• feat: add shard TLS parser and proto primitives by @Ledatu in #2303
• Do not display greeting in regress by @Denchick in #2301
• Collect read-only query relations, and route only on them by @reshke in #2304
• Add ASCII art logo to admin console greeting 🔥💌 by @Denchick in #2293
• Try to toggle defaults without failing v1 by @reshke in #2274
• Add RETRY TASK GROUP ALL by @diPhantxm in #2299
• Persist shard TLS config and add UpdateShard RPC with hardened router sync by @Ledatu in #2105
• Fix after 9c5623f by @reshke in #2306
• Fix setting up VM for benchmarks by @diPhantxm in #2308
• Fix docstring in (*Conn),Auth by @EinKrebs in #2307
• Fix for strict_operators routing by @reshke in #2309
• Bump golang.org/x/crypto from 0.49.0 to 0.50.0 by @dependabot[bot] in #2312
• Do not tolerate column or relation ambiguity by @reshke in #2310
• Do not move data on redistribute check by @EinKrebs in #2311
• Fix panic on concurrent access to qdb.Distribution maps by @EinKrebs in #2315
• Use TTS for SPQR GUC report rather that immediate client interaction. by @reshke in #2316
• Add generic boolean GUC machinery. by @reshke in #2317
• Add regression test for post-processing feature. by @reshke in #2318
• Add --display_greeting CLI option by @reshke in #2319
• Make COPY FROM STDIN work correctly in xproto. by @reshke in #2320
• Add regression test for hashed insert routing by @reshke in #2321
• Implement __spqr__route_key function. by @reshke in #2322
• Refactor and Fix shard tls tests by @diPhantxm in #2314
• docs: add default value column by @Denchick in #2325
• Remove SPQR params logic duplication by @reshke in #2324
• Ignore pg repo link in spell checker by @EinKrebs in #2327
• proto_test.go: DISCARD/DEALLOCATE tests by @rkhapov in #2329
• Add basic regress test for 2pc by @reshke in #2328
• proto_test.go: DEALLOCATE PREPARE by @rkhapov in #2330
• HELP SHOW by @Denchick in #2331
• Simple support for DEALLOCATE [PREPARE] / DISCARD [ALL] by @reshke in #2333
• Fix: Issue #1764 - reject unreachable shard hosts on CREATE SHARD and AddDataShard by @Daewoow in #2178
• test: Create route_test.go with test TestSetParams by @AnastasiaKostareva in #2252
• proto_test.go: split deallocate test by @rkhapov in #2336
• Add test for reference relation lock on transfer by @EinKrebs in #2326
• Remove all usages of ParserState by @reshke in #2335
• Fix hint parsing in relay by @EinKrebs in #2337
• HELP for distribution commands by @Denchick in #2334
• Fix SQL-level PREPARE support by @reshke in #2339
• Bump lyx by @reshke in #2340
• Bump & pin spqrguard version by @EinKrebs in #2338
• Support SQL-level EXECUTE by @reshke in #2341
• Forbid external reference table modification via spqrguard by @EinKrebs in #2292
• Change TimeQuantiles type to string & deprecate TimeQuantilesStr by @EinKrebs in #2342
• qdb meta transaction infra for CreateSequence by @xelavopelk in #2167
• Fix unused-parameter linter issues by @Denchick in #2344
• Handle shard config as generic options by @diPhantxm in #2323
• Bump github.com/docker/go-connections from 0.6.0 to 0.7.0 by @dependabot[bot] in #2348
• Add tests for task group retry after some QDB syncs by @EinKrebs in #2346
• introduce batch limit for meta tran batch by @xelavopelk in #2216
• Add regression test for PREPARE/EXECUTE support by @reshke in #2345
• Bump github.com/jackc/pgx/v5 from 5.9.1 to 5.9.2 by @dependabot[bot] in #2352
• Partially undo e88aebd and fix. by @reshke in #2353
• Fix generic bool GUC name reporting by @reshke in #2354
• Fix deadlock by @reshke in #2349
• proto_test.go: add Flush test by @rkhapov in #2355
• Remove 'xproto-regress' from needs in workflows by @Denchick in #2358
• Bump github.com/rs/zerolog from 1.35.0 to 1.35.1 by @dependabot[bot] in #2360
• Add two-phase commit transactions recovery isolation test by @EinKrebs in #2357
• Fix panic: set tx meta storage if shard data is not empty by @diPhantxm in #2359
• Use __spqr__run_2pc_recover function in mempgqdb.feature by @EinKrebs in #2362
• Parametrized benchmarks by @diPhantxm in #2361
• Add options to SHOW SHARDS output by @diPhantxm in #2350
• Remove aggregated rps stats by @diPhantxm in #2364
• Automatically clean up committed & rejected tx metadata from postgres by @EinKrebs in #2363

New Contributors

• @Minetoff made their first contribution in #2114
• @Daewoow made their first contribution in #2172
• @Ledatu made their first contribution in #2177
• @rkhapov made their first contribution in #2225
• @tmchow made their first contribution in #2264
• @AnastasiaKostareva made their first contribution in #2252

Full Changelog: 2.9.8...2.9.9