Merged PRs

dolt

• 12130: docker: bypass provisioning on cluster standbys and fail fast on invalid init SQL
  Fix #12115
• 12128: docker: skip entrypoint setup on container restart when initialized
  Fix #12115
• 12108: kvexec: allow NULL in the constant lookup key of an index lookup join
  Fix #12106
• 12092: Authenticate cluster role headers before applying transitions
  Fixes #12090.
• 12091: Fix standalone remotesrv path traversal
  • Reject traversal and malformed repository paths in RPC requests and sealed HTTP file requests.
  • Use a rooted filesystem for standalone remote store directory creation, and verify the resolved store path stays inside the configured root.
  • Validate upload table file IDs before converting them to hashes.
    Fixes #12089.
• 12088: go: gcctx: GCSafepointController: Explicitly support concurrent Waiters.
  This adds support for concurrent GC runs against different databases on the same server. Each GC run captures the outstanding sessions that are present when it starts and blocks on visiting only their roots. The sessions themselves can keep track of more than one outstanding waiters.
  Explicit semantics are added for sessions which are themselves within a GC Safepoint Waiter, in order to prevent a potential deadlock. While the Waiter is outstanding, new Waiters which come to wait on that session can immediately visit its roots, just like the sessionAwareSafepointController would have done for that session itself when creating its own Waiter.
• 12086: merge_base overhaul
  • Add new dolt_merge_bases() table function, allowing for multiple merge bases
  • Add --all flag to dolt merge-base cli operation.
  • Existing dolt_merge_base() behavior remains unchanged.
    A significant amount of testing git and it's merge-base --all behavior led to the testing harness added here. the go/libraries/doltcore/commitgraph/merge_bases_test.go is effectively the git behavior of
    merge base codified.
    Behavior of merge-base with shallow clones (ghost commits) was incomplete, and errored in situations it should not have. This area is much more thoroughly tested now.
    Related: #12050
• 12077: schema, diff, dtables: keep history across same-key table recreate
  • Diff accepts retagged primary keys (same names and types, new column tags) instead of warning, so history survives a recreate.
    Fix #12067
• 12075: Stop embedded SQL engine workers and clean up primary binlog streams
  Closing an embedded SQL engine now stops and joins its branch-activity tracker and primary binlog manager, including when engine construction fails. Primary shutdown interrupts blocked replica writes and waits for active streams; disconnecting a replica preserves other streams, and stream exit closes the current rotated binlog file.
  Based on #12068 by Alex Giurgiu; the original contributor commits and authorship are preserved.
• 12066: Stop keyless index producers on Close
  Closing a keyless secondary-index scan before it is drained could leave its producer goroutine blocked on a full channel. The scan state and its expanded duplicate rows then stay reachable after the caller stops reading.
  Close now cancels an iterator-owned child context and waits for the producer, without cancelling the caller's context. No public API, SQL syntax or configuration changes.

  Test

  TestKeylessIndexIterCloseStopsProducer inserts ten identical keyless rows, reads one, and closes the iterator inside testing/synctest, which fails if the producer is still blocked when the test ends. It fails without the fix and passes with it.

  cd go && go test -run '^TestKeylessIndexIterCloseStopsProducer$' -count=1 ./libraries/doltcore/sqle/index

  The test shows the producer stops. That Close also waits for it comes from the code (eg.Wait()), not from the test.

  Example

  In a manual check (not part of this PR) with a keyless table k(a BIGINT, b BIGINT), index idx_b(b) and 20 rows per value of b:
  • SELECT a FROM k FORCE INDEX (idx_b) WHERE b = 3 LIMIT 1 left one blocked producer before the fix and none after.
  • A correlated subquery with LIMIT 1 over 50 outer rows left 50 blocked producers before the fix and none after.

  Checked

  • The index package tests, normal and race; the new test ran 50 times under race.
  • TestIndexes, TestIndexPrefix and TestGeneratedColumns, normal and race.
  • go vet, copyright headers and check_fmt.sh.
    In one of our environments, TestQueries/select_length(compress(repeat('a',_1000))) fails (expected 24, actual 21) on both main and this branch. It is outside the index path. We did not run the full Go or SQL engine suites, and we have not measured how often this happens in production.

  ---

  Generated by Perfloop. Human sponsor: Tomás Senart.
• 12062: gitauth: fix Windows FETCH_HEAD lock contention, dedup write fetches, and omit auth hints for file errors
  Fix #11904
• 12061: dbfactory: init the git remote cache repository with the remote's object format
  Fix #12060
• 12057: SyncRoots: return the source root read error instead of reporting success
  Fixes #12056.
  Close #12072
  SyncRoots returned nil when srcDb.NomsRoot failed, so DOLT_BACKUP sync, sync-url and restore reported success without transferring the source root. This returns the wrapped error instead, matching how the destination root read just below already handles its error.
• 12052: merge: detect conflicting generated column additions
  When both branches add the same generated column, report a schema conflict if its AS (...) expression or VIRTUAL/STORED mode differs, instead of silently keeping one definition. Identical additions still merge cleanly.
  Compare these properties in both column equality methods and add merge tests for each case.
  Fixes #11367
• 12049: more dolt optimizations to help dumbodb
  Three smallish changes:
  • capture hash of object rather than recalculating
  • allocate in one shot in ReadBytes
  • reuse unchanged leaves in adaptive encoding
• 12047: go/store/nbs: journal_file.go: Use .bss allocated zeros slice for the write ahead.
  This lets us write the zeros without interacting with allocation or GC and without the allocated array needing to RSS (outside the VMM mapping overhead).
• 12041: Return ErrTableAlreadyExists for existing nonlocal tables
  Fix #12039
• 12037: bats: qualify pk in auto_increment_lock_modes check query
  test1 and ranges both have a pk column, so the unqualified min(pk) and max(pk) are ambiguous. go-mysql-server b4f13d62c36d rejects them, failing the test on every run since #12035.
• 12034: Accelerate lookup joins with SQL row inputs
  Accelerates lookup joins with SQL row inputs using shared iteration and type-aware key conversion.
  Fixes #11636
  Companions: dolthub/doltgresql#3554 and dolthub/go-mysql-server#3997.
• 12031: perf: Use fdatasync on linux with a padded journal
  fdatasync avoid unnecessary metadata updates to the file system, resulting in ~25% improvement in write performance.
• 12027: dolt version: cache failed release checks to prevent hanging
  • Add 3s HTTP timeout for latest release fetch.
  • Cache failed checks under version check file to avoid a second HTTP timeout.
    Fix #12021
• 11928: Support atomic multi-branch transactions and commits
  Enables atomic updates across branch working sets and heads with dolt_commit_all and the dolt_multi_branch_commit setting.
  Companion Doltgres PR: dolthub/doltgresql#3615

go-mysql-server

• 4041: Clear sign on negative zero during DecimalMod
  Fixes #12124
  This is done in DecimalDiv, but it seems like the step was skipped for DecimalMod.
• 4040: Fix off-by-one error when finding aggregated column during replaceSortedMinMax
  Fixes #4037
• 4034: Set orderByScope.cols properties for SetOps using that of the merged columns
  Fixes #11714
  Based on #3977
• 4032: Fix nullsafe logic for tuples containing null values
  Previously, left and right elements in tuples were converted to a common type before comparing.
  For DATETIME types, certain values would be converted to NULL when they did not need to be converted at all.
  Fixes: #12109
• 4029: sql: Server lifecycle, background work and test harnesses support EngineVariables.
  Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.
  • Server lifecycle, background work and test harnesses support EngineVariables.
  • Migrate remaining engine variable consumers.
  • Keep global-only variables live outside session overlays.
  • Bind collation wire harness sessions to their engines.
  • Make COMPRESS assertions independent of the zlib byte stream.
• 4028: sql: SQL planning and execution access EngineVariables through Context/Session.
  Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.
• 4027: sql: Engine can carry EngineVariables and they are exposed on the Context and Session.
  Part of a body of work to remove the global SystemVariables and StatusVariables registries and make them state on the Engine itself.
• 4026: sql: Add factories for independent registries of SystemVariables and StatusVariables.
  Part of a body of work to remove these globals and make them state on the Engine itself.
• 4024: Fix panic for JSON_PATHs pointing to null objects and null slices
  This PR adds regression tests for JSON_EXTRACT where the JSON_PATH points to a nil object and attempts to slice into it.
  Depends on: dolthub/jsonpath#8
  Fixes: #12103
• 4022: sql/types: build the JSON_EXTRACT result array only from paths that match
  Fix #12107
• 4021: Do not remove GroupBy node when replacing sorted MIN/MAX aggregates
  Based on #4014
  Fixes #12093
  MIN/MAX aggregates without an explicit GROUP BY should still return a single NULL row when no rows match. For such cases when MIN/MAX was aggregating over a primary index, we were returning no rows; this was because in replaceAgg, we were removing the GROUP BY node, which was still needed to return the correct results via the groupByIter.
  This PR does the following:
  • renames replaceAgg to replaceSortedMinMax to more accurately reflect what the rule does
  • preserves GROUP BY node
  • have replaceSortedMinMax return a LIMIT 1 over an IndexedTableAccess instead of a LIMIT 1 ORDER BY node. Because the GROUP BY node is preserved, the replaceIdxSort rule no longer replaces the previously created LIMIT 1 ORDER BY node with a LIMIT 1 over an IndexedTableAccess¹. Instead, replaceIdxSortHelper must be called directly within replaceSortedMinMax.
• 4020: Handle locate case when str length is 0 and position is greater than 0
  fixes: #12102
• 4015: Support conditional column drops in ALTER TABLE plans
  Carries column-level IF EXISTS into drop plans and skips missing columns during schema validation. Adds coverage for existing/missing columns, repeated drops, and missing-table errors.
  Uses Vitess main following the merge of dolthub/vitess#501. ALTER TABLE and analyzer tests pass. Part of dolthub/doltgresql#3536.
• 4014: Return a NULL row for MAX/MIN without GROUP BY when no row matches
  MAX/MIN without GROUP BY returned no row instead of one NULL row when no row matched, whenever replaceAgg could read the aggregated primary key column in index order:

  SELECT MAX(i) FROM emptytable;                            -- no row
  SELECT MAX(pk3) FROM three_pk WHERE pk1 = 2 AND pk2 = 0;  -- no row since #3907

  replaceAgg replaced the GroupBy with Limit(1) over a Sort, so empty input produced no row. It now keeps the GroupBy and puts the Limit(1) over the Sort beneath it, so the scan still reads a single row when the index provides the order. replaceIdxSort doesn't descend into a GroupBy, so replaceAgg converts that Sort to an index scan itself. The 13 changed plans are the existing MAX/MIN plans, each gaining the GroupBy.
  The empty-input queries added to QueryTests fail without the fix. Reported in #12093.
• 4012: Match MySQL Punctuation for DATETIME string parsing
  This PR drops the regexes used for DATETIME string parsing in favor of regular logic.
  The new implementation covers many missing warnings, addresses several edge cases, and handles additional delimiters missing from the previous implementation.
  Additionally, fixes some wire tests that were previously silently failing.
  Fixes:
  • #11750
  • #11939
• 4009: End non-recording spans in NewSpanIter.
• 4008: Make Handler.doQuery start and end the root query span. Remove Context.RootSpan and deprecate WithRootSpan.
• 4007: Stop ending the query in the process list when the top-level row iter is closed. The caller of BeginQuery owns EndQuery.
• 4005: Generate correct GetField expressions when optimizing joins on table aliases with pruned columns.
  This PR fixes two related but distinct panics.
  We had some duplicated logic when optimizing joins, where each implementation had a different bug that was fixed in the other.
  In convertSemiToInnerJoin, we generated GetField expressions that used the table name of the underlying table, not the table alias. This could cause these expressions to not resolve correctly.
  In convertAntiToLeftJoin, we used the pruned table schema instead of the full table schema when calculating ColumnIds, resulting in a potential out-of-bounds access.
  Dolt PR: #12082
  Doltgres PR: dolthub/doltgresql#3589
• 4003: Estimate a join of two empty estimates as empty on every architecture
  A join whose sides both have a row count and a distinct count of 0, as filtered reads of tables without statistics are estimated, made the selectivity 1/0 = +Inf and the cardinality 0 * +Inf = NaN. Converting NaN to uint64 is implementation-defined: arm64 gives 0, amd64 gives 1<<63. So on amd64 such a join was estimated at 2^63 rows, a hash or lookup join over it was priced in the 1e16s, and a nested-loop join won; on arm64 the same statement planned a hash join.
  The distinct count is now at least 1, so the cardinality is 0 on both, as arm64 already estimated it. The test fails on amd64 without the fix.
• 4001: Preserve aggregate input column IDs for HAVING
  Preserve GetField column IDs when adding aggregate inputs so HAVING retains grouping columns and evaluates the correct values. Fixes the incorrect result exposed by #3797.
  Adds the minimal repro and related ordinary/prepared regressions, with a TODO to migrate them after #3990. Engine regressions and planner, analyzer, row execution, and aggregation tests pass; SQL expectations match MySQL and PostgreSQL.
• 4000: Have nested stored procedures reuse existing transaction
  Nested stored procedure calls should not create a new transaction. Instead their changes are applied to the existing transaction.
  Fixes: #11950
• 3998: Prevent subquery filter pushdown from substituting set-returning functions into scalar predicates
  Filters on SRF outputs now remain above row expansion, while safe predicates still push down and preserve correlation tracking.
• 3997: Fix extended-type hash key conversion
  Fixes extended-type hash joins by converting keys into the comparison type.
  Companions: #12034 and dolthub/doltgresql#3554.
• 3995: Fix SIGN to handle signed floating points < 0.5 and string truncation
  Fix dolthub/go-mysql-server#3988
• 3994: Improve TIME parsing
  This PR further improves TIME parsing/conversions to more closely match MySQL's behavior.
  fixes: #10000
• 3993: Moved the external function provider into the builder overrides
  This moves the ExternalFunctionProvider so that integrators can set it via overrides. We also add a call to it so that integrators are able to use functions from their provider in column defaults.
• 3992: Added view recursion detection
  This fixes an OOM issue that occurs from recursive views.
• 3991: Optimize wildcard-only LIKE filters
  Wildcard-only LIKE filters such as LIKE '%' and LIKE '%%%' now simplify to IS NOT NULL for string operands. The rewrite only runs when LIKE is the full filter expression, preserving NULL behavior in nested expressions.
  This also covers non-UTF-8 strings and constant-folded patterns.
  Tests:
  • go test -tags=gms_pure_go ./sql/expression -count=1
  • go test -tags=gms_pure_go ./sql/analyzer/... -count=1
  • go test -tags=gms_pure_go ./enginetest -run '^TestCharsetCollationEngine$' -count=1
    Fixes #3943
• 3989: Infer precision for TIME function
  based off of: dolthub/go-mysql-server#3986
  partially addresses: #11944
• 3985: Add regression test for ValueRowIter
  This PR adds a flag to QueryTests that asserts that the generated row iterator supports the ValueRowIter interface. This interface is typically more performant than the traditional RowIter, so we want to make sure that we don't regress on queries that use it.
  Dolt PR: #12029
  Doltgres PR: dolthub/doltgresql#3549
• 3977: Sort set operation ORDER BY by the output column
  Fixes #11714
  A set-op ORDER BY that named a projected column re-evaluated the left branch's expression, rather than reading the set op's output column. buildSortConditions only rewrites aliases, so an unaliased literal, or a function such as abs(-1000), sorted by a constant. This change points those terms at the output column, typed with the merged column type. Terms that are already a GetField are left unchanged, so existing plans don't change.
  The expected results in the new script test come from MySQL 8.4.11. Without the fix, 10 of the 16 assertions fail. The rest are guards, including select 10 union select '9' order by 1, which has to keep sorting as strings. go test ./sql/... ./enginetest/... passes.
  There are two related problems on main that I left out of scope:
  • select a from t union select 'x' order by 1 still sorts the table column with its own int type. MySQL compares as strings here.
  • select a, a from t union select b, 1 from u order by 2 sorts by column 1, because both projections share one column id.
• 3916: Prune table columns beneath HAVING
  Allow projection pruning to traverse HAVING while retaining the columns needed by grouping, aggregates, and joins. HAVING expressions with subqueries continue to keep the outer table unpruned.
  Part of #11923
• 3893: Fix window expression identity collisions
  Use a structural identity when deduplicating window expressions so distinct resolved columns with colliding rendered names are retained independently.
  The identity includes expression structure and resolved field IDs plus window partition, order, NULL-ordering, and frame semantics, while excluding the assigned root output ID.
  Fixes #11855
• 3891: Fix implicit peer frames for window functions
  Implicit ordered LAST_VALUE, BIT_AND, BIT_OR, and BIT_XOR did not consistently use the default RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW peer semantics.
  Use the resolved peer group for LAST_VALUE and the shared aggregate default-frame path for the bitwise aggregates. Regression coverage includes peer rows, the exact running BIT_OR case, explicit-equivalent frames, descending order, and compound ordering. The new assertions match MySQL 8.0.45 and PostgreSQL 15.17.
  Fixes #11856
  Fixes #11857
• 3890: Fix nullable RANGE frame boundaries
  Nullable numeric RANGE frames could include unrelated rows when NULL boundary comparisons disagreed with the window's ordering. Preserve the effective NULL placement in RANGE framers and use it when scanning boundaries, supporting both MySQL's NULL-first and PostgreSQL's NULL-last ascending defaults.
  Regression coverage checks bounded nullable frames and explicitly verifies sorted rows and peer intervals for all four direction/NULL-rank combinations. The shared SQL results match real MySQL 8.4.11 and PostgreSQL 15.17.
  Fixes #11858
• 3797: Resolve correlated aggregates at their outer scope
  Correlated aggregates in nested queries could be assigned to the syntactically inner query even when their arguments belonged to an outer query. This could evaluate an aggregate in the wrong scope and produce runtime type errors.
  This change determines aggregate ownership from resolved argument columns instead of AST nesting. It:
  • Tracks semantic query blocks separately from transient builder scopes.
  • Resolves aggregate arguments through the query-block chain while preserving normal name resolution and shadowing.
  • Registers each aggregate with the closest query that supplies one of its arguments.
  • Replaces correlated argument dependencies with the resulting aggregate dependency.
  • Applies the same ownership rules to GROUP_CONCAT and its ordering expressions.
  • Centralizes aggregate registration and dependency collection to keep IDs and references consistent.
    Coverage includes ordinary and prepared execution, nested and parenthesized queries, joins, CTEs, unions, alias shadowing, mixed local and outer arguments, ordered GROUP_CONCAT, and strict-grouping failures.
    The related grouped-window validation gap remains tracked in #3856.
    Fixes #3814

vitess

• 501: Represent DROP COLUMN IF EXISTS in the DDL AST
  Parses and formats ALTER TABLE ... DROP COLUMN IF EXISTS, with a column-level flag separate from table-level IF EXISTS. Adds parse/format round-trip coverage; the full parser test package passes.
  Part of dolthub/doltgresql#3536.
• 500: Preserve subquery input expressions in EXPLAIN queries
  Previously, because Explain parse nodes did not implement WalkableSQLNode, Vitess would not walk them to populate input expressions on subqueries. This let to cases where go-mysql-server would produce a different plan for a DESCRIBE PLAN query compared to the original query.

Closed Issues

• 12115: dolt-sql-server image: a demoted cluster node can't restart (entrypoint setup is rejected on a standby, and the exit is silent)
• 12124: negative dividend in mod yields incorrect negative results
• 11714: Top-level ORDER BY does not seem to be applied to unions
• 12109: Fix null safe equals for tuple comparisons
• 12106: Lookup join panics on a NULL literal key for a NOT NULL column
• 12107: JSON_EXTRACT panics when one of several paths matches nothing
• 12093: Ungrouped MAX/MIN over a primary-key column returns no row when nothing matches
• 12103: JSON_EXTRACT('{"a": null}', '$.a[0]') panics
• 12102: LOCATE panics on empty string
• 11505: Date-valued window arithmetic appends a zero-time suffix
• 7690: Support MySQL datetime parsing behavior
• 10399: TIMESTAMPDIFF should return null for invalid datetimes
• 12067: dolt diff cannot track beyond a 'drop & recreate' action with identical index definition
• 11750: Throw warning for non-standard DATE and DATETIME delimiters
• 11939: extend delimited datetime regex(es)
• 12060: git+https remote push fails with 'does not support this repository's hash algorithm' against SHA-256 git repos
• 12089: Standalone remotesrv Table-File Upload Endpoint Allows Unauthenticated Path Traversal File Writes on Dolt
• 12090: Dolt Cluster Replication Endpoint Processes Client-Controlled Role Headers Before JWT Authentication, Allowing Persistent Unauthenticated Primary Demotion
• 12056: SyncRoots returns success when reading the source root fails (DOLT_BACKUP sync/restore report success)
• 11367: Merge silently picks one expression when branches add the same generated column with different AS(...) definitions
• 11904: git+https remote: Windows push fails on FETCH_HEAD 'Permission denied'; redundant fetch fan-out, no --no-write-fetch-head, console per git spawn
• 11636: Query is 1.76x slower in Dolt than MySQL
• 11950: Stored procedure that CALLs another procedure inside an explicit transaction silently discards the caller's earlier writes
• 12039: CREATE TABLE IF NOT EXISTS fails for an existing nonlocal table on a non-target branch
• 10000: truncate strings when casting to time
• 12036: DESCRIBE PLAN {{query}} can result in a different plan than {{query}}
• 11858: Nullable numeric RANGE bounds expand to the whole partition
• 11855: Dolt merges two different window operands whose qualified-column strings collide
• 11857: Dolt BIT_OR ignores ORDER BY when selecting its window frame
• 11856: Dolt LAST_VALUE uses a ROWS-like implicit frame instead of the default RANGE peer frame
• 12021: dolt version blocks for 30 s or more on every call when api.github.com is slow or unreachable
• 4037: Panic index out of range in replaceAgg for ungrouped MIN/MAX of a non-key column when WHERE fixes the whole primary key
• 3943: Simplify col LIKE '%' to col IS NOT NULL in analyzer
• 3988: SIGN() returns 0 for non-zero values with magnitude below 0.5
• 3814: Plan correlated aggregates correctly across nested semantic scopes