v0.17.0 — a WHERE skips catalog cells it cannot reach; catalog discovery; no overwrite without force (breaking); v2.0.0-only
This release makes a catalog read open only the cells that a query can reach,
on both axes: a WHERE on CHROM/POS drops variant blocks and a WHERE on IID
drops sample blocks. It also discovers
.pfile.json from a plain prefix and
reads version 1 catalogs. It has two breaking changes. It builds for DuckDB
v2.0.0 only (the v1.5.6 line stays on v0.14.2).
BREAKING — no writer replaces existing output unless the COPY sets force (#94)
COPY ... (FORMAT pvar | pgen | psam | pfile) now errors at bind when its
output exists. Add force (or force true) to replace it. A refused COPY
touches no file.
- Sharded writes also refuse an existing
.pfile.json,
.pgen,
.pvar,
.psam, or shard tree
/.
- Why: a plain COPY over a sharded output replaced
.psam and left the
old catalog, so a later read labelled the old shards with the new
sample IDs.
BREAKING — ATTACH c.genotypes on a matrix is one row per variant (#98)
On a version 2 matrix, c.genotypes is now read_pfile(catalog := ...):
one row per variant with every sample, not one row per variant per
sample block. The genotypes column is a fixed ARRAY of the cohort's
sample count; uneven sample blocks no longer give a LIST.
- Version 1 catalogs and bare filesets: output is unchanged.
- A matrix with holes keeps the per-block rows.
New — catalog discovery and version 1 catalogs (#94)
- read_pfile('
') reads
.pfile.json when that file exists. The
catalog wins over a
.pgen beside it, as in ATTACH. So
read_pfile('out') now reads a sharded writer's output.
- catalog := false turns discovery off; catalog := true requires a
catalog. There is no discovery for a URL, a pathmacro: path, a glob
or a list.
- catalog := now reads version 1 catalogs (shard_by, shard_samples, a
single fileset) as well as version 2 matrices.
New — a catalog read opens only the cells a query can reach (P13-006)
- region := and integer samples := drop cells at bind (#96).
- A WHERE on CHROM (=, IN) and POS (=, <, <=, >, >=, BETWEEN), joined
with AND, drops variant blocks before any of their files is read,
directly and through ATTACH views (#98).
- A WHERE IID = ... or IID IN (...) drops sample blocks: in genotype
orient (#105), and in sample orient for every genotypes mode
(#108, #109). In sample orient this applies to a read with no
af_range, ac_range, include_genotypes or genotype_range.
- Every filter is still applied, so the rows equal a full read.
Measured on a 10 x 4 grid (30000 variants x 10000 samples, 4 threads):
a sample orient array read with WHERE IID = one sample went from
2.6 s to 0.11 s, and its peak memory from 587 MB to 54 MB.
Changed — a catalog read does less work at bind
- A catalog read in variant or genotype orient loads its .pvar files
when the query starts, not at bind (#97).
- A cell's .pgen header is opened only if the scan keeps the cell
(#103).
- Bind takes a cell's paths as the catalog writes them, with no
existence check (#107). DESCRIBE or PREPARE of a catalog read opens
no .pgen and checks no cell file; on a remote store it sends no
HEAD request for them.
- So an error in a cell's files, or a count that disagrees with the
catalog, now shows when the query runs, and only for a cell it reads.
- Sample orient (catalog read, no filter parameter): the genotype
matrix is read when the query starts, after the IID prune.
plinking_max_matrix_elements is checked against that pruned matrix,
so DESCRIBE over the limit no longer errors; the query does (#109).
Fixes
- A failed pfile COPY removes its staging files before the statement
returns (#95).
- A catalog read from a URL resolves its relative cell paths under
that URL (#106). Before, such a catalog could not be read.
- Builds against DuckDB main again (the advisory canary), through a
compat shim for the moved registered-state accessor (#101).
Known limitation
A catalog with a hole (an absent cell) still errors for the 2-D read,
because a 2-D read of a holed grid is not defined yet.