teaguesterling/plinking_duck v0.4.1

v0.4.1 — Biobank-scale bind performance

on GitHub

Biobank-scale bind performance

Region-filtered read_pfile random-access queries now bind in ~milliseconds instead of ~seconds at biobank scale. Measured end-to-end on synthetic 7M samples × 1.5M variants with parquet companion files:

Projects to sub-100ms bind at the 170M-variant imputation panel scale. No SQL API changes.

What changed

• Columnar VariantMetadataIndex — the internal pvar representation is now typed vectors (chroms, positions, ids, refs, alts) instead of a synthesized #CHROM\tPOS\t… text buffer that had to be re-parsed. Adds a chrom_offsets index for O(log N) region lookup and a sparse vidx_map + local_to_vidx mode for pushdown.
• Parquet region pushdown — when region := is set and the pvar is parquet, the bind issues SELECT … WHERE "#CHROM" = $1 AND POS BETWEEN $2 AND $3 with file_row_number so only matching rows are materialized. Uses row-group statistics pruning + column projection.
• Smart psam load — when the query doesn't need IID strings (no VARCHAR sample filter, no genotypes := 'columns'/'struct', not orient := 'genotype'/'sample'), the loader returns a sample count via parquet footer metadata in O(num_row_groups) and skips the ~600ms of string copies.
• Lazy iid_to_idx — built only on first VARCHAR sample lookup, with duplicate-IID detection moved to the lazy-build path.
• Binary-search region/CPRA lookup — ParseRegion, ResolveByCpra, and the read_pfile effective-variant-list loop all use chrom_offsets + std::lower_bound instead of linear scans.
• Env-gated bind timing probes — PLINKING_BIND_PROFILE=1 emits phase timings to stderr. Zero overhead when off.

Correctness hardening

• Text .pvar/.bim parser rejects truncated/malformed lines with a clear InvalidInputException (file + byte offset + missing fields) instead of silently pushing default-initialized columns.
• BuildChromOffsets detects and throws on non-contiguous chromosome runs in generic SQL-view sources (previously silently dropped). PLINK-format files are unaffected.
• SampleInfo::EnsureIidMap throws with an actionable message when called against a count-only sample load (indicates an internal bug), instead of a confusing downstream "sample not found".
• Region-pushdown SQL uses a prepared statement for user-controlled values; the chrom column (#CHROM vs CHROM) is detected via upfront schema introspection instead of error-driven retry.

Other readers

plink_ld's pairwise lookup now goes through a shared public BuildVariantIdIndex, which is also sparse-safe. plink_score's region-restricted ID map uses columnar access.

Full Changelog

v0.4.0...v0.4.1