teaguesterling/plinking_duck v0.11.0

v0.11.0 — Sharded FORMAT pfile (shard_by) + .howard catalog emit

on GitHub

Sharded FORMAT pfile + .howard catalog emit

Builds on v0.10.0's pfile writer with a sharded write mode and the first .howard catalog emit.

• shard_by '<col>' on COPY … TO (FORMAT pfile): a single ordered pass rolls one hive shard per key transition — out/<col>=<val>/data.{pgen,pvar} — with one shared .psam written at the root. Designed to mirror PARTITION_BY semantics without native partitioning (which reorders rows and would corrupt .pgen row order).
• .howard.json catalog written last, as the completion marker: per-file global half-open partition {start, stop, source_ordinal} and per-file genomic range (CHROM list + true running POS min/max), consumable by howard's .howard loader.
• Atomicity: per-shard staging, .pgen promoted last on each roll; the catalog-last write means a partial run leaves K durable shards and no catalog (readers treat a missing catalog as incomplete).
• Validation: path-unsafe shard_by key values are rejected with an actionable error; per-shard monotonicity/POS-extent reset per shard; a reappearing shard key across shards is an error.

Reviewed adversarially (PASS/PASS) and the write→open_howard→compare round-trip verified end-to-end. Sharded test: 48 assertions; full suite 10830 assertions / 178 cases.

🤖 Generated with Claude Code

https://claude.ai/code/session_011csELQ5cVyHrxMbvgBfC6o