allelix 2.0.0

Allelix v2.0.0

on Python PyPI

Major version bump. New architectural anchor: VCF + gVCF support with a batched annotation pipeline tuned for WGS scale, plus position-keyed rsID resolution that makes the flagship feature actually work on real-world caller output. Internal CLI restructure into a multi-file package. Second FTDNA file shape (Illumina raw). A CI sentinel for ClinVar CLNSIG vocabulary drift.

Headline

• VCF + gVCF parser. Plain VCF, gVCF (GATK reference-confidence), and .vcf.gz all handled via the standard library — no pysam dependency on the base path. gVCF reference blocks (lines with <NON_REF> ALT and END= INFO) are skipped entirely. Verified on real GIAB HG002 WGS (4M variants) and a real GATK HaplotypeCaller gVCF (19.3M data lines, 99.77% reference blocks correctly skipped).
• Position-keyed rsID resolution. Real VCFs from variant callers (GATK HaplotypeCaller, DeepVariant) emit ID=. — no rsID. Pre-v2.0.0 this meant zero annotations on real WGS. v2.0.0 resolves rsIDs by (chromosome, position) through ClinVar, disambiguating co-located rows by user_alleles ⊆ {ref, alt} so anchor-base indels at clustered positions (CFTR/BRCA-class regions) don't silently misroute. Real GIAB HG002 GRCh38: 0 → 4,843 annotations.
• Multi-sample VCF handling. --sample <ID> selects the column; multi-sample without --sample fails loudly with the available list (truncated to 10 + "and N more" for 1000 Genomes-scale files).
• FTDNA Illumina raw parser for the tab-delimited variant of the FTDNA export (distinct from the comma-delimited CSV shape).
• R-4 ClinVar CLNSIG vocabulary drift CI test. Fast tier validates code consistency; slow tier queries the installed cache and fails on any unknown CLNSIG term. Catches vocabulary additions in ClinVar releases.
• PharmGKB rebranded to ClinPGx. Display surfaces (HTML/terminal/JSON), URLs, and docs use ClinPGx. Internal identifier Annotation.source == "pharmgkb" preserved for cache backward compatibility — existing user caches keep working unmigrated.

Added

• Position-keyed rsID resolution (ClinVarAnnotator.bulk_resolve_rsids) with (chromosome, position) index on clinvar_variants (existing caches self-heal via CREATE INDEX IF NOT EXISTS on first connection).
• Position-keyed enrichment fallback for gnomAD + AlphaMissense (bulk_lookup_by_position) for rsIDs whose rsid index entry is missing/sparse.
• High-value rsID matching runs post-resolution so resolved rsIDs are flagged the same as natively-rsID'd variants.
• VCF + gVCF parser (allelix/parsers/vcf.py).
• Multi-sample VCF handling (--sample <ID>).
• Optional [vcf-index] extras for pip install allelix — pulls pysam>=0.22 and enables a tabix-backed fast path for extract --snps on huge VCFs.
• FTDNA Illumina raw parser.
• Chromosome normalization helper (normalize_chromosome in parsers/_helpers.py).
• R-4 ClinVar CLNSIG vocabulary drift CI test.

Changed

• analyze's annotation pipeline restructured to two-phase batched flow (build-detection prelude → batched annotation at _BATCH_SIZE = 5000). At WGS scale (4–6M variants), this replaces millions of per-variant SQLite roundtrips with a few thousand chunked WHERE rsid IN (...) queries.
• Annotator interface gains batch_annotate(variants). Base class default loops over annotate(variant) so existing annotators work unchanged.
• PharmGKB rebranded to ClinPGx (display + URLs + license attribution + reports). Internal identifiers preserved for backward compatibility.
• CLI restructured from allelix/cli.py (1596 lines) to the allelix/cli/ package (8 files, 36–516 lines each). External API unchanged.

Fixed

• rsID resolver: indel-anchor disambiguation (ship-blocker, found in external review). The position-keyed resolver previously matched ClinVar rows by ALT only, silently misrouting at positions where co-located rows share an ALT but differ in REF anchor length (CA>C vs CAG>C, common in clustered indel regions). Subset-of-{REF, ALT} match closes the het case; ambiguous hom-alt and 1/2 multi-allelic now abstain rather than coin-flip. Real-data measurement on GIAB HG002 GRCh38: pre-fix 4,867 surviving annotations dropped to 4,843 after the fix — two rsIDs (rs34079299 IFNG, rs59439148 ALOX5; both anchor-base indels) were silently misrouted to a co-located neighbor's rsid before this fix.
• VCFs from variant callers now annotated (GH #8). Pre-fix, real GIAB HG002 GRCh38 benchmark VCF (4M variants, ID=. per GATK convention) produced 0 annotations against every rsID-keyed source. Pivoted the resolution filter to not v.rsid.startswith("rs") so positional synthetic IDs (1000 Genomes 22:10519265:CA:C) and COSMIC IDs also route through resolution.
• SampleNotFoundError and MultiSampleError messages truncate the sample list to the first 10 with "... and N more" tail. On 1000 Genomes (3,202 samples) the error previously dumped every sample ID verbatim.
• Build auto-detect failure now warns and recommends --build. When the input has no rsIDs AND no ##contig assembly tag, the pipeline used to silently default to GRCh37; now prints a yellow warning recommending an explicit --build.
• VCF _canonical_rsid lowercases the rs prefix on return; non-rs IDs (COSMIC) preserve case; colon-containing positional synthetic IDs return empty.

Known limitations

• rsID resolver abstains on two cases that can't be disambiguated from the Variant model alone: (1) 1/2 multi-allelic at multi-rsid positions; (2) 1/1 hom-alt at shared-anchor indel positions. Both abstain rather than silently mis-stamp. A future Variant.ref field (v2.1+) would resolve both.
• Strand-aware annotator-level carrier matching is deferred to v2.1+ (R-1 on the roadmap). compare and export plink use strand resolution; the rsID-keyed annotators don't. Real impact: a few-percent slice of rsIDs reported on minus-strand chips silently miss their ClinVar entry. ADR-0010 documents the deferral.
• GWAS enrichment numbers at multi-allelic positions can be wrong-allele because GWAS Annotation doesn't carry the user's alt (GWAS data is rsID-keyed). The GWAS carrier decision itself is correct; only the gnomAD/AlphaMissense/CADD scores stamped next to it are sometimes the highest-AF alt rather than the user's. Architectural fix in v2.1+ bundled with Variant.ref.

Test data asset

test_data.tar.gz is attached to this release (~1.27 GB). v2.0.0 additions to the v1.1.1 baseline live under test_data/real/vcf/:

Synthetic fixtures (small, deterministic):

• mock_vcf.vcf, mock_gvcf.g.vcf, mock_multisample.vcf, mock_vcf_rsidless.vcf — match the committed fixtures under tests/fixtures/ for convenience

Real-data WGS samples (all public-domain — NIST GIAB, Broad GATK test data, 1000 Genomes):

• HG002_GRCh38_benchmark.vcf.gz (+ .tbi) — full GIAB HG002 GRCh38 v4.2.1 benchmark, ~4.05M variants
• HG002_GRCh38_chr22.vcf.gz (+ .tbi) — chr22 subset of the above (~50K variants) for fast smoke tests
• HG002_GRCh37_benchmark.vcf.gz (+ .tbi) — full GIAB HG002 GRCh37 v4.2.1 benchmark, ~4.03M variants (exercises per-build dispatch + the v2.0.0 ClinPGx position-keyed resolver, since GIAB benchmarks ship without rsIDs)
• HG00187_gatkhc.g.vcf.gz (+ .tbi) — real GATK HaplotypeCaller gVCF (--emit-ref-confidence GVCF), GRCh37, 19.3M data lines with ~99.8% reference blocks — exercises the gVCF reference-block skip semantic at real scale
• thousandG_chr22.vcf.gz (+ .tbi) — chr22 of the 1000 Genomes high-coverage 3,202-sample phased VCF — exercises --sample <ID> selection on a real multi-sample input

test_data/real/vcf/README.md inside the archive documents each file's source and intended use.

scripts/fetch_testdata.sh still references the v1.1.1 asset URL on dial481/allelix; repointing to this release is the first v2.0.1 task.