pypi genvarloader 0.36.0
v0.36.0

latest releases: 0.43.2, 0.43.1, 0.43.0...
4 months ago

v0.36.0 — The Rust Migration

This is the largest release since 2.0. GenVarLoader's entire core read and write path has been ported from Python/numba to a self-contained Rust crate behind a thin PyO3 (abi3) binding, and numba has been removed from GVL's own code entirely. Python keeps only the ergonomic surface — Dataset indexing, torch integration, validation and error messages — and dispatches into Rust for everything else.

The work landed as Phases 0–5 of the Rust migration roadmap (50+ commits, ~30k lines, mostly differential-parity tests and docs). Every kernel was ported under a strict contract: implement in Rust → differential-test byte-identical against the old numba implementation on property-generated inputs → flip the default and delete the numba version. main stayed shippable at every step.


⚠️ Breaking changes

On-disk dataset format → 2.0 (migration required)

Track intervals are now stored struct-of-arrays (starts.npy / ends.npy / values.npy sharing offsets.npy) instead of array-of-structs. This lets the memmaps cross the Python→Rust boundary zero-copy, and fixes an out-of-memory defect at sample-scale where the old layout forced a full contiguous copy on every batch.

Datasets written by gvl < 0.36 must be migrated. Opening one now raises:

ValueError: Dataset at <path> uses format version 1.x but this genvarloader
expects 2.0.0. Run `genvarloader.migrate('<path>')` to upgrade it in place.

Migration

import genvarloader as gvl

gvl.migrate("path/to/dataset")   # in place, then open as usual

gvl.migrate is in-place, streaming, and crash-safe (peak extra disk is a single track's interval store; metadata is bumped last via atomic replace). It's idempotent — a no-op on an already-2.0 dataset — and leaves genotypes, regions, and the reference untouched. No re-write() needed.

Other breaking changes

  • GVL_BACKEND removed. The numba/rust runtime backend switch and _dispatch.py are gone — Python now calls Rust directly. If you set GVL_BACKEND anywhere, drop it (it's a no-op now).
  • Unsupported track types now raise TypeError at write time (the dead custom IntervalTrack write path was removed).

✨ New

  • gvl.migrate(path) — the in-place 1.x → 2.0 dataset migrator described above (new public API).
  • Self-contained Rust crate. The core data structures and algorithms are now a cargo test-able Rust crate (114 Rust tests), usable from Rust directly and built on the pyo3-free seqpro-core Ragged layout — no reimplementation in GVL.
  • Rust-accelerated variant-windows assembly — an entire Python assembly pass for windowed/bare-allele variant output now happens in a single Rust call.

⚡ Performance

Final single-thread rust-vs-numba Dataset.__getitem__ A/B (the last apples-to-apples comparison before numba was deleted; chr22_geuv, single thread). Rust is parity-or-better on every output mode:

Output mode rust ÷ numba
tracks-only 1.07×
haplotypes / tracks+seqs 1.66×
annotated 1.43×
variants 1.38×
variant-windows 4.58×

Highlights:

  • Five fused __getitem__ kernels (plain / annotated / spliced / annotated-spliced haplotypes, plus fused tracks) each cross the FFI boundary exactly once.
  • gvl.write() / update() are fully Rust-backed: single-pass streaming bigWig writer and a COITrees-based table/annotation overlap engine. The bigWig write slice is ~1.88× faster with ~28% less allocation than the legacy path.
  • Rayon batch parallelism on every read kernel, gated byte-identical to the serial result and only engaged above a size threshold (production-scale batches, e.g. SEQLEN ≥ 131072 or BATCH ≥ 256).
  • GVL no longer contributes the ~3 GB llvmlite JIT cost it used to (note: seqpro still pulls numba transitively for now — its own numba removal is upstream).

📦 Dependencies

  • Requires seqpro ≥ 0.20 (Rust-backed Ragged) and the seqpro-core 0.1.0 crate.
  • awkward dropped from the hot paths.

🔬 Correctness notes

  • The Rust port is byte-identical to the previous numba implementation across the differential-parity suite. Two narrow sub-domains are intentionally excluded from the parity oracle because numba was the buggy side there (the #242-family start >= contig_len interval clip, and a reconstruct trailing-fill under-write) — Rust is correct in both, and these are now covered by direct oracle tests.
  • Verification at release: 973 pytest passed / 44 skipped / 5 xfailed / 0 failed; 114 cargo tests passed; ruff, pyrefly, and clippy clean.

Full changelog (conventional commits)

Feat

  • rayon: parallelize get_diffs_sparse + intervals_to_tracks (C3)
  • rayon: parallelize shift_and_realign_tracks_sparse and tracks_to_intervals (Task C2)
  • rayon: parallelize reconstruct_haplotypes_from_sparse with rayon batch parallelism
  • delete numba backend — rust-only read path (Phase 5 W5)
  • rust: fuse annotated+spliced haplotype reconstruction into one FFI crossing (Phase 5 W3)
  • register rc_alleles dispatch (rust default, seqpro reference)
  • rust: rc_alleles PyO3 wrapper + registration
  • rust: rc_alleles_inplace core for variant-allele RC
  • rust: debug_assert to_rc mask length in kernel RC blocks
  • fold strand RC into rust kernels; numba post-pass retained as oracle
  • rust: optional in-kernel RC for spliced haplotype kernel
  • rust: optional in-kernel RC for annotated haplotype kernel
  • rust: optional in-kernel reverse for track realign kernel
  • rust: optional in-kernel RC for reconstruct_haplotypes_fused
  • rust: optional in-kernel RC for get_reference
  • rust: in-place reverse/reverse-complement primitives for read path
  • py: assemble_variant_buffers numba oracle, rust shim, and dict parity harness
  • ffi: assemble_variant_buffers_{u8,i32} pyfunctions
  • variants: assemble_windows_mode (token windows + bare alleles)
  • variants: assemble_variants_mode (alt/ref bytes + flank tokens)
  • variants: add fetch_windows reference-read helper
  • variants: add tokenize/slice_flanks/assemble_alt_window cores
  • ffi: zero-copy boundary guard for sample-scale memmaps
  • migrate: add gvl.migrate for 1.x AoS -> 2.0 SoA
  • open: gate dataset open on format_version major
  • format: store track intervals as struct-of-arrays (gvl 2.0)
  • intervals: route intervals_to_tracks through backend dispatch (default rust)
  • ffi: Rust intervals_to_tracks + ffi seam module
  • utils: route splits_sum_le_value through backend dispatch (default rust)
  • ffi: Rust splits_sum_le_value + ffi seam module
  • dispatch: backend registry for Rust migration strangler window
  • ragged: route to_padded through seqpro-core Rust bridge
  • rust: consume seqpro-core via rlib; add ragged_to_padded bridge

Fix

  • rayon: debug_assert offset monotonicity in C1 carve; correct test comment
  • env: keep conda numba pin (seqpro needs working libllvmlite); guard stays own-code
  • threads: remove conditional numba import; update thread tests for pure-OS detection
  • test: drop stale _recon_mod.intervals_to_tracks spy (B2 removed that import)
  • test: restore generate_goldens regeneration; clean dead GVL_BACKEND in bench conftest (W5 B1)
  • reconstruct,tracks: pad full tail in numba trailing-fill on ref overshoot
  • reconstruct: pad full tail when ref exhausted, not from index 0
  • test: add init.py to disambiguate test_write collision; ruff fmt
  • variants: drop unused ArrayView2 import
  • indexing: SpliceIndexer.parse_idx double-applies sample-subset map
  • intervals: clip sub-query interval starts in both kernels (#242)
  • tracks: clamp writable_ref when deletion extends past track end
  • reconstruct: guard non-annotated parity test against numba SystemError; correct rayon-deferral comment
  • reconstruct: strengthen SAFETY comments; rename batch test to match serial-only impl
  • reconstruct: clamp writable_ref when ref_idx past contig end; skip numba annotated flake
  • reference: revert padded_slice leniency — mirror numba's loud failure for start>=clen (parity twin)
  • test: update stale _gather_v_idxs_ss import after Task 5 rename; lint/docstring cleanup
  • variants: gather_rows must preserve data dtype (dosage/custom fields)
  • bench: profile variants variable-length (with_len is meaningless for variants)
  • stub: sync genvarloader.pyi — bigwig_intervals rename + intervals_to_tracks
  • test: drop stale rv._rag access in flat_mode_equivalence after Ragged subclass refactor
  • dispatch: guard isinstance(Ragged) sites for RaggedVariants subclass

Refactor

  • delete numba kernels; numpy fallbacks for #231 dtype paths
  • backend: B2 — collapse backend-conditional branches; delete GVL_BACKEND/_active_backend
  • dispatch: B1 — replace all get() call sites with direct rust calls, delete _dispatch
  • rust: extract reverse::rc_row shared helper
  • write: delete dead legacy track path + splits_sum_le_value
  • drop unreachable spliced variant-RC guard
  • route variant-allele RC through dispatched rc_alleles kernel
  • genotypes: delete dead filter_af kernel + its dead test (superseded by inline numpy)
  • variants: RaggedVariants subclasses Ragged, drop _rag composition

Perf

  • rust: fuse rc_alleles_inplace — 186→308 instrs (rc_row inlined), drop Vec alloc + rescan
  • rust: tune reconstruct_haplotypes_from_sparse — 2839→1279 instrs, 0.655→0.589 rust÷numba
  • rust: tune rc_flat_rows_inplace — 212→283 instrs (vectorized), 0.664→0.635 rust÷numba
  • rust: tune assemble_alt_window — 518→727 asm lines (memcpy-expanded), 1.146→0.835 ms/batch
  • rust: tune slice_flanks — 389→429 total instrs (hot-path: byte-loop→memcpy), 2.115→1.136 ms/batch
  • rust: tune shift_and_realign_tracks_sparse — 3 do_slice calls→0 in hot path
  • rust: tune tokenize — 16→4 hot instr/elem, 0.55→0.43 rust÷numba
  • rust: tune intervals_to_tracks — 480→283 instrs, 0.628→0.624 rust÷numba
  • intervals: paint tracks via raw contiguous slice
  • variants: route windows/variants assembly through one rust call
  • ffi: skip zero-init of fully-overwritten fused output buffers
  • haps: cache FFI-ready sub-linear per-variant arrays
  • reconstruct: fused spliced-haps / annotated-haps / haplotypes getitem kernels
  • tracks: fused tracks getitem kernel
  • intervals: port tracks_to_intervals RLE numba→rust (default rust)
  • tracks: port shift_and_realign_tracks_sparse + apply_insertion_fill (4 strategies) + PRNG
  • reconstruct: port reconstruct_haplotypes_from_sparse (singular + batch)
  • reference: port get_reference + padded_slice numba→rust
  • variants: port _fill_empty / _compact_keep / _gather_alleles / gather_rows (dtype-preserving)
  • genotypes: port choose_exonic_variants + get_diffs_sparse numba→rust

Don't miss a new genvarloader release

NewReleases is sending notifications on new releases.