v0.36.0 — The Rust Migration
This is the largest release since 2.0. GenVarLoader's entire core read and write path has been ported from Python/numba to a self-contained Rust crate behind a thin PyO3 (abi3) binding, and numba has been removed from GVL's own code entirely. Python keeps only the ergonomic surface — Dataset indexing, torch integration, validation and error messages — and dispatches into Rust for everything else.
The work landed as Phases 0–5 of the Rust migration roadmap (50+ commits, ~30k lines, mostly differential-parity tests and docs). Every kernel was ported under a strict contract: implement in Rust → differential-test byte-identical against the old numba implementation on property-generated inputs → flip the default and delete the numba version. main stayed shippable at every step.
⚠️ Breaking changes
On-disk dataset format → 2.0 (migration required)
Track intervals are now stored struct-of-arrays (starts.npy / ends.npy / values.npy sharing offsets.npy) instead of array-of-structs. This lets the memmaps cross the Python→Rust boundary zero-copy, and fixes an out-of-memory defect at sample-scale where the old layout forced a full contiguous copy on every batch.
Datasets written by gvl < 0.36 must be migrated. Opening one now raises:
ValueError: Dataset at <path> uses format version 1.x but this genvarloader
expects 2.0.0. Run `genvarloader.migrate('<path>')` to upgrade it in place.
Migration
import genvarloader as gvl
gvl.migrate("path/to/dataset") # in place, then open as usualgvl.migrate is in-place, streaming, and crash-safe (peak extra disk is a single track's interval store; metadata is bumped last via atomic replace). It's idempotent — a no-op on an already-2.0 dataset — and leaves genotypes, regions, and the reference untouched. No re-write() needed.
Other breaking changes
GVL_BACKENDremoved. The numba/rust runtime backend switch and_dispatch.pyare gone — Python now calls Rust directly. If you setGVL_BACKENDanywhere, drop it (it's a no-op now).- Unsupported track types now raise
TypeErrorat write time (the dead customIntervalTrackwrite path was removed).
✨ New
gvl.migrate(path)— the in-place 1.x → 2.0 dataset migrator described above (new public API).- Self-contained Rust crate. The core data structures and algorithms are now a
cargo test-able Rust crate (114 Rust tests), usable from Rust directly and built on the pyo3-freeseqpro-coreRaggedlayout — no reimplementation in GVL. - Rust-accelerated variant-windows assembly — an entire Python assembly pass for windowed/bare-allele variant output now happens in a single Rust call.
⚡ Performance
Final single-thread rust-vs-numba Dataset.__getitem__ A/B (the last apples-to-apples comparison before numba was deleted; chr22_geuv, single thread). Rust is parity-or-better on every output mode:
| Output mode | rust ÷ numba |
|---|---|
| tracks-only | 1.07× |
| haplotypes / tracks+seqs | 1.66× |
| annotated | 1.43× |
| variants | 1.38× |
| variant-windows | 4.58× |
Highlights:
- Five fused
__getitem__kernels (plain / annotated / spliced / annotated-spliced haplotypes, plus fused tracks) each cross the FFI boundary exactly once. gvl.write()/update()are fully Rust-backed: single-pass streaming bigWig writer and a COITrees-based table/annotation overlap engine. The bigWig write slice is ~1.88× faster with ~28% less allocation than the legacy path.- Rayon batch parallelism on every read kernel, gated byte-identical to the serial result and only engaged above a size threshold (production-scale batches, e.g. SEQLEN ≥ 131072 or BATCH ≥ 256).
- GVL no longer contributes the ~3 GB llvmlite JIT cost it used to (note:
seqprostill pulls numba transitively for now — its own numba removal is upstream).
📦 Dependencies
- Requires
seqpro≥ 0.20 (Rust-backedRagged) and theseqpro-core0.1.0 crate. awkwarddropped from the hot paths.
🔬 Correctness notes
- The Rust port is byte-identical to the previous numba implementation across the differential-parity suite. Two narrow sub-domains are intentionally excluded from the parity oracle because numba was the buggy side there (the
#242-familystart >= contig_leninterval clip, and a reconstruct trailing-fill under-write) — Rust is correct in both, and these are now covered by direct oracle tests. - Verification at release: 973 pytest passed / 44 skipped / 5 xfailed / 0 failed; 114 cargo tests passed; ruff, pyrefly, and clippy clean.
Full changelog (conventional commits)
Feat
- rayon: parallelize get_diffs_sparse + intervals_to_tracks (C3)
- rayon: parallelize shift_and_realign_tracks_sparse and tracks_to_intervals (Task C2)
- rayon: parallelize reconstruct_haplotypes_from_sparse with rayon batch parallelism
- delete numba backend — rust-only read path (Phase 5 W5)
- rust: fuse annotated+spliced haplotype reconstruction into one FFI crossing (Phase 5 W3)
- register rc_alleles dispatch (rust default, seqpro reference)
- rust: rc_alleles PyO3 wrapper + registration
- rust: rc_alleles_inplace core for variant-allele RC
- rust: debug_assert to_rc mask length in kernel RC blocks
- fold strand RC into rust kernels; numba post-pass retained as oracle
- rust: optional in-kernel RC for spliced haplotype kernel
- rust: optional in-kernel RC for annotated haplotype kernel
- rust: optional in-kernel reverse for track realign kernel
- rust: optional in-kernel RC for reconstruct_haplotypes_fused
- rust: optional in-kernel RC for get_reference
- rust: in-place reverse/reverse-complement primitives for read path
- py: assemble_variant_buffers numba oracle, rust shim, and dict parity harness
- ffi: assemble_variant_buffers_{u8,i32} pyfunctions
- variants: assemble_windows_mode (token windows + bare alleles)
- variants: assemble_variants_mode (alt/ref bytes + flank tokens)
- variants: add fetch_windows reference-read helper
- variants: add tokenize/slice_flanks/assemble_alt_window cores
- ffi: zero-copy boundary guard for sample-scale memmaps
- migrate: add gvl.migrate for 1.x AoS -> 2.0 SoA
- open: gate dataset open on format_version major
- format: store track intervals as struct-of-arrays (gvl 2.0)
- intervals: route intervals_to_tracks through backend dispatch (default rust)
- ffi: Rust intervals_to_tracks + ffi seam module
- utils: route splits_sum_le_value through backend dispatch (default rust)
- ffi: Rust splits_sum_le_value + ffi seam module
- dispatch: backend registry for Rust migration strangler window
- ragged: route to_padded through seqpro-core Rust bridge
- rust: consume seqpro-core via rlib; add ragged_to_padded bridge
Fix
- rayon: debug_assert offset monotonicity in C1 carve; correct test comment
- env: keep conda numba pin (seqpro needs working libllvmlite); guard stays own-code
- threads: remove conditional numba import; update thread tests for pure-OS detection
- test: drop stale _recon_mod.intervals_to_tracks spy (B2 removed that import)
- test: restore generate_goldens regeneration; clean dead GVL_BACKEND in bench conftest (W5 B1)
- reconstruct,tracks: pad full tail in numba trailing-fill on ref overshoot
- reconstruct: pad full tail when ref exhausted, not from index 0
- test: add init.py to disambiguate test_write collision; ruff fmt
- variants: drop unused ArrayView2 import
- indexing: SpliceIndexer.parse_idx double-applies sample-subset map
- intervals: clip sub-query interval starts in both kernels (#242)
- tracks: clamp writable_ref when deletion extends past track end
- reconstruct: guard non-annotated parity test against numba SystemError; correct rayon-deferral comment
- reconstruct: strengthen SAFETY comments; rename batch test to match serial-only impl
- reconstruct: clamp writable_ref when ref_idx past contig end; skip numba annotated flake
- reference: revert padded_slice leniency — mirror numba's loud failure for start>=clen (parity twin)
- test: update stale _gather_v_idxs_ss import after Task 5 rename; lint/docstring cleanup
- variants: gather_rows must preserve data dtype (dosage/custom fields)
- bench: profile variants variable-length (with_len is meaningless for variants)
- stub: sync genvarloader.pyi — bigwig_intervals rename + intervals_to_tracks
- test: drop stale rv._rag access in flat_mode_equivalence after Ragged subclass refactor
- dispatch: guard isinstance(Ragged) sites for RaggedVariants subclass
Refactor
- delete numba kernels; numpy fallbacks for #231 dtype paths
- backend: B2 — collapse backend-conditional branches; delete GVL_BACKEND/_active_backend
- dispatch: B1 — replace all get() call sites with direct rust calls, delete _dispatch
- rust: extract reverse::rc_row shared helper
- write: delete dead legacy track path + splits_sum_le_value
- drop unreachable spliced variant-RC guard
- route variant-allele RC through dispatched rc_alleles kernel
- genotypes: delete dead filter_af kernel + its dead test (superseded by inline numpy)
- variants: RaggedVariants subclasses Ragged, drop _rag composition
Perf
- rust: fuse rc_alleles_inplace — 186→308 instrs (rc_row inlined), drop Vec alloc + rescan
- rust: tune reconstruct_haplotypes_from_sparse — 2839→1279 instrs, 0.655→0.589 rust÷numba
- rust: tune rc_flat_rows_inplace — 212→283 instrs (vectorized), 0.664→0.635 rust÷numba
- rust: tune assemble_alt_window — 518→727 asm lines (memcpy-expanded), 1.146→0.835 ms/batch
- rust: tune slice_flanks — 389→429 total instrs (hot-path: byte-loop→memcpy), 2.115→1.136 ms/batch
- rust: tune shift_and_realign_tracks_sparse — 3 do_slice calls→0 in hot path
- rust: tune tokenize — 16→4 hot instr/elem, 0.55→0.43 rust÷numba
- rust: tune intervals_to_tracks — 480→283 instrs, 0.628→0.624 rust÷numba
- intervals: paint tracks via raw contiguous slice
- variants: route windows/variants assembly through one rust call
- ffi: skip zero-init of fully-overwritten fused output buffers
- haps: cache FFI-ready sub-linear per-variant arrays
- reconstruct: fused spliced-haps / annotated-haps / haplotypes getitem kernels
- tracks: fused tracks getitem kernel
- intervals: port tracks_to_intervals RLE numba→rust (default rust)
- tracks: port shift_and_realign_tracks_sparse + apply_insertion_fill (4 strategies) + PRNG
- reconstruct: port reconstruct_haplotypes_from_sparse (singular + batch)
- reference: port get_reference + padded_slice numba→rust
- variants: port _fill_empty / _compact_keep / _gather_alleles / gather_rows (dtype-preserving)
- genotypes: port choose_exonic_variants + get_diffs_sparse numba→rust