genvarloader 0.27.0

v0.27.0

on Python PyPI

Highlights: robust on-disk artifacts — gvl.write now creates datasets atomically under an advisory file lock, and Dataset.open validates dataset format version + structural integrity before use. FASTA caches move to a self-describing .gvlfa format that builds atomically, auto-migrates legacy .gvl caches, and auto-rebuilds when stale. Plus correct drop_last handling across all dataloader modes.

✨ Features

Atomic, concurrency-safe dataset creation
gvl.write builds into a private sibling temp directory and publishes via an atomic os.replace, so a destination directory is never observed half-written. A best-effort filelock lets parallel jobs sharing one destination avoid redundant rebuilds — correctness rests on the atomic rename; the lock is advisory only (new filelock dependency).

Dataset format versioning + integrity validation
Metadata now records a format_version, and Dataset.open validates both the version and structural integrity (file presence and sizes) before returning. An incompatible or corrupt dataset raises a clear ValueError instructing you to regenerate with gvl.write. Datasets do not auto-rebuild.

New .gvlfa FASTA cache
gvl.Reference.from_path now builds and reuses a self-describing, fingerprint-validated .gvlfa cache directory (and accepts a .gvlfa path directly). The cache is published atomically under a best-effort lock — concurrent builders sharing one reference are safe — and, unlike datasets, auto-rebuilds from its source when stale or missing. Legacy .fa.gvl caches are migrated in place by reusing their bytes.

🐛 Fixes

• to_dataloader(drop_last=...) now honored across all modes: drop_last is no longer forwarded to the underlying DataLoader in default mode (it was double-applied); buffered/double_buffered modes honor drop_last=False; and ChunkPlanner keeps the trailing partial batch.
• BatchSampler conflict warning: to_dataloader now warns when a BatchSampler overrides an explicit batch_size.
• .gvlfa migration guards against stale/truncated legacy bytes, and a format-too-new sibling cache now raises instead of silently downgrading.

🔧 Internals

• New atomic_dir directory-publish primitive (temp build + os.replace) underpins both dataset and FASTA-cache creation.
• _fasta_cache module: FastaCache models + fingerprinting, three-way source resolution, build/load/validity guards, legacy migration, and an ensure_cache orchestrator.
• Concurrency regression tests for atomic cache + dataset creation (closes #21); coverage for too-old format versions and the genotypes-without-ploidy branch.

Out of scope: genoray .gvi and pysam .fai/.gzi index files are created by those upstream libraries and are not covered by gvl's atomic/locked creation.

Feat

• _open: validate dataset format version + integrity on open
• _write: atomic dataset creation + format_version in Metadata
• _fasta_cache: publish cache atomically via atomic_dir + locked double-check
• _atomic: add atomic_dir directory-publish primitive
• fasta: use .gvlfa cache module and accept .gvlfa input
• _fasta_cache: add ensure_cache orchestrator and dispatch
• _fasta_cache: migrate legacy .gvl caches by reusing bytes
• _fasta_cache: add build, load, and validity guards
• _fasta_cache: add source hints and three-way resolution
• _fasta_cache: add FastaCache models and fingerprint

Fix

• torch: warn when BatchSampler overrides explicit batch_size
• torch: buffered modes honor drop_last=False
• torch: do not forward drop_last to DataLoader in default mode
• chunked: keep trailing partial batch in ChunkPlanner
• test_fasta: move mid-file imports to top (E402, CI lint)
• _fasta_cache: guard legacy migration against stale/truncated bytes
• _fasta_cache: raise on format-too-new sibling cache instead of silent downgrade

Refactor

• _write: use plain with-atomic_dir; restore warnings filter; add atomicity + format_version on-disk tests
• reference: build cache via ensure_cache, accept .gvlfa
• _fasta_cache: fix progress bar advance and tighten load status type