figwig 0.2.0

figwig 0.2.0

on Python PyPI

figwig 0.2.0 adds the figwig file format: any number of tracks over one genome in one file, read many windows at a time from any subset of them, with converters from bigWigs and BAMs, a per-cell converter for fragments files, and a Claude Code skill that covers it. The bigWig reader and writer are faster, and figwig bam2bw gains a coverage mode. For anyone who reads training data from many tracks, or turns reads into count tracks.

Commits

• 787cc7e FIX the coverage mode's changelog entry moves from the released 0.1.0 section, where PR #21 appended it, to 0.2.0
• 758ec5a RELEASE bump version to 0.2.0 with changelog entries
• b02f0c7 DOC the first-call times after the kernels pass constants as plain values, and the changelog entry
• 8b10618 UPDATE the kernels pass constants to each other as bool() and int(), so numba compiles one copy of a callee per type rather than one per literal value: a quarter fewer specializations to compile on a first call
• 8261395 FIX CI compiles the kernels for a generic CPU, so numba's cache loads on any runner of an OS
• 4d467ab FIX the commit-time step uses touch -t, which macOS's touch takes too
• ee48d7b FIX the CI numba cache key is the hash of the kernel files with no fallback, since numba does not notice a constant or kernel used from another file changing
• 9caaa5f UPDATE the test suite runs in a quarter of the time and CI keeps numba's cache: bam2bw's tests run the command in the test process, the damage test samples the chunks, pytest-xdist in the dev extra and on every runner core
• 836c987 DOC the skill, docs and README follow the dense-path and converter-check PRs: the defaults for dense float tracks, the BAM header and bw2fw sizes checks where the commands are described, and six corrected sentences
• 7fb5d57 FIX figwig bam2bw warns, rather than refuses, when a BAM's header disagrees with the sizes, keeping bam2bw 0.5.1's discarding of reads outside them
• ec4ed35 ADD figwig bam2fw and bam2bw check each BAM's header against the sizes: a shared chromosome of another length, or no shared chromosome, is refused before any read
• 11fc36e UPDATE dense float tracks read 3 to 6 times faster and write up to twice as fast on many threads: runs chunks decode in place, and read batches and write tasks are cut per thread
• c91109c FIX bw2fw refuses a sizes file sharing no chromosome with a bigWig, an empty track round-trips through fw2bw and bw2fw, and a bad BAM is a message from bam2fw and bam2bw
• 474f3be UPDATE the statistics' minimum and maximum are those of the nonzero bases, each summary stream carries a CRC-32, and the specification gains worked chunks and the rules a clean-room reader asked for
• 70a1c17 DOC the figwig file format, version 1: a specification, a reader written from it, a Kaitai Struct definition and tests against the files the package writes
• 86cbd8f UPDATE the bigWig writer plans each batch on its own thread and encodes and compresses it on the pool, and confirms sorted input in one compiled pass
• 653f352 ADD engine='isal' on the bigWig reader: ISA-L's inflate through python-isal's extension, from numba without the GIL, after a one-block self-check
• 58f7f90 ADD tests of the coverage mode's edges: the run decoder, every BED reader, pooled and separate files, shifts that clip or empty reads, small write batches, empty inputs and the command line
• d097dcc DOC the coverage mode against deepTools on ten experiments: the same value at every base, 3 to 8 times faster
• d0ebdee FIX the coverage helpers are private names
• 16d45b4 ADD figwig bam2bw -c and bam2fw -c, the coverage of every aligned base, as deepTools counts it
• 6bc259a DOC the per-cell conversion's numbers after the third optimization pass
• 1260cd7 UPDATE chunk CRCs in one numba call per batch, and entry batches of at least a million entries bounded by their tiles
• b6c4473 UPDATE the cell scanner inflates a unit in one call, joins chromosomes on the pool, and the writer folds a batch's statistics in one pass
• 66accbb DOC the per-cell conversion's numbers after the optimizations, every method timed again at a load of 31 to 40
• cb2946e UPDATE the entries encoder visits only the tracks a tile touches
• dcf9114 UPDATE the cell scanner finishes each chromosome on the pool, and the writer's presorted path makes no pass per entry
• 3cd0603 UPDATE the cell scanner skips an unlisted barcode in the kernel, so the scan gains to 8 threads
• 3610bcc UPDATE bam2fw --barcode-column hands the writer its entries in the encoders' order, which the writer then does not sort
• c8eb849 UPDATE bam2fw --barcode-column groups a file's cells within a chunk's limit, hands on 32-bit arrays, and is timed against pandas, scipy and SnapATAC2
• 71ee8cb FIX the key-packing test builds its expectation per sampled chromosome, not a 56 GB array
• 874208f FIX bam2fw --barcode-column refuses a truncated gzip or BGZF file with a message, and tests every malformed case
• 1b0fbbb ADD figwig bam2fw --barcode-column, a fragments file as one track per cell
• da379e8 UPDATE the skill as a router to operations, with recipes for whole jobs and no machine-bound timings
• 4c4b4ae DOC unify the README, docs, changelog and skill after the format work
• 3375917 Merge pull request #17 from jmschrei/figwig_format
• 5549e12 DOC changelog entry for the zoom-record kernel holding the open record in locals, and its comment in the present tense
• e27aa29 Keep the open zoom record in locals in _zoom_records
• b1abdc7 DOC list every figwig bam2bw argument in a README dropdown
• ad3da25 Update README.md

figwig files

New — the figwig file (.fw) holds the per-base values of any number of tracks over one genome, with a JSON record of attributes per track and the name of the genome, and reads many windows at a time from any subset of its tracks into one (windows, tracks, width) array in the tracks' own dtype. FigWigReader, read_figwig, FigWigWriter and write_figwig mirror the bigWig API with a tracks= argument; select(**attributes) gives the tracks whose records match, stats each track's sum, nonzero bases, minimum and maximum over its nonzero bases, and written_by the figwig that wrote the file. Version 1 of the format is specified and frozen on the format page of the docs, with tests/format_reference.py, a reader written from the page with numpy and zlib alone, and docs/figwig.ksy, a Kaitai Struct definition, both run over files of every layout; a reader refuses a file of a later version or with a required feature it does not know.

New — layout. Every track lies on a grid of tile bases, and the tiles of a group of tracks are one chunk: nothing when every value is zero, else dense, sparse, runs or a per-tile matrix, whichever is smallest, in the narrowest element type that holds the values exactly, with a one type that stores no values when every nonzero value is 1 and a float64 type for an integer above 2**24 beside a fraction. Chunks are compressed with zlib or libdeflate on n_jobs threads, or stored raw when deflate would grow them (level=0 stores every chunk raw). The index holds only the tiles with values, so a file of 100,000 single-track groups opens in a tenth of a second. order='tracks' stores each track's nonzero bases in blocks per chromosome instead, for reading a few tracks over the genome; figwig reorder rewrites a file in the other order.

New — reads. read(..., sparse=True) returns the nonzero bases as a SparseWindows of entries, labels= sums the tracks per label (cells into pseudobulks), bins= sums each window per bin of bases, read_entries(tracks, chroms) returns every nonzero base of tracks over chromosomes, and channels=/items= arrange the tracks read by two attributes into (windows, items, channels, width). A read of mixed tracks comes back in the narrowest type that holds every track exactly, and a dense read whose output would pass 8 GiB raises instead of allocating it. sums(chroms, starts, ends, tracks=None, nan='propagate', paired=False) sums intervals of any lengths; nan='skip' leaves NaN bases out and nan='count' gives the bases that are not NaN, so a mean over covered bases as pyBigWig and deepTools give it is one over the other; paired=True sums interval i over track i alone. tile_sums gives each track's sum per tile from the file's summaries, and tile_sums(level=k) the zoom levels a file written with zooms= keeps, without decoding a chunk.

New — writes. write takes windows or intervals of whole groups, write_sparse(chroms, positions, tracks, values) nonzero bases as entries, both encoded and compressed on the n_jobs threads. A track's dtype is declared (uint8 to uint32, int8 to int32, float32, float64) or inferred; a float32 track keeps float32's rounding, and a value beyond float32, a 64-bit integer beyond 2**53 or an infinite value is refused before anything is written. The file is written beside its path and moved there at close, so a failed write, a writer never closed (which warns) or a conversion over its own input leaves the path as it was. FigWigWriter(path, None, tracks, append=True) adds tracks in place under the file's lock, writing the new chunks after its end and the header last, so a writer that fails or is killed leaves the file readable as it was; add_file and figwig merge copy the tracks of files written with the same chromosomes, tile, level and order.

New — integrity. Every chunk carries a CRC-32, checked with verify=True; the header, the tables and each index record carry CRC-32s checked at open, and every count and pointer is bounded against the file before its bytes are read, so a damaged, crafted or truncated file raises ValueError rather than a MemoryError or a crash. A reader checks on every read that the file at its path is the one it opened, so a file replaced by a conversion raises instead of reading with a stale index.

New — device="cuda" on read, sums and read_figwig lands the output in that GPU's memory as a cupy.ndarray, which PyTorch, JAX, TensorFlow and CuPy take without a copy through DLPack; the cuda extra adds cupy. Sparse tracks are scattered on the device from their entries, dense tracks copied from a pinned host buffer kept on the reader.

Faster — dense float tracks (conservation, fold change) read 3 to 6 times faster and write up to twice as fast on many threads: the runs chunks the writer chooses for them decoded at 70 ns a change, and now fill a window's row in place at 3 ns; read batches are cut per thread and dense write calls into one task per thread. Three chr22 conservation tracks, 8,192 windows of 2,048 bp: 0.50, 0.07 and 0.03 s on 1, 8 and 32 threads of the benchmarks' server, against 0.80, 0.12 and 0.11 s from their bigWigs.

Faster — the first call compiles a quarter fewer kernel specializations: a constant one kernel passed to another was typed by numba as a literal and compiled a copy of the callee per value; the kernels now pass constants as plain values. A process that reads and writes every way compiles for about 85 s where it compiled for 116 s.

Converters and commands

New — figwig bw2fw, fw2bw, bam2fw, merge, reorder, info. bw2fw writes bigWigs as the tracks of one figwig file, named after their files or by a metadata table whose other columns become the tracks' attributes, with --missing nan for methylation and conservation tracks whose zeros are measurements, --append to add bigWigs to a file and -z zoom levels; fw2bw writes a file's tracks back as bigWigs with such a table, so the two round-trip names and attributes. bam2fw takes what figwig bam2bw takes and writes each input file as its own pair of stranded tracks, or one unstranded track. info prints what a file holds, and -b the bytes its chunks, tables and unused space take. Every command refuses to write over an output that exists unless -f is given, writes beside its path and moves the output into place when complete, and prints a message rather than a traceback for an input that is missing, not a BAM or cut short.

New — figwig bam2fw --barcode-column N writes a fragments file as one track per cell, named by the barcode in column N, with --cells listing the cells to keep and their attributes and --groups the cells per group, scanned in BGZF units on the -p threads. The 257.6 million fragments of the 10x PBMC experiment become its 10,246 called cells in 9 s on 8 threads of the benchmarks' server, 5.5 s on 32 and 60 s on one, against 140 s for a pandas recipe and 217 s for SnapATAC2.

New — -c/--coverage on figwig bam2bw and figwig bam2fw counts every aligned base of each read (the CIGAR's M, = and X runs, as pysam's get_blocks() and deepTools' bamCoverage count them), or every base of each BED/tsv interval, in place of one end; -ps/-ns move a read's first base and its end and -r divides by the reads. On the 2.4 GB ATAC-seq BAM and ten more experiments the bigWig holds the same value at every base as bamCoverage --binSize 1, 3 to 8 times faster; the 2.4 GB BAM takes 3.4 s on 8 threads against deepTools' 14.9 s.

Fix — bam2fw and bam2bw check each BAM's header against the sizes before reading: a shared chromosome of another length, or no shared chromosome, is refused by bam2fw and reported on stderr by bam2bw, which goes on as bam2bw 0.5.1 does. An hg19 BAM against hg38 sizes placed every read at its hg19 position in a file of hg38 lengths with nothing said, and sizes of other names wrote a file of zeros.

Fix — bw2fw -s refuses a sizes file that shares no chromosome with a bigWig, naming both sides' chromosomes, where it wrote an empty file with a warning and exited 0.

Fix — an empty track survives the round trip through fw2bw and bw2fw: BigWigReader opens a bigWig without a data index, which BigWigWriter writes for a track with no values, and reads every window as missing, where it refused the file as unreadable.

Reading

Fix — natural chromosome order. The bigWig reader takes a file whose index leaves lie in natural chromosome order (chr1, chr2, ..., chr10) where the ids run chr1, chr10, chr2, by sorting the leaves; two bigWigs of the test collection were refused as unsorted where pyBigWig read them.

New — keep_nan= on BigWigReader.read and read_bigwig reads an interval whose value is NaN as NaN rather than missing.

New — engine='auto' | 'zlib' | 'isal' on read and read_bigwig. With python-isal 1.8 built on ISA-L 2.31.1 installed (the new isal extra, which the bam2bw extra includes), 'auto' inflates data blocks through ISA-L from the numba kernel without the GIL, so no compiler is needed, and only once one known block inflated through it has given zlib's bytes. Every engine gives the same values. Reading the benchmarks' ATAC-seq and DNase-seq bigWigs takes 1.3 to 1.5 times less on 1 to 8 threads. Thanks @YaoxiangLi!

Writing

Fix — zoom levels on threads. The kernel that summarizes items into a zoom level holds its open record in locals instead of writing the level's state arrays for every item; levels summarized on different threads could share a cache line, which made the kernel many times slower in the processes where the arrays happened to be allocated next to each other. The files written are unchanged. Thanks @YaoxiangLi!

Faster — the threaded writer. With threads, BigWigWriter plans each batch of items on a thread of its own, encodes and compresses the blocks on the pool, and folds the batch into the file's summary on another thread, so the calling thread only takes the next call's values; sorted input with valid values is confirmed in one compiled pass. The files written are unchanged byte for byte; writing the benchmarks' tracks with libdeflate and no zoom levels on 8 threads takes 1.4 to 1.5 times less, and peak memory without zoom levels falls by 3 to 12%. Thanks @YaoxiangLi!

Claude Code skill

Updated. The skill covers figwig files and the converters, routes by task and by symptom, lists every error message in references/errors.md, and references/recipes.md walks through whole jobs: a training set from many bigWigs or BAMs, genome-wide predictions, a single-cell file, summaries over peaks, a lossless signal round trip and reads as tracks. Re-run figwig install-skill --force to pick it up.

Tests

The test suite runs in a quarter of the time, on several cores with uv run pytest -n 8, and CI keeps numba's compiled kernels between runs.

What is now possible

Previously figwig read and wrote bigWigs, one track per file. A figwig file holds hundreds of tracks, or every cell of an experiment, in one file read in windows from any subset of them, summed per label or per interval, or straight onto a GPU, and bam2fw and bw2fw build one from reads or from bigWigs. Back-compatible: the bigWig API only gains keywords (keep_nan, engine), every bigWig written by 0.1.0 reads as before, and the files written are unchanged byte for byte.