Written by Claude Code on behalf of Jacob Schreiber.
figwig 0.3.1 adds figwig fiberbam2fw, which writes a Fiber-seq BAM as one figwig file with a track per read and channel, and fixes read_bigwig and figwig bw2fw stopping with Too many open files on more than about 1,000 bigWigs. It also adds a script that measures the README's table of read time, write time and file size by assay. Nothing in the public API or the file format changes.
Commits
- 094007b RELEASE bump version to 0.3.1 with changelog entries (#64)
- 7580c3d ADD a script that measures the README's table of read time, write time and size by assay (#63)
- 253a4c1 FIX a read of thousands of bigWigs opens each file when its first batch runs and closes it after its last (#62)
- 30370c3 ADD figwig fiberbam2fw, a Fiber-seq BAM into a track per read and channel (#61)
Converters and commands
New — figwig fiberbam2fw. Writes a coordinate-sorted Fiber-seq BAM as one figwig file with five tracks per read: the m6A and 5mC probabilities of its MM and ML tags, and its nucleosomes, MSPs and FIRE qualities (the ns, nl, as, al and aq tags, or MA and AQ), on reference coordinates, in groups of 1,024 consecutive reads of a channel. Each alignment's span, strand, mapping quality, flag, length, HP, RG, rq, ec, sample and per-channel call counts are stored as attributes. read(..., channels="channel", items="read") then gives (windows, reads, channels, width). The values match ft extract --all for every call and footprint of the 8,647 alignments of a K562 BAM, and htslib's reading of MM and ML at every probability. The 99,685 alignments of the large BAM convert in 13 s on 4 threads and 35 s on one, into 605 MB (886 MB as BAM), and the file does not depend on the thread count.
Speed — write_sparse. It skips its range check of declared integer dtypes when the values' dtype fits every one of them, and cuts a call's batches by a binary search for the end of each tile, where it divided every entry's position. The 99,685 reads of the Fiber-seq BAM, written by figwig fiberbam2fw as 498,425 uint8 tracks, take 8.5 s on 16 threads and 13 s on 4, where they took 15.6 s and 17 s. The file is the same to the byte.
Reading
Fix — files open at once. read_bigwig and BigWigReader.read opened every bigWig at the start of a call and held its descriptor to the end, so a read, or a figwig bw2fw of a group of 1,024 bigWigs, stopped with [Errno 24] Too many open files at the usual soft limit of 1,024. Each file now opens when its first batch runs and closes after its last, so a call holds about n_jobs descriptors at a time. Reads of 1, 3, 48 and 1,024 files on 1 and 8 threads took the same time as before, within 1% by the median of 7 runs, and returned the same values. A process at its limit of open files now raises an OSError that names the limit and ulimit -n.
Benchmarks
New — README table script. benchmarks/readme_table/run.py measures the rows of the README's table of read time, write time and file size by assay, against the tool most people use for each: pybigtools and pyBigWig for bigWigs, SnapATAC2 and a scipy matrix for scATAC-seq, a pysam loop and a scipy matrix for Fiber-seq, cooler and scool for Hi-C. --update-readme rewrites the table from benchmarks/readme_table/results.json; the README's numbers had been typed by hand from several pages. Each measurement is its own process, the median of three, and each row checks that figwig and its baseline return the same values.
What is now possible
Fiber-seq reads, with their m6A and 5mC calls and footprints, can be kept as figwig tracks and read in windows or summed per read, which 0.3.0 had no converter for. A read or bw2fw of more than about 1,000 bigWigs now runs at the default open-file limit where it failed before.
Fully back-compatible — no public API changes. The file format stays version 1, 0.3.0 and 0.3.1 read each other's files, and the only changed behaviour is the new error message for a process out of file descriptors.
🤖 Generated with Claude Code