figwig 0.3.0 on Python PyPI

figwig 0.3.0 is for anyone who keeps many tracks in one figwig file. A file now keeps the tile sums and zoom levels of every group whose sums fit 256 MiB, where 0.2.0 kept none once a file passed about 350 genome-wide tracks; an optional regional summary index lets a reader decode one region's summary without inflating a whole level; and reads, writes and the converters run faster on threads. It also fixes figwig bam2bw -c -r and bam2fw -c -r, which divided by aligned blocks rather than reads, and writes that could leave a file no reader could open. The format stays version 1: 0.2.0 reads the files 0.3.0 writes, and 0.3.0 reads 0.2.0's.

Commits

41e4b3d RELEASE bump version to 0.3.0 with changelog entries (#60)
3e9b8f7 DOC numbers measured again, statements fixed, and the README and changelog condensed (#59)
e34e2c5 FIX bam2bw -c through pysam's loop cuts the record kernel's blocks (#58)
b97ed42 UPDATE summaries and zoom levels are compressed by libdeflate where it compresses the chunks (#57)
12cb28f FIX writes of few count tracks on threads: the block copy runs at memory speed and small integers cut fewer tasks (#56)
db1c615 FIX bam2bw -c -r divides by reads, not by aligned blocks (#55)
dec8069 FIX writes that left unreadable files, bam2fw -c bytes on threads, sums with verify, and install-skill's error; take ISA-L 2.32.1 (#54)
72e3417 FIX first calls compile fewer copies of the read and write kernels (#53)
053da8f FIX a group of 2**29 values per chunk is refused, since its dense float64 chunk overflows the u4 size (#52)
891cd88 UPDATE each group keeps its summary past 356 tracks, and readers keep the tile sums they read (#51)
e900897 FIX fw2bw writes a group of more tracks than -p at most -p bigWigs at a time (#49)
d968815 UPDATE figwig writes and conversions scale better with threads (#48)
e6d549e UPDATE faster Hi-C and single-cell Hi-C reads and writes (#47)
ade50aa Merge pull request #44 from jmschrei/io_speed
e568164 DOC the changelog, the threads page and the skill describe faster reads and writes on threads
1cfc627 FIX float sums take the threads main gives them
8350dae FIX a read whose tasks are all of one group deals them to its threads
f896e9b FIX a few windows summed per bin keep every thread the read asks for
2b7fffe FIX shares are waited for on an interrupt, paired errors raise in batch order, fw2bw stops its reader
f029e5f FIX interval and edge tiles fold their statistics in the order of one thread
f6705b7 FIX paired sums run their batches side by side on at most 4 threads
de23ffe FIX reads of a few groups keep their groups' batches
4fd41dc UPDATE reads check for compiled kernels by numba's overloads
a7a30cc UPDATE interval writes do not copy their values when _copies is off
0933b0d FIX paired sums in track order run their batches one after another
ed0b64d UPDATE fw2bw reads the next piece while the writers take the last
e670fe4 UPDATE the bigWig writer lays out a window's items without a branch per base
158f8e0 UPDATE reorder into track order hands the writer its entries as sorted
72b7f44 UPDATE track-order writes sum tiles in one pass over sorted positions
2472809 UPDATE sealing a tile uses the group's item dtype
3d89703 UPDATE a writer checks its tracks and groups, and infers dtypes at close, an array at a time
e5ac5a2 UPDATE write_sparse packs pieces into tasks of at least _PACK_ENTRIES and encodes each task's buffer in one kernel call
7352d07 UPDATE write encodes intervals on the pool
4f82dd4 UPDATE the writer checks values for infinity with a kernel
198147f UPDATE bigWig reads share one thread pool between reads
9eb0257 UPDATE a reader inflates group indexes, summaries and zoom levels with libdeflate
2436bb5 UPDATE sums(paired=True) sums its batches of tracks side by side on the reader's threads
485bfaa UPDATE figwig reads run sums of every group at once, cut batches over the whole read, and take threads by a read's bytes
dc99952 Merge pull request #43 from jmschrei/bam2fw_last_chromosome
bc9fab6 FIX bam2fw --barcode-column losing the last chromosome of a power-of-two count
76a35f7 Merge pull request #42 from jmschrei/fix-bam2fw-test-fast-exit
c285dce Merge pull request #41 from tuner/codex/regional-summaries
5e22877 UPDATE regional summary index: format specification, reorganized module, converter flags, new tests
3ad5db1 FIX bam2fw coverage test calls main with fast_exit=False
4942e6f Merge pull request #40 from jmschrei/hic_speed
b5990f3 Merge origin/main into hic_speed
bd04a4a Merge pull request #35 from casblaauw/bigwig_oob
9fff893 Merge origin/main into bigwig_oob
e298611 DOC oob= in the reading table, README, bundled skill and changelog; credit @casblaauw
6de78b1 Merge pull request #33 from casblaauw/is_figwig
bec0a20 Merge main into is_figwig
1122192 DOC credit @casblaauw for is_figwig and name it in the bundled skill
bbc9c22 Merge pull request #32 from jmschrei/hic_recipe
1fbe798 Document paired Python summary indexing benchmarks on Apple M5
4eb3ecd Remove the indexing plan and trim repeated documentation
e1d3bad Explain indexed summaries for interactive browser use
bea15d7 Clarify the 32 KiB summary index extension
9868eb0 Remove development-status labels from summary documentation
6d439ea Add experimental 32 KiB regional summary indexing
6697cb9 DOC the changelog, the benchmarks page's single-cell and Fiber-seq rows and the Fiber-seq times in the format page and the skill give this branch's Hi-C, single-cell Hi-C, cell and read write and read times
d02e0d4 UPDATE a read cuts a group of up to 8 chunks into batches in Python, where calling the kernel cost more than the loop, so that a read of a few chunks plans as fast as before the kernel
b9c60e8 UPDATE a writer without a pool encodes each batch of entries, and each flush of a track's blocks, as a task of its own, as before they were packed, since packing them without threads only copied their bytes
c8adbfa UPDATE write_sparse sorts keys in a few runs, and on one thread keys out of order only near each other, as reads given in position order are, by numpy's merge sort, which took them faster than the radix sort, and keeps the radix sort for keys out of order across a call, as a contact map's columns are
6226eb7 FIX read_entries reads chromosomes together only while the read's plan holds at most 2**20 (tile, group) pairs or one chromosome's, so that it plans in about the memory a read per chromosome took, where one read of many chromosomes over many groups planned them all at once
8c316f6 FIX read_entries reads its chromosomes in as few sparse reads as keep the read's sort key of window, track and offset within int64, where one read of every chromosome could pass it and return its entries out of order
26363fe FIX a reader whose metadata gives a track a group beyond int64 raises ValueError for the record, as it did before the records were checked by column, where it raised OverflowError
5b3efd2 FIX write_sparse takes values of any float dtype but float32 and float64, such as longdouble, as float64 before its radix sort's gather, as it did before the radix sort, where numba could not type them
3cd7532 FIX a sparse read's counting sort takes another thread's row of buckets only while the rows past the first take at most a byte per entry, and turns its counts into starts in place, so that a read of many windows and tracks with few entries sorts in the memory one row of buckets took
1c4a347 FIX entries a caller hands over already sorted, as bam2fw --barcode-column does, have their tracks' slots looked up in each encode task again, where the calling thread looked up every entry's before handing them over
4116ac3 UPDATE a task of packed track blocks joins their bytes and sets their records' offsets itself, so the writing thread writes the task in one call and adds one offset per track
3095905 UPDATE a task of packed group batches joins their chunks and builds their tile records itself, so the writing thread writes the task's bytes in one call and only appends each batch's records and folds its stats
477d06a UPDATE a file in track order hands the tracks' block flushes to the pool packed into a task per share of their entries, where each track's flush on each chromosome was a task of its own
e66e646 UPDATE a read cuts each group's chunks into batches in a kernel, where a Python loop stepped through every chunk
def5218 UPDATE read_entries reads every chromosome in one sparse read of a window per chromosome, so their tiles share the threads, where it read one chromosome at a time
13e403a UPDATE a reader checks its tracks' records a column at a time, with the name table doubling as the check for repeated names, and walks them one by one only to report the first at fault
078a61f UPDATE a sparse read sorts its entries by a counting sort straight from its tasks' parts, cut among the threads, where it joined the parts and sorted them on one thread
dbb7e87 UPDATE write_sparse takes each sorted entry's slot in its group from the sort key and hands slots to the encode tasks, so neither the gather nor the tasks look tracks up again
469aaa9 UPDATE write_sparse packs a call's small group batches into a task per share of its entries, and skips the tile cuts of a group that fits one batch
ecab925 UPDATE write_sparse finds the tracks with a declared integer dtype from an array kept on the writer, where it looked up every present track's record in each call
2a83588 UPDATE write_sparse sorts its entries by a radix sort of one packed key per entry, cut among the pool's threads, and gathers them in the same kernels; the numpy sort stays for keys of more than 63 bits
1052310 Merge pull request #39 from jmschrei/indexed_coverage
c4d43ad DOC the region reader of -c: when bam2bw and bam2fw take it, the coverage tables timed again, the changelog, the skill's -p and -c rows
31ea9ef UPDATE the indexed -c path counts mapped reads from the .bai's pseudo-bins alone before deciding, in a few ms, and parses the whole index only for files it takes
2c7ecc4 UPDATE bam2bw -c takes the indexed path from 128 MiB of BAM as well as 5,000,000 mapped reads, and checks the bytes first, so that a small file is not held up parsing its index
72c9d22 UPDATE bam2bw -c on the indexed path reads regions on all its cores, since its bigWig writer takes little of the time; bam2fw -c still leaves the figwig writer a quarter
1b2ecc3 DOC the comments where bam2bw and bam2fw hand -c to the indexed reader say it takes large files on 8 cores or more
47ab576 UPDATE the indexed -c path takes 8 cores or more: below that the writer has one core, and the whole-file reader, which reads on all of them and then writes on all of them, was faster
065b7da UPDATE the indexed -c path takes 2 cores or more, and bam2fw -c files of 128 MiB and 50,000 mapped reads each, the sizes from which it was faster than the whole-file reader on 25 BAMs
14d605e UPDATE the indexed -c path keeps 16 regions per reading thread in flight where regions hold only runs, so that a region of many reads late in the genome, or a write on the calling thread, does not leave the readers idle
46da25b FIX the indexed -c path reads a chromosome no file has reads on as one region, and gives an empty region's depth as no runs rather than a window of zeros
cf27451 UPDATE the indexed -c path finds a region's runs from its reads before any window, and fills a window only where the depth changes at least once per bam2fw's run spacing
4b50af4 UPDATE the indexed -c path takes only inputs large enough to gain: bam2bw -c from 5,000,000 mapped reads in the indexes, bam2fw -c when each file has 100 MiB; smaller ones go to the whole-file reader
1365e33 UPDATE the indexed -c path writes a strand's sparse regions in a row in one call, before its next window or chromosome
f183e76 UPDATE the indexed -c path tells sparse regions from dense ones by bam2fw's run spacing
6095866 UPDATE the indexed -c path hands the writer a sparse region's depth as runs rather than a dense window, and reads in regions of up to 32 Mb when it keeps runs, whose memory the cut by BAM bytes bounds
9ce5fb4 UPDATE the indexed -c path stops reading a region at the batch whose last whole record starts past it, rather than inflating the next batch to find out
4b80881 FIX the indexed -c path: bam2bw writes its bigWigs beside their names and moves them there once done, -v lines print after the read, a reference without reads counts as none mapped, an index pointing past the BAM or reading nothing falls back, dense depth scales only covered bases, and two cores read on a thread beside the writer
2104250 UPDATE bam2bw -c and bam2fw -c from an indexed BAM hold one region per reading thread, of at most 4 Mb, with 8 MB read buffers, which keeps their peak memory below the whole-file reader's
fb82a97 UPDATE bam2bw -c and bam2fw -c from an indexed BAM keep to -p: one core reads each region on the main thread between writes, and more cores are split between reading regions and the writer's threads
0543dce UPDATE _indexed's entry points carry the leading underscore of a private module's functions
f07097f UPDATE bam2bw -c and bam2fw -c read a coordinate-sorted, indexed BAM region by region on every core, from the index's offset for each region's first base to the first read of a 16 kb bin past its end, and join the regions' runs into the bytes the whole-file reader writes
7896c34 Merge pull request #38 from jmschrei/dense_sums_and_writes
3c1fb6e DOC bam2fw -c's sparse pieces as runs, and its benchmark rows and changelog numbers measured again
46cdfdb UPDATE bam2fw -c on threads writes each stretch of sparse pieces as runs in one call, where a call per piece cost sparse coverage the time it gained
74d90c5 FIX bam2fw -c on threads writes a piece of fewer runs than one per 1024 bases as its runs, which made sparse coverage such as RNA-seq's slower as dense windows than as intervals
2d68ac7 DOC benchmarks for coverage sums and threaded bam2fw -c, and changelog entries for sums over runs chunks, write_sparse's chunk kinds and bam2fw -c
2965138 UPDATE sums check whether any interval spans a whole tile before converting the chromosome names, so intervals shorter than a tile pay nothing for the tile-sum path
876e845 UPDATE bam2fw -c on one thread writes coverage as intervals, which costs less there than dense windows with no threads to encode them
7bf302c UPDATE sums read every chromosome's tile sums in one call, which inflates each group's summary once rather than once per chromosome
6a6d79c TEST the format's reference reader and Kaitai parser read dense and runs chunks written from entries
19808fd TEST bam2fw -c in pieces of a few tiles writes the file one piece per chromosome writes, and the pileup
17b2de5 UPDATE write_sparse stores a tile as a dense or runs chunk when either takes fewer bytes, so reorder and entries of dense tracks write the chunks write writes for the same rows
0fa4429 UPDATE bam2fw -c writes each 4 Mb of a chromosome's coverage as one dense window, whose whole tiles the writer encodes on its threads
13b6022 UPDATE sums of dense integer tracks: binned runs chunks skip the changes before an interval and add a run's value once per bin, and whole tiles come from the tile sums
90bbb54 ADD oob= on BigWigReader.read and read_bigwig, the value of bases past the end of a chromosome, as pybigtools' values(oob=) gives it; NaN by default as before
e6f0be1 ADD is_figwig(path), which tells a figwig file from a bigWig or another file by the magic number in its first four bytes
c0e5d65 Update README.md
7b04e44 DOC a Hi-C reference in the bundled skill: a track per bin in bin coordinates, a diagonal track for one-read coarse matrices, single-cell layouts
492caef Merge pull request #31 from jmschrei/zarrs_row
3123431 DOC a zarrs row in the count-track benchmark tables: zarr-python's arrays through the Rust codec pipeline, reads pinned to one chunk per call

Reading

Compatibility — ISA-L 2.32.1. The ISA-L kernel knows the inflate state of ISA-L 2.32.1, which python-isal 1.9 bundles, as it knew 2.31.1's (python-isal 1.8), so engine='auto' reads with ISA-L under either, where python-isal 1.9 read with zlib. The two versions' struct inflate_state is the same, field by field.

New — oob=. oob= on BigWigReader.read and read_bigwig sets the value of bases past the end of a chromosome, as in pybigtools' values(). The default stays NaN. Thanks @casblaauw!

Speed. A bigWig read with ISA-L gives its inflate kernel ISA-L's two functions in one tuple, where it gave them apart and numba typed each on every call under the GIL: a call of the kernel on an empty batch takes 2.9 µs, where it took 58.3 µs. On the benchmarks' server, 1,000 windows of 2,048 bp from the K562 ATAC-seq fold-change bigWig read in 9.4 ms on 8 threads, where they took 10.8 ms, and 20,000 windows in 103 ms on 32 threads, where they took 117 ms; 20,000 windows on 8 threads read in 106 ms, where they took 112, faster in only three of six pairs. The values are the same.

figwig files

New — summaries for every group that can hold them. A figwig file keeps the tile sums and zoom levels of each group whose sums take at most 256 MB, where it kept none for any group once the whole file's took more, about 350 genome-wide tracks on hg38 at the default tile, however its tracks were grouped. add_file, figwig merge and append=True keep the summary each group had, where they dropped every summary of a file that grew past that; figwig merge takes its zoom levels from the first file that keeps a summary, and FigWigReader.zooms from the first group that does. The writer holds the tile sums past 1 GB in a temporary file beside the output until it closes. The 384 genome-wide tracks of 64 MethBase2 samples in groups of two now keep every group's summary and six zoom levels, in a file 1.6% larger, written on 16 threads of the benchmarks' server in 396 s with 3.6 GB of memory at the peak, where the file without summaries took 374 s and 2.2 GB. Large groups of sparse tracks pay more, since a group's summary holds a sum per track for every tile that any of them touches: the K562 Hi-C map at 10 kb, one track per bin in groups of 1,024, now keeps its summaries in a file 6.8% larger, written in 2.52 s on one thread, where it took 0.97 s, and in 0.95 s on 8 threads, where it took 0.60 s, with 885 MB of memory at the peak on one thread, where it took 245 MB. Files that kept every summary are written to the same bytes where zlib compresses the summaries (see below). The format is unchanged: a file in which some groups keep a summary and others none is a version 1 file, which figwig 0.2.0 and the parser generated from the Kaitai Struct definition read, summaries included.

Change — summaries compressed by libdeflate. A figwig file's tile sums and zoom levels are compressed by libdeflate where it is installed and the writer's engine, as the chunks are, and by zlib otherwise, at level 6 either way. With libdeflate their bytes differ from those earlier versions wrote; every reader reads both, and the format is unchanged. Writing the 10,246 cells of the benchmarks page's 10x PBMC file in track order, one group each with 1.2 GB of tile sums before compression, takes 24.9 s on one thread of the benchmarks' server, where it took 58.5 s with zlib, and 8.8 s on 8 threads, where it took 12.2 s, in a file 0.4% smaller. The 90,543 Fiber-seq reads of the benchmarks page in track order, whose summaries hold a few tiles each, write in 2.4 s on one thread as with zlib, in a file 0.2% larger.

Speed — tile_sums keeps what it inflates. tile_sums keeps the tile sums it inflates, up to 1 GB, for later calls on the same reader, where it inflated a group's whole genome-wide summary on every call. On one thread of the benchmarks' server, a 10 Mb view of one sample's two count tracks in a figwig file of 64 MethBase2 samples takes 0.23 ms after the first, where it took 3.1 ms and read 1.0 MB of the file each time, and 0.33 ms when the 64 samples are one group, where it took 155 ms and read 49 MB. The standard benchmark's genome browser views drawn from tile sums take 0.65 to 0.84 times as long.

New — summary_index=True. FigWigWriter and write_figwig accept summary_index=True (default False), which writes an optional index that lets a reader decode the summary of one region without inflating a whole level, and FigWigReader uses it automatically for selected-chromosome tile_sums and exact integer sums, reading only the requested ranges, with the same values and the same fallback to raw values. The format stays version 1, and readers that do not know the index read the file as before. On one thread of the benchmarks' server, over 64 integer tracks in position order, tile_sums of a chromosome that is 20 Mb of 300 takes 19 ms, where it took 225 ms, and four integer sums of 200 kb 23 ms, where they took 81 ms; the file is 1.9% larger, and tile_sums of the whole genome takes 280 ms, where it took 262. Reading the sums no longer loops over the chromosomes in Python, so with or without the index tile_sums over a file of 100,000 contigs takes 21 ms, where it took 49. See regional summaries and the benchmarks page. Thanks @tuner!

Speed. sums over a track stored as runs, such as the coverage figwig bam2fw -c writes, only decodes the changes of value before each interval, adds a run of integers once rather than base by base, and takes the tiles an interval covers whole from the file's tile sums when the track holds integers whose sums those keep exactly. On one thread of the benchmarks' server, the coverage of ENCODE ATAC-seq BAM ENCFF877LRY sums over 78,733 gene bodies in 0.47 s, where it took 7.9 s, and over 100,548 peaks in 0.41 s, where it took 1.6 s; a fold-change track over the gene bodies in 1.55 s, where it took 6.8 s; 48 count tracks over the gene bodies in 6.8 s, where they took 14.4 s. The sums are the same to the bit.

Change — smaller reordered files. write_sparse, and so figwig reorder, stores a tile as a dense or runs chunk when either takes fewer bytes than a sparse one, as write does. Reordering the ATAC-seq coverage file gives the same 79 MB of runs chunks, where it gave 119 MB of sparse chunks. Scattered entries, such as cells' cut sites, are stored as before: the 10,246-cell PBMC file is the same to the byte.

New — is_figwig. is_figwig(path) tells whether a path is a figwig file from the magic number in its first four bytes, without opening it as a reader, so that code taking either a figwig file or bigWigs can choose its reader: it is False for a bigWig, a missing path or a directory. Thanks @casblaauw!

Speed. write_sparse sorts its entries by a radix sort of one packed key per entry on the writer's threads, where a numpy sort of the keys ran on the calling thread; keys in a few runs, and on one thread keys out of order only near each other, as reads given in position order are, still go to numpy's merge sort, which takes them faster. It hands the threads a call's small batches of groups packed into tasks that join their own chunks and tile records, so that the calling thread writes each task in one call. The 382 million pixels of the 4DN K562 Hi-C map (4DNFI18UHVRO) at 10 kb, as one track per bin in bin coordinates with both triangles, take the writer 12.1 s on 8 threads of the benchmarks' server, where they took 48.3 s, and 59 s on one thread, where they took 81 s; its 466 million pixels at 5 kb take 19.2 s on 8 threads, where they took 69.0 s; and the 3,882 cells of the Nagano 2017 single-cell Hi-C experiment at 10 kb, one track per cell, take 15.1 s, where they took 27.6 s. The files are the same to the byte, and the writer holds less memory beside its input: 1.14 GB on 8 threads at 10 kb, where it held 1.32 GB.

Speed. A file in track order hands its tracks' blocks to the threads packed into tasks of a share of the entries each, where each track's blocks on each chromosome were a task of their own: the Nagano cells at 1 Mb take 49 s on 8 threads in track order, where they took 70 s, and figwig reorder writes the ATAC-seq coverage file in track order in 27 s, where it took 45 s, and in position order in 26 s, where it took 44 s. The files are the same to the byte.

Speed. A sparse read sorts its entries by a counting sort straight from its tasks' parts, cut among its threads, where it joined the parts and sorted them on the calling thread, and read_entries reads its chromosomes together while their plan holds at most about a million (tile, group) pairs or the longest chromosome's, where it read one chromosome at a time. The 38.6 million entries of chromosome 1 in the 10 kb Hi-C file read in 0.13 s on 8 threads, where they took 0.52 s; one cell's contacts over the genome in the Nagano file read in 1.9 s at 10 kb, where they took 3.1 s, and in 0.064 s at 1 Mb, where they took 0.41 s. The entries are the same, in the same order.

Speed. A read cuts a group's chunks into batches in a compiled loop where the group has more than 8 chunks in the read, where a Python loop stepped through every chunk, and a file opens checking its tracks' records a column at a time, where it checked them one by one: the 617,666 tracks of the 5 kb Hi-C file open in 0.45 s, where they took 0.53 s.

Speed. sums over tracks of several groups runs every group's batches at once on the threads, where it summed one group at a time, so that a file of many small groups summed on one thread whatever n_jobs was. A read of many small groups takes batches of up to 128 KB per thread of the whole read's chunks, where it cut every group's chunks into two batches per thread: 1,024 tracks in 128 groups on 32 threads were 4,505 batches of a chunk or two, and are now 128. On a read of several groups each thread takes the next batch as it finishes one, and a read takes no more than one thread per 256 KB of the chunks it decodes and per 2 MB of a dense output, and at least 4, since reads of a few hundred KB of chunks took longer on 16 or 32 threads than on 2 to 8. Numba's list of a kernel's compiled signatures, which a read asked for twice and which hashes every signature's types in Python, is no longer asked for. On the benchmarks' server, 1,024 tracks in groups of 8 over 20 Mb of chr1 sum over 4,096 intervals in 1.6 s on 8 threads, where they took 3.1 s, and in 0.44 s on 32, where they took 3.1 s; 128 windows of them read in 0.046 s on 32 threads, where they took 0.48 s; and a genome browser's views of 256 of them, from a base per pixel out to the whole region and back, take 0.62 s in all on 8 threads, where they took 3.9 s. The values are the same.

Speed. sums(..., paired=True) in position order sums its batches of tracks side by side on the reader's threads, one each, where each batch was a read spread over every thread: the 90,543 Fiber-seq reads of the benchmarks page, each over its own span, sum in 1.6 s on 8 threads and 2.0 s on 32, where they took 2.9 s. Batches of float tracks are still read one after another on every thread, since their float sums depend on the threads. A file's group indexes, summaries and zoom levels inflate through libdeflate where it is installed.

Speed. write(..., ends=) encodes intervals on the writer's threads, where it encoded every tile on the calling thread, so that a write of read depth as runs took as long on 32 threads as on one. Each tile's sums are kept and folded into the statistics in the order the calling thread folded them, so the file is the same to the byte. The read depth of a full HCT116 ATAC-seq BAM, written as runs, takes 3.4 s on 8 threads of the benchmarks' server, where it took 12.4 s, and 2.1 s on 32, where it took 12.3 s.

Speed. write_sparse packs a call's pieces of entries, a group on a chromosome each, into tasks of at least 65,536 entries, each encoded in one kernel call per buffer, where each piece was a task with its own encoder call, and keeps no tile sums for a file too wide to keep summaries, where each piece zeroed a (tiles, tracks) array that was never read. A file of many tracks also opens and closes checking and inferring its tracks an array at a time, where it went track by track in Python. The 4DN K562 Hi-C map at 10 kb, a track per bin, writes in 9.1 s on 32 threads of the benchmarks' server, where it took 11.2 s, and the 3,882 Nagano cells at 10 kb in 8.9 s, where they took 10.3 s; the large-tier PBMC cells write in 0.52 s on 8 threads, where they took 0.65 s, holding 153 MB beside their input, where they held 447 MB. The files are the same to the byte.

Speed. sums(..., paired=True) over integer tracks and intervals of a few tiles each reads every interval at once, each with its own track's chunks or blocks alone, where it read batches of up to 16 tracks over all their intervals: the 90,543 Fiber-seq reads of the benchmarks page, each over its own span, sum in 0.69 s on one thread, 0.15 s on 8 and 0.13 s on 32 in position order, where they took 2.8, 1.5 and 1.6 s, and in 0.36 s on 8 threads in track order, where they took 9.9 s. The sums are the same; a batch's error for a sum outside an integer output and nan='count' keep the batches.

Speed. A read with labels over several groups of integer tracks runs in bands of windows side by side on the threads, where it ran a group at a time, and sums of integer tracks below 2**24 go straight into a float32 output, where they went through a float64 copy of it: the sum of every Fiber-seq read over 8,192 windows takes 0.056 s on 8 threads, where it took 0.12 s, and 0.061 s on 32, where it took 0.81 s; ten pseudobulks of the 10,246 PBMC cells over 8,192 windows take 0.11 s on 8 threads, where they took 0.19 s, and hold 0.68 GB, where they held 2.0 GB.

Speed. A few long windows summed per bin, as a genome browser's views are, are read as parts of whole bins on the threads, where each was a task per group: a session of views over four dense conservation tracks takes 0.20 s on 8 threads, where it took 0.53 s, with the same values. A part's float sums can differ from those of one thread, and of a whole window on several threads, in rounding, since where windows overlap one read bins a chunk that four or more of them share from a scratch tile. A group's index is checked on its live tiles alone and its runs of chunks laid out at once, and the indexes a read needs load on up to 8 threads once one of them spans 131,072 tiles or more, where they loaded on the threads the read took: a reader's first view of six genome-wide count tracks takes 13 ms on 8 threads, where it took 22 ms, and of 256 tracks in 32 groups 7.7 ms, where it took 15 ms; a process running the count tracks' session peaks at 169 MB, where it peaked at 146 MB.

Speed. A track-order read plans every track's windows at once, and joins the tracks' block indexes as bytes, where it planned a track at a time: 16 PBMC cells over every chromosome read in 9.1 ms, where they took 15 ms, every Fiber-seq read's sparse values over 1,024 windows in 0.047 s on 8 threads, where they took 6.8 s, and the sum of every read over 8,192 windows in 0.089 s on one thread, where it took 15 s. A track-order writer encodes each task of tracks' blocks in one compiled call, of at most 2**18 entries, adds a call's pieces in one loop, and joins its block records as bytes at close: the Fiber-seq reads write in track order in 1.5 s on 1, 8 or 32 threads, where they took 3.9, 5.6 and 5.7 s, and the 10,246 PBMC cells in 5.9 s on 8 threads, where they took 18.1 s.

Speed. write_sparse checks the ranges of declared integer dtypes in one compiled pass, where it took each track's minimum and maximum with numpy, and checksums use libdeflate's crc32, 19 to 46 GB/s, where a table kernel ran at 2.4 to 2.8 GB/s: the full PBMC cells write in 7.7 s on 8 threads, where they took 12.2 s, and 4.5 s on 32, where they took 9.7 s. The files are the same to the byte.

Speed. write_sparse sorts its entries in fewer radix passes, starting above a tile's offset bits where a call's positions ascend, and on one thread sends keys out of order only near each other to numpy's merge sort only while at most one in 64 descends or the call has fewer than 16,384 entries, so that a single-cell contact map's calls, whose cells interleave within each tile, take the radix sort, at 9.1 ns an entry where the merge sort took 22. Its passes over a call's entries, the checks of positions, track ids, int64 ranges, infinity and declared dtypes, run in parts on the writer's threads, its int64 inputs are no longer copied, and a packed task's statistics fold in one call. The 4DN K562 Hi-C map at 10 kb, a track per bin, writes in 51 s on one thread of the benchmarks' server, where it took 58 s, and in 8.9 s on 8 threads, where it took 11.9 s; the 3,882 Nagano cells at 10 kb in 80 s on one thread, where they took 90 s, and in 13.2 s on 8, where they took 14.4 s, the fastest of 4 to 14 paired runs. On 32 threads, in the same pairs on a busier server than the timings above, the map writes in 6.8 s, where it took 13.1 s, and the cells in 6.6 s, where they took 9.9 s. The files are the same to the byte, and the writer holds less beside its input: 711 MB on one thread for the map, where it held 968 MB, and 625 MB for the cells, where it held 789 MB.

Speed. A read of fewer than one in 16 of a group's tracks, as one cell's of a single-cell file is, checks only those tracks' offsets in a chunk and passes over the chunks in which none of them has a base, where it checked every track's and decoded every chunk; reads of more tracks run as before. Where libc's pread and libdeflate both load, as everywhere but Windows, a batch is read, checked and inflated in one compiled call that holds no GIL, where each thread took the GIL between numpy calls and compiled ones; a read's planner finds every batch's arrays once per group, where it made some twenty numpy calls per batch on the calling thread; and an array of track indices is checked at once, where its items were checked one by one. On the benchmarks' server, one cell's contacts over the genome in the Nagano single-cell Hi-C file at 10 kb read in 8.5 s on one thread, where they took 12.8 s, in 1.23 s on 8 threads, where they took 1.97 s, and in 0.75 s on 32, where they took 1.45 s; 16 cells in 1.54 s on 8 threads, where they took 2.22 s; and 50 rows of the 10 kb Hi-C map, each a bin's contacts over the genome, in 0.15 s on 32 threads, where they took 0.24 s. The entries and values of these reads are the same.

Speed. A sparse read lists its entries' rows, channels and offsets as int32 while a window is narrower than 2**31 bases, and widens them to int64 as it gathers them, where it listed them as int64: chromosome 1's cis matrix of the 10 kb Hi-C map, read with sparse=True on one thread, adds 1.92 GB to its process at its peak, where it added 2.41 GB, and takes 0.52 s, where it took 0.58 s. The entries are the same, in the same order.

Speed. A write of windows on threads checks its values for infinity and against declared integer types in parts on the writer's threads, the latter in one compiled pass where it went through numpy a track at a time, and copies each group's block once, every group's of a call in one pass on the pool, where it copied a group's channels twice on the calling thread. The copies go in waves of at most 4 MB per thread, the next copied once the tiles of all but the last are encoded, so that a call holds two waves of copies at most, and the calling thread takes a part itself while the writer's threads are busy, they taking the next as they come free. On the benchmarks' server, 256 tracks over 20 Mb of chr1 write in 1.98 s on 32 threads, where they took 6.6 s, in 3.2 s on 8, where they took 6.3 s, and in 16.9 s on one, where they took 18.9 s; 1,024 tracks write in 10.4 s on 32 threads, where they took 21.1 s. The files are the same to the byte. On 32 threads the waves hold up to about 150 MB more than the calling thread's copies did: 2.01 GB at the 256 tracks' peak, where it was 1.86 GB.

Speed. A write's chunks are compressed by libdeflate and checksummed in one compiled call that holds no GIL, where the calling thread's numpy calls let it go and took it back three times a batch, and write_sparse plans and encodes each task of entries in two such calls, where it made some fifty numpy calls a task. write_sparse takes entries already in the writer's order without sorting them, and tests each piece of a call against the order rules before calling the check that raises. A writer with threads writes its chunks and blocks to the file in order on a thread of its own, with at most 64 MB queued for it, where the calling thread wrote them between batches. On 32 threads of the benchmarks' server, the 4DN K562 Hi-C map at 10 kb writes in 5.2 s, where it took 5.9 s, the Nagano cells at 10 kb in 5.8 s, where they took 6.6 s, and the 10,246 PBMC cells in 3.05 s, where they took 3.41 s, and in track order in 4.7 s, where they took 5.6 s, in 3.2 s on 8 threads, where they took 5.4 s, and in 7.7 s on one, where they took 9.3 s. The files are the same to the byte.

Speed — first calls. A first call compiles fewer copies of the kernels. A variable that starts at a constant and is passed from one kernel to another had the kernel it goes to compiled twice, for the constant and for its type; the two passes of a sparse read passed int64 and then int32 arrays; and a plain read compiled its decoders for a float64 scratch tile as well as for its own output type. Alone in a new process with an empty numba cache, on the benchmarks' server, read_entries compiles for 20 s, where it took 69 s, a dense read for 21 s, where it took 38 s, a sparse read for 22 s, where it took 40 s, and a track-order read for 3.6 s, where it took 7.9 s; a process that reads and writes every way compiles for 87 s, where it took 132 s. The values read and the bytes written are the same.

Speed. A writer of many tracks looks each distinct dtype up once and sets its groups up in arrays, where it went through them in Python: a writer of the 90,543 Fiber-seq reads opens in 0.060 s, where it took 0.151 s, and in track order in 0.139 s, where it took 0.362 s. With the changes above, the reads write in track order in 1.21 s on 8 threads, where they took 1.61 s, and in 1.45 s on one, where they took 1.74 s.

Fix. FigWigWriter refuses a group of 229 values per chunk, tracks times tile, such as 8,192 tracks at a tile of 65,536 or 32,768 at the default tile, and takes one track fewer, where it took it and wrote a file no reader could open once the group's tracks were dense with float64 values: such a chunk takes 232 bytes, one more than its u4 size holds. figwig bam2fw --barcode-column puts at most 32,767 cells in a group at the default tile, where it put 32,768.

Fix. write_sparse refuses entries in a tile that a group from append=True or add_file already holds, and in track order entries at or before a track's last base, where it took them and wrote a file whose index could not be read.

Fix. In track order, write and write_sparse refuse a call on a chromosome that comes before one an earlier call wrote, in chrom_sizes order, where they took it and wrote a file whose index could not be read; figwig 0.2.0 did the same. Position order still takes chromosomes in any order across calls.

Fix. sums on a reader with verify=True reads and checks every chunk under its intervals, where for integer tracks it took the tiles an interval covers whole from the tile sums and so returned a sum over a corrupt chunk without checking it; without verify it still takes them from the tile sums.

Speed. write copies each call's values for the writer's threads in a vectorized loop, where the loop copied one value at a time, and cuts a call of integer values of one or two bytes into tasks by their own bytes, where it counted four bytes a value and cut a call of count tracks into more tasks than encoding them repays. On 8 and 32 threads of the benchmarks' server, 6 uint16 count tracks over chr1, written in calls of 1 Mb, take 0.49 s, where they took 0.81 and 0.89 s, and 48 such tracks 3.6 and 4.0 s, where they took 4.1 and 4.3 s. The files are the same to the byte.

Converters and commands

Fix. figwig install-skill over an installed copy without --force prints the message saying to use it and exits with status 1, where it printed a traceback.

Fix — results differ. figwig bam2bw -c -r and figwig bam2fw -c -r divide by the reads, one for each read whose first aligned block starts inside its chromosome, as -r without -c and deepTools' CPM count them, where they divided by the aligned blocks, so that a read cut by a deletion or a skipped intron counted once per block: a K562 polyA RNA-seq BAM of 31.5 million mapped reads was divided by 42.6 million, and the 2.4 GB ATAC-seq BAM by 45.9 million for its 44.8 million. Outputs without -r are the same to the byte, and so is -c -r where every read is one block: no deletion or skipped intron, nor, in a SAM file or another file read by pysam's loop, an insertion.

New — --summary-index. --summary-index on figwig bw2fw and bam2fw writes the regional summary index, and figwig reorder and merge write one when the header of their input (of the first file, for merge) announces one, unless --no-summary-index, or when --summary-index is given.

New. figwig info -b counts a file's regional summary index apart from its unused bytes.

Speed. figwig bam2fw -c on threads writes each 4 Mb of a chromosome's coverage as one dense window, whose tiles the writer encodes on its threads, where it encoded every run on the calling thread, unless the 4 Mb hold fewer runs than one per 1,024 bases, as sparse coverage such as RNA-seq's does, which goes in as runs as before: the 2.4 GB ATAC-seq BAM takes 7.2 s on 8 threads, where it took 14.2 s. The file is the same to the byte, and one thread writes it as before.

Speed. figwig bam2bw -c and figwig bam2fw -c read a coordinate-sorted BAM file with a .bai no older than it a region at a time on every core, from the index's offset for each region's first base, and write each chromosome while the next is read, rather than keeping two events per aligned block of every read until the file ends. They do so at -p 8 or more, for bam2bw from 128 MiB and 5 million mapped reads in its files, for bam2fw from 128 MiB and 50,000 mapped reads in each file; below these, set from timings on 25 BAMs of the test-data collection with a margin on the whole-file reader's side, and for a file the region reader cannot read exactly as the whole-file reader would, the whole-file reader runs as before. The 2.4 GB ATAC-seq BAM takes 1.6 s with bam2bw -c -u -p 8, where it took 2.5 s, and 1.05 s at -p 32, where it took 2.5 s; 4.9 s with bam2fw -c -u -p 8, where it took 7.2 s, and 2.3 s at -p 32, where it took 7.1 s. At -p 8 the peak memory falls from 2.0 to 0.54 GB (bam2bw) and from 1.9 to 0.63 GB (bam2fw); at -p 32 it is the regions in flight, 2.4 GB against 2.0 GB for bam2bw. The bigWigs and unstranded figwig files are the same to the byte; a stranded figwig file holds the same values and the same bytes of chunks, the two strands' interleaved, so its index holds more runs: 0.14% more bytes in all on 8 million ChIP-seq reads.

Fix. figwig bam2fw --barcode-column no longer loses the counts of the last chromosome when the sizes file has 2, 4, 8, 16, ... chromosomes. The last cut between chromosomes, n << shift, was 2**63 and wrapped to a negative number in int64, so that chromosome's keys were never decoded, with no error or warning: 32 chromosomes with one fragment on each wrote a file whose chr32 read as all zero. Files written before with such a count are missing that chromosome and have to be written again; for other counts the cuts are the same as before, and the files of 3, 5, 7 and 24 chromosomes are the same to the byte. Thanks @martinappberg!

Speed. figwig bam2fw -c of a BAM the region reader does not take, as one below 128 MiB is, writes its runs on the writer's threads (above): the collection's 50 MB K562 ATAC-seq BAM takes 1.2 s at -p 8, where it took 3.5 s.

Speed. figwig fw2bw reads the next piece of a group while the bigWig writers take the last, where reading and writing alternated, and the bigWig writer lays out a window's items without a branch per base: twelve bigWigs (ATAC-seq and DNase-seq cut counts, and PRO-cap, PRO-seq, csRNA-seq, TT-seq and NET-seq per strand) convert from their figwig file in 44 s on one thread, where they took 67 s, and in 10.3 s at -p 8, where they took 17.8 s. figwig bw2fw checks the values for infinity in one compiled pass, and bigWig reads keep one thread pool between reads, where each read started its own: the same twelve bigWigs convert in 52 s on one thread, where they took 59 s, and in 10.5 s at -p 16, where they took 13.4 s. The files are the same to the byte.

Speed. figwig fw2bw writes the bigWigs of a group of at most -p tracks each on a thread of its own, with at most two pieces of the group in flight, where every piece waited for each of its tracks' bigWigs before the next was written; a larger group's bigWigs are written -p at a time, each piece whole before the next is read, as before: the twelve bigWigs above convert in 8.95 s at -p 8, where they took 9.99 s, and in 9.1 s at -p 32, where they took 10.1 s. With the writer's changes above, figwig bw2fw converts them in 9.3 s at -p 8, where it took 10.4 s, and in 10.7 s at -p 32, where it took 14.0 s. The files are the same to the byte.

Speed. figwig reorder into track order hands the writer each chromosome's entries as already sorted, and the track-order writer sums its blocks' tiles in one pass: the read depth of the HCT116 ATAC-seq BAM reorders in 39 s on one thread, where it took 57 s, at a peak of 7.9 GB, where it took 14.0 GB. The file is the same to the byte.

Fix. figwig bam2bw -c and figwig bam2fw -c take a SAM file's reads, and those of a BAM file the record kernel hands to pysam, in the kernel's blocks: a run of M, = and X operations split only at D and N, where pysam's get_blocks() began a block at every operation. An insertion, or a change between = and X, no longer splits a bigWig interval there, so a SAM gives the bigWigs of the same reads as a BAM to the byte; and -ps/-ns empty a read they reach across whole, where they emptied only its first or last operation and left the others covered, in bigWigs and figwig files alike. Without shifts, bam2fw -c already wrote the same figwig files, which keep values rather than intervals.

Packaging

Packaging. The bam2bw and dev extras require python-isal 1.8, as the isal extra does, where they took 1.0 or later, with which ISA-L reads fall back to zlib.

Documentation

Documentation. The bundled Claude Code skill has a reference for Hi-C contact matrices, references/hi-c.md: one track per bin with chromosomes measured in bins, since in bp a tile stops at 65,536 bases and a bp file took twice the bytes; both triangles and a diagonal track, so that labels= over rows and bins= over columns read a coarser resolution in one call with the diagonal counted once, as cooler counts it, where summing both triangles counts it twice; writing cooler's row-major pixels by the column's chromosome, after summing the pixels published .mcool files repeat, which write_sparse refuses; single-cell layouts; and the limits Hi-C sizes reach (tile_sums, track counts, a pseudo-chromosome's length). The reading reference also says that bins= sums count bases past a chromosome's end as 0, where the general rule read as NaN.

Documentation. The benchmarks page's count-track tables have a row for zarrs 0.2.3, the Rust codec pipeline for zarr-python, on the same arrays as the zarr row: it writes them 1.1 to 3.5 times faster than zarr-python and reads one window per call at zarr-python's one-thread speed, without gaining from the Python threads, with its chunk concurrency pinned to one chunk (its default read twice as slowly on these calls).

What is now possible

A figwig file of hundreds or thousands of tracks keeps the coarse profile a genome browser draws from: the 384 genome-wide tracks of 64 methylation samples in groups of two, a Hi-C map with a track per bin in groups of 1,024, or a track-order file with a group per cell all keep their tile sums and zoom levels, so tile_sums and chromosome-wide views read them without decoding a chunk, where 0.2.0 kept none. With summary_index=True (or --summary-index on bw2fw and bam2fw) a view of one region reads only that region's summary. Code that takes either a figwig file or bigWigs can choose its reader with is_figwig(path), and oob= sets what bases past a chromosome's end read as.

figwig 0.3.0 figwig 0.3.0 on Python PyPI