piaso-tools 1.2.6

piaso-tools 1.2.6

on Python PyPI

pip install -U piaso-tools

piaso.tl.leiden is PIASO's own: parallel, and the same clusters on any number of threads

Leiden now runs in PIASO's Rust extension by default. It is the same algorithm (Traag, Waltman and van Eck 2019), with the same objective and the same randomised refinement as igraph's, and it reaches the same modularity. What differs is how it is run:

Measured on a 10-core workstation (20 threads); memory is what the call adds to the process.

One graph and one random_state give one partition. The number of threads, the cores of the machine and the layout of the graph in memory do not enter the result: the labels are identical at 1, 2, 4, 10 and 20 threads. Parallel clustering usually gives that up, because threads see each other's moves in an order that changes from run to run. Here a round of moves reads the state the round began with, every random number is a hash of the seed and the cell, and edge weights are summed as integers, so nothing depends on which thread did what first.

piaso.tl.leiden(adata, resolution=1.0)                  # all the cores this process may use
piaso.tl.leiden(adata, resolution=1.0, n_threads=4)     # the same labels
piaso.tl.leiden(adata, resolution=1.0, backend="igraph")  # the partitions of 1.2.5

What to know when upgrading:

• Clusters change. The partitions differ from those of 1.2.5 by about as much as another seed would change them (ARI 0.82 to 0.88 between the two on the datasets above, where two seeds of igraph agree at 0.80 to 0.89). Pass backend="igraph" to reproduce an earlier result.
• So does everything that clusters. leiden_local and runGDR call piaso.tl.leiden, so their results follow. leiden_local takes backend= and n_threads= and passes them on.
• The graph must be symmetric. A directed graph is refused, with the pair of nodes whose two directions differ. igraph's path took the upper triangle and clustered it without a word.
• n_iterations stays at 10. Two runs give 96 to 99 % of the partition of ten in under half the time, which is worth knowing at tens of millions of cells; ten make two seeds agree a little more.
• Ctrl-C stops a run. A graph of fewer than 32,768 nodes is clustered on one thread, which is faster there.

The Leiden at scale tutorial runs it on 200,000 cells and shows the clusters beside igraph's.

Reading a store you cannot write

The plotting functions open a .cytome read-only, so plotting from a downloaded or shared store leaves the file exactly as it was. Statistics that are normally cached in the store (cell depth, INFOG and TF-IDF parameters) are computed for the call instead when it cannot be written. Functions that compute and then write (leiden, leiden_local, neighbors, umap, runSVD, runGDR, infog, run_TFIDF) check first, and a read-only store fails in a second with a message saying why, not at the end of the run.

The motif scanner: twice as fast, and safe after fork

piaso.pp.scan_motifs(backend="rust") scores a window by walking it and the motif's columns together, with no index arithmetic in the inner loop: 300 motifs against 100,000 sequences of 300 bp take 5.7 s instead of 10.4 s, and peak memory is 1.03 GB instead of 1.26 GB. The scores are identical to the bit.

It also hung for ever in a process forked after it had run, which is Python's multiprocessing on Linux. It now runs on a thread pool of its own, releases Python's lock while it works (other Python threads keep running) and stops on Ctrl-C. Every parallel function of the extension now builds its pool the same way, and n_threads=0 means the same in each: RAYON_NUM_THREADS if set, else the cores the process may use.

runSVD on a cytome no longer opens the file a second time

The SVD engine used to open the store with its own copy of SQLite while Python's connection held it. The two could not see each other's locks, and the engine could reset the shared index of the write-ahead log that Python had mapped; the next large write then ended the process with a bus error (SIGBUS). The engine now computes on the compressed chunks the open dataset hands it (ds.chunk_source), so no second SQLite touches the file. The result is identical to the bit, and cache_chunks=True now works with the engine, keeping the compressed chunks within half the memory available.

Also in this release

• leiden_local on a cytome makes every group's store in one scan of the matrix (ds.split), instead of one subset per group. The labels are unchanged.
• piaso.data.chrom_sizes(genome) gives the chromosome sizes of hg38, mm10, Mmul_10, mCalJac1, hg19 and mm39 by name, or of any chrom.sizes or .fai file, reduced to the primary chromosomes.
• piaso.data.fetch_dataset checks a download before it takes its final name, and never replaces a file that is already there: a downloaded .cytome is a working file, and checking it on every call re-downloaded it over the results written into it.
• color= given a NumPy array of gene names draws one panel per name. It was read as one value per cell.
• Fixed: plot_embeddings_split without groups= could give categories each other's colours. Each panel's palette was built in alphabetical order but applied in the column's category order, so wherever the two differed (a categorical order, or one stored with ds.set_categories) the colours were swapped, with no warning. Every panel now colours each category as plotEmbedding does. Figures made with 1.2.5 or earlier without groups= are worth a second look.

Requires cytome>=0.3.6.