piaso-tools 1.2.7

pip install -U piaso-tools

piaso.tl.umap is PIASO's own: parallel, and the same picture on any number of threads

piaso.tl.umap now lays out the graph with PIASO's extension by default (backend='auto'), whenever a kNN is stored (after piaso.tl.neighbors) or passed. It computes umap-learn's algorithm: the same fuzzy graph, the same curve, the same spectral start and the same schedule of attractions and repulsions. What differs is the order of the work. umap-learn makes a seeded run single-threaded, because its updates share one stream of random numbers. PIASO's layout cuts the nodes into 128 blocks by a hash of their names, and an epoch visits every pair of blocks once; the pairs that run at the same time share no node, so the threads together compute one sequential pass in a fixed order. One graph and one random_state give the same coordinates on 1 thread or 20.

• Quality. Its pictures keep each cell's graph neighbors as umap-learn's do, within the variation between umap-learn's own seeds.
• Speed. The layout uses every core the process may use, so a large dataset takes a fraction of umap-learn's single-threaded time.
• The start does not depend on the seed, as umap-learn's does not, and each axis's sign is fixed, so two seeds no longer give mirrored pictures.
• Memory. The layout keeps 12 bytes per edge of the graph and about 40 per cell, the same on any number of threads.

What to know when upgrading:

• The coordinates change. They differ from umap-learn's as two umap-learn seeds differ. To reproduce an embedding made with 1.2.6:

  piaso.tl.umap(data, ..., backend="umap-learn")

• Threads. n_threads=0 (the default) uses the cores the process may use; the picture does not depend on it.

• The whole stored kNN is used. umap-learn takes a stored kNN at its own n_neighbors (15): it cuts a larger one to its first 15 columns and sets a smaller one aside for neighbors it computes itself. PIASO's layout uses every column. With neighbors' default of 15 the two agree.

• umap-learn stays a dependency; without a stored kNN, tl.umap still calls umap-learn, which computes its own neighbors, with a warning.

score() leaves a gene set's own genes out of its controls

Every path now draws control genes the same way: in numpy, a fresh random state per gene set seeded by random_seed, gene by gene, with the genes' means and variances summed in float64. The same seed gives the same control sets and the same scores on an AnnData with or without the compiled extension, in the single-set mode and on a cytome.

The new default, exclude_set_genes=True (in score, calculateScoreParallel and predictCellTypeByMarker), draws each gene's controls from its nearest genes outside its own set. A member among a gene's nearest genes gives its place to the next nearest non-member, so only the draws that would have been a member change.

What to know when upgrading:

• Scores change where a set's genes are among each other's nearest genes, which is common for co-expressed marker sets. exclude_set_genes=False keeps the earlier draws of the numpy path; the compiled path, which drew its controls with another generator, now draws as numpy does, so its scores change once either way.
• A panel with fewer features than the neighbors asked for (twice n_nearest_neighbors with the exclusion) still scores, and one warning per call names the genes left with fewer candidate controls.
• The nearest-gene table is int32, so the wider table the exclusion needs takes the memory the int64 table took before.

predictCellTypeByMarker(smooth_by='scores')

smooth_by='scores' averages each cell's scores (or -log10 p-values) over its smoothing neighbors before taking the best cell type, instead of voting the neighbors' own best types (smooth_by='labels', the default, unchanged). The neighbors are the vote's, the averaged matrix is kept in obsm['{key_added}_score_smoothed'] (or '_nlog10pvals_smoothed'), and the confidence is the share of the neighbors whose own best type is the one chosen. On an AnnData and on a cytome.

runSVD stops when its singular vectors have converged, and ATAC embeddings come at unit length

Two defaults of piaso.tl.runSVD change, and with them the embedding a fixed random_state gives:

• The stopping rule: stop_on='residuals', the new default, stops when every component's relative residual is below tol, so the singular vectors themselves have converged. 1.2.6 stopped when the leading singular values settled, which can happen while the vectors still depend on the random start. A run may take a few more passes.
• Unit length for ATAC: with unit_length unset, the TF-IDF of ATAC peaks or tiles is embedded at unit length, so neighbors, leiden and umap work with the angle between cells rather than a length that follows sequencing depth. Each cell's length is kept (adata.obs['X_svd_length']; in a cytome, the cells table's <embedding>_length column). RNA embeddings are not scaled.

What to know when upgrading:

• Embeddings change. To reproduce an embedding made with 1.2.6, pass both:

  piaso.tl.runSVD(data, ..., stop_on="singular_values", unit_length=False)

• So does what embeds through runSVD. leiden_local passes both to each group's SVD (svd_stop_on, svd_unit_length), and runGDR's own SVD stops on the residuals.

• An AnnData whose matrix already holds TF-IDF values carries nothing that says so: pass modality='ATAC' or unit_length=True.

runSVD can leave out the first component, and runs the same engine on an AnnData as on a cytome

• drop_first_component=True computes n_components + 1 components and keeps components 2 to n_components + 1, so the embedding still has n_components columns, and S and Vt hold the same components. For non-negative, uncentered inputs such as the TF-IDF of peak counts, the first singular vector is the axis of the mean profile and follows sequencing depth more than cell type. The choice and the left-out singular value are recorded with the embedding. It is not meant for RNA inputs.
• An AnnData gets the solver and the Rust pass engine a cytome gets, with the engine reading the matrix in memory: method, tol and n_iter are honored, every allowed core is used, and an AnnData and the same matrix in a cytome give the same embedding, S and Vt to the bit. auto_tfidf=True, cell_mask, scale_data, return_svd, tfidf_params and infog_params now work on an AnnData (the in-memory path ignored the last two).
• What to know: an AnnData's embedding changes once, since it came from scikit-learn's randomized SVD (seven fixed rounds on one core). auto_tfidf=True on an AnnData whose values are not counts raises.

Fixed: runSVD(ds, modality='ATAC') reads the peak counts

On a cytome, runSVD with no measurement read an INFOG layer whatever the modality, so the plain call on a store holding ATAC peak or tile counts stopped with KeyError: no ATAC_infog matrix. Left unnamed, the layer now follows the modality: the counts for 'ATAC' and 'tiles', which the default TF-IDF transforms, and INFOG for RNA. The call gives what measurement='counts' gave, value for value.

Fixed: after runGDR on labels with "/", the AnnData can be written

anndata 0.13 refuses a "/" in a key of an h5ad file by default (anndata.settings.disallow_forward_slash_in_h5ad), and runGDR kept its marker sets in adata.uns['gdr_reference'] as a table whose columns are the cell types, so on labels such as "L2/3 IT" adata.write_h5ad failed with "Forward slashes are not allowed in keys". Such names are now stored with "/" written as "%2F", and the exact names beside them (marker_gene_names); projectGDR reads them back unchanged. Labels without "/" are stored as before, and a reference written by 1.2.6 still reads.

Also in this release

• Small inputs. neighbors caps the number of neighbors at the number of cells, and replaces a graph in which the approximate search left a slot unfilled by the exact graph (a 10-cell batch failed with "negative axis 1 index: -1"). runGDR(batch_key=...) gives a batch of fewer than 16 cells no markers of its own, with a warning, instead of failing, and still embeds its cells.
• leiden_local(tmp_dir=) places the groups' temporary stores, and when the disk is short writes them in batches that fit, one scan of the matrix each; the labels are those of a single split.
• filter_cells(check_integrity=True) on a cytome asks for SQLite's integrity check of the filtered store; cytome 0.3.7 otherwise runs only its own consistency checks.
• makeGenomeFiles reads the GTF with cytome.read_gtf: one gene per gene_id, so genes that share a name are no longer merged into one span (in Cell Ranger ARC's Mmul_10 reference, the copies of TRNAN-GUU spanned 118.6 Mb of chr1).
• specificity_matrix(cosg_remove_lowly_expressed=False) builds runSCALAR's input: without COSG's detection filter the matrix has no negative entries.
• A renamed PIASOmarkerDB study's former name still works in analyzeGenes and analyzeMarkers: studies= and exclude_studies= are resolved through the server's map of former names (studyAliases()), in any case, with one warning per renamed name. A server without the map leaves the names as given.
• Requires cytome 0.3.7 (for cytome.read_gtf); pip install -U piaso-tools installs it.