cellucid 0.9.6 on Python PyPI

Version 0.9.6 is a patch release on 0.9.5. Files prepared, sessions saved and
dataset catalogs written by 0.9.5 are read as before. The viewer protocol is
schema 2: use the Python and R packages of this release with the web app of
this release. The main changes are:

Highlights

Subsampled reading. Open a seeded share of a dataset's cells; only the
parts of the file that hold them are read, and every part of Cellucid
(view, filters, analyses, genes, exports, notebooks) works on those cells.
The same cells in the web app, Python and R, in a link and in a session.
Files Cellucid writes store their cells in priority order, so a share
of the cells is the first rows of every array.
Datasets too large for the device open fitted, and large ones open
progressively, growing in place; a larger or smaller share is applied in
place, reading only the cells added.
One notebook widget for Python and R (show_anndata(), cellucid.Widget,
R cellucid_view() in Jupyter): no port, no proxy.
Round trip in one call: viewer.pull(adata) / cellucid_pull() write
every highlight, label, score and gene set back onto the object;
push() / cellucid_push() send yours in.
Up to 100 million cells in 3D on a 16 GB laptop, with levels of detail
that are uniform samples of the cells.

Web app

Subsampled reading

Cells to open (Session › Open data): one slider (or a typed share) with
the exact count it stands for ("228,245 of 2,282,447 cells"). Below 100 %
the method, its seed and what the method needs appear, with Apply.
Datasets opened next in this tab take the same share; a method a dataset
cannot use (a stratifying field or embedding it lacks) opens the same share
by the file's default method, and says so.
Four methods, one specification. uniform (every cell equally
likely), blocks (runs of neighbouring cells aligned to how the file
stores them, so parts holding only left-out cells are never read),
stratified (proportional or the same count per category of a categorical
or text field, optional minimum per category) and spatial (balanced over
a grid on an embedding, keeping sparse regions). Every method is seeded;
with uniform and blocks a smaller share is inside a larger one. The
specification is published, with a conformance fixture shared by the web
app, Python and R.
The file's default. A file Cellucid wrote opens a share as uniform
under the file's own seed (the first rows of every array); any other file
as blocks, whose block size is set by the file's layout alone (about four
chunks of its dominant per-cell array, or 64 KiB of it when it is read byte
by byte), so the same file and share always open the same cells. The panel
says what uniform would cost on such a file ("fetches 45 % of the matrix
for these 10 % of the cells") and how far a blocks share's categories are
from every cell's.
At least two cells. One cell has no coordinate variation to draw (the
rule a one-cell file is refused by): the control does not offer a share of
one cell, a link asking for cells=1 is refused with that reason, and
fitting to the device never goes below two.
In a link. ?data=…&cells=10%&sample=blocks&seed=0 (also by,
alloc, min, embedding, grid, block) opens exactly that share,
overriding fitting to the device; cells=100% opens every cell. Parsed
strictly: a malformed value is named and the dataset opens as if none were
given. A share chosen in the panel is written to the address as its exact
count, method and seed (never one fitted to the device, never the cells),
Copy link copies one, and the panel shows the cells' digest so two
people can confirm they see the same cells.
Sessions keep their cells. A session saved on a share records the cells
(compactly), how they were chosen and their digest; loading it opens
exactly those cells, reading the dataset again first when needed. Per-cell
session data names cells by original position, so it applies whatever
order a file stores its cells in.
Counts say "k of n". The Dataset box ("228K of 2.3M cells"; Details:
the method and seed), the filter line (Showing 120 of 240 of 1,200
cells), categorical legends, the category builder, the screen-reader
status line and the dataset picker all say how many of every cell are open.
Neighbour graphs on a share keep the edges between open cells; the
Dataset box says how much of the open cells' graph that is, and Data
mapping offers every graph on the open cells.
A figure of a share says so: SVG and PNG provenance name the open
cells, how they were chosen and their digest.

Too large for the device, progressive opening, Apply in place

Fitted to the device. When every cell would not fit this device's
memory (77 bytes a cell, as measured with levels of detail on, against half
the device's memory; or, for a matrix stored cell by cell, the index that
makes its genes readable), a share sized to fit opens by the file's default
method, with a notice and Open all cells. About 104 million cells fit a
16 GB machine.
Progressive opening. When reading the cells asked for is measured to
take more than about a second (the link's speed and latency, the stored
bytes of the embedding's parts the first view needs and their decoding, and
how long this device took to show cells before), and a first view would
arrive at least twice as soon, a first view of fewer cells shows at once and
grows in place, doubling, to the cells asked for. Every step is a sample of
them. A file stored in parts much coarser than its blocks can grow part by
part, each part read once, ending exactly as opening the share directly.
The same rule prices a file picked from this computer.
Apply in place. A larger or smaller uniform or blocks share under
the same seed (and every cell of a file Cellucid wrote) changes the open
cells in place: raising reads only the added cells (their positions, their
values of every loaded field and gene, the gene index for them alone),
lowering reads nothing. Camera, every view's field, filters and settings,
highlights, labels, pushed columns, hand-hidden cells and analysis windows
stay; highlights and labels made from a column take the new cells as
opening them would. Each per-cell structure ends bit for bit what opening
at that share directly holds. On the graphics card only the added cells are
sent, and lowering cuts the end. Another method or seed reads the dataset
again through the parts already open, carrying the same state by original
cell position.
Nothing read twice. Another share of the same file, in the same or a
new reader, takes what was already fetched and decoded (within the tab's
memory budget); Zarr folders on a web host, picked folders and file sets
are reused while their version is unchanged.

Reading at the floor

A share reads only its cells' parts, element by element (obs of every
encoding, embeddings, vector fields, graphs, X as CSR, CSC or dense,
layers, raw, stored outlier quantiles, MuData modalities): a permanent
test holds every element, at every share, to the stored chunks that hold a
kept cell, each fetched once. Opening 1 % of CELLxGENE's 2.3M-cell Human
Lung Cell Atlas: first view 3.8 MB in 143 requests at 8.2 s (was 20.3 MB in
2,645 requests), at the floor computed from the file's layout.
The first view never waits for gene names. The gene list (the var
index, or the column Data mapping names) is read right after it, and no
other var column; meanwhile the gene search, gene-based analyses and their
pickers say Reading gene names…, and sessions, exports with genes and
data-mapping changes wait.
Faster reads over a network: HDF5 chunk-index nodes no longer read
before a gene (the lung atlas read 12,382 of them first); index nodes of
one task are read as one burst (all HDF5 formats; 100 Loom genes 11.1 →
7.8 s over 150 ms / 5 MB/s); every S3 address of a public bucket is used;
neighbouring reads are merged when that saves time; HTTP/1.1 hosts are
recognised; sharded Zarr v3 reads all its shard indexes at once, then all
inner chunks; uncompressed Zarr chunks are read in part; two columns of a
latent space read only the open cells' rows.
Stricter, steadier reads. A host answering another byte range, a file
whose size, ETag or modification time changed while open, is reported,
never read into what was read before. Busy hosts (429, 502, 503, 504) are
waited for with spaced retries and fewer requests at once. A failed gene is
read again on the next attempt. A shared read is cancelled only when every
reader waiting on it has withdrawn.
Decoders. Safari reads Zstandard-compressed Zarr about seven times
faster (55 → ~395 MB/s); a Blosc frame with the delta flag is now decoded
as c-blosc2 writes it (it was misread).

Genes on every path

Genes leave the viewer by identifier (the var index) everywhere: gene
sets, the gene coloured by, GMT files, DE and marker CSVs, .h5ad
exports. When Data mapping names the genes by a var column (a CELLxGENE
file's feature_name), the name shown goes alongside; named by the var
index, nothing does.
The first gene of a cell-major matrix says what it costs before it is
read (bytes and time at the measured speed), with Cancel; on a share,
its colours fill in as cells are indexed, each step a sample of the open
cells. No silent rescans; errors say what failed.
The gene index is kept across visits. For a matrix stored cell by cell
(CSR, or dense row-chunked) the gene index is kept in this browser under
the file, the matrix and how the cells were chosen (never the share):
opening the same cells again, another day, reads genes at once, and a
larger uniform or blocks share indexes only its new cells. Kept only for
a file whose version is known; checked before use against its record, the
file's row layout and, unless the version names the content, a fingerprint
of the file; each gene's piece carries a CRC-32.
Fixed: the first gene of a share of a Cell Ranger .h5; a text field
for stratifying; SingleCellExperiment graph weights (were all 1); a
MatrixMarket first gene or cell that looked like a header (IDH1,
cell0).

Files that cannot be read in parts

Said up front, kept for next time. An R object, a MatrixMarket folder
or any file on a host that serves no parts of files: the opening says it is
read whole, how much, and that a Cellucid-ready AnnData is read in parts
instead. What it is read into is kept in this browser's storage across
visits (for a file with a known version) and opens again in milliseconds,
reading nothing; converted stores are written in priority order with a
gene-major X (a gene of a 50,000-cell Seurat object: 7 ms). Session ›
Open data › Stored counts what is kept, with Clear.
Checked as it is read. Everything kept carries a CRC-32 of every
16 KiB; damaged bytes fail the read as damaged, the copy is removed and the
next opening reads the file again.
Safari keeps converted files and gene indexes across visits like
Chromium and Firefox; only private windows keep them in memory for the tab.

Rendering at scale

Levels of detail are uniform samples. A level is the first cells of
the open cells' priority order (1,000 cells, then a quarter more at each
level), so a reduced level shows the cloud's density and mix of categories
as all cells do. Cells already in that order (a file Cellucid wrote, a
share) draw every level at once (100M 3D: 0.33 s after levels of detail
were turned on, was about 29 s); the spatial order culling uses is taken a
few million cells a frame, without stopping a frame.
100 million cells in 3D. Levels of detail at 100M 3D were silently
unused (a GPU texture Chromium refused); fixed, with a notice if a GPU ever
refuses. Positions and colours live in segments of 16,777,216 points, so
no WebGL buffer limits the cell count; an array the browser will not
allocate in one piece is made in WebAssembly memory (3D up to 357.9M
cells); the velocity overlay's textures are layered. Memory per cell
measured at 100M 3D: 54 bytes drawn, about 50 after levels of detail are
built (was 65); 150M cells in 2D fit 16 GB.
Keep view and recolouring above 16.7M cells on Chromium work (a
vertex attribute released without a buffer kept stale bindings); released
kept views free their GPU memory; GPU errors name the call that failed.

Analyses on a share

First answers on a sample. Differential expression, Marker genes, Gene
signature, Correlation, Composition, Dot plot, Detailed and Quick Insights
show a first pass on a uniform sample of the open cells when a run is
predicted to take more than about a second, then refine in place to every
cell; the final result is bit for bit a single run, and only it is saved
or exported. A first pass reads only its sample's part of each gene where
the file allows.
Analyses follow the open cells. When the open cells change, results are
marked Stale · Run (its tooltip gives both counts); a stale result is
not saved in a session, not exported and withdrawn from notebooks until it
is run again. Differential expression and Marker genes no longer kept the
earlier cells' groups after an Apply.
Stored outlier quantiles hold on a share (they describe the whole
dataset); quantiles computed on the open cells say they are an estimate.
Sparse statistics. Differential expression and Marker genes read each
gene's stored values, never a dense column (100M cells at 10 %: peak
2.4 GB → 0.6 GB).
Every analysis is checked against an independent reference on every way
cells open (each method, progressive, Apply, sessions, served data) and
every format and source.

Round trip and exports

Viewer protocol schema 2 (shared fixture re-pinned in all three
repositories): every cell either side names is an original position; new
events result, result_removed, results, color_by; new commands
setField, setGeneSet; ready carries n_obs. Results carry an epoch,
so a reloaded page replaces an older page's results.
Exports without a notebook: the highlighted cells as an obs-name CSV
or an .h5ad subset (written in the browser; anndata, h5py and Cellucid
read it), any obs field as a CSV (Export beside Duplicate), gene
sets as GMT. Exports list cells in the dataset's own order; genes by
identifier, with the names shown beside when a var column names them; DE
and marker CSVs name genes by identifier.
From a notebook: pushed columns and gene sets show as fields and under
From notebook in Gene signature and Dot plot, and are saved with the
session.

Genes, legend and Session panel

Gray for zero, off by default: a switch in the legend (for genes with
no negative value) draws a gene's zeros in the theme's gray beneath the
expressing cells, everywhere the colouring is drawn (2D/3D, smoke, kept
and compared views, the cell card, figures). Per field and saved in
sessions; a session saved with 0.9.5 restores with it off.
Genes read for: under Cells to open, a slider of the share of the open
cells a gene is read for (now, 1, 5, 10, 25, 50, 100 %), priced in cells,
download and time at the connection's measured speed. Read grows the
sample, reading only the added rows; Stop keeps what is read.
A compact Session panel: Dataset, Open data, Stored (one line beside
Manage…), Session file, then Subsampling and Data mapping,
collapsed with a one-line summary. Minimum and Maximum of the visible range
are one row each.
A gene chosen is shown, or the app says why. Enter on a gene while more
cells are opening colours by it; a gene chosen while the gene list is
replaced follows to the new list; a gene or field that cannot be shown yet
says so ("Gene names are still being read: CD3E can be chosen once they are
listed."). Every gene search uses one lookup: case-insensitive, dotted ids
as written, Ensembl accessions with or without a version. In the analysis
gene picker, Enter chooses the gene typed (it chose None).
A dataset is called by its name, never by its address, in openings,
failures, the opening progress and sessions.

When a host or the connection fails

Whose problem it is, for every source. An opening, a gene or an
analysis names the provider, its host and the status: not answering (5xx),
limiting this network (429, with the wait it asks for), refusing this
network, private or expired (401/403), nothing at the address (404/410),
or this computer offline. The requests made are under Details, with
Try again where asking again may help; every place that reports a
host's answer uses the same words. When unrelated hosts all stop answering
at once, the message says it is most likely this computer's connection.
GEO no longer calls a link wrong while NCBI is down: before a listing
is taken to mean a file is missing, NCBI itself is asked.
A worker that cannot load its code says why ("this browser is
offline"), and a build that failed offline is tried again once online.

Steadier reading and drawing

Closing a dataset stops every read it started, background reads
included, and an opening that fails cancels the other requests it made.
Closing the page ends what the dataset on screen was waiting for. Work
cancelled before it began no longer runs (a MatrixMarket pass, the Google
file picker, a catalog open).
A MatrixMarket folder reads its matrix once, however its cells open
(1M cells: 90.6 MB read once, genes in 6.5–6.8 s, was 24.8–25.8 s); a pass
serves every read waiting for it. A folder whose matrix is still arriving
opens in every engine.
Level of detail and culling are kept only where they are proven
faster. A frame's GPU time is read through the GPU timer where the
browser has one, else bounded by the fence polls; the page turns them on
when a frame of every cell takes over 33 ms on the GPU and keeps them only
for a gain larger than it can measure, checking again after a restored
graphics context, cells added or cut, or another representation. A
caller's own choice (a notebook, the benchmark) is kept. The Performance
tip says what was measured, and nothing is measured where nothing could
be drawn fewer.
Nothing under the pointer moves when the page changes its own drawing:
the highlight count, the selection bar, the Force LOD level row and the
Dataset box hold their place.
A failed Apply of Cells to open says so and keeps the cells open
before; a picked file is read again through the copy already held.
Exported figures keep their tick step in every browser (a power of ten
rounded one bit low in some engines).
The welcome card fits phones and tablets, upright or on its side; the cell
info card waits out a lost graphics context; no Firefox warnings about
unflushed GPU fences.

Python

cellucid.subsample_cells() gives exactly the cells the viewer opens
for a method, size and seed (any count from 1; the viewer opens at least 2).
prepare() stores cells in priority order (priority_seed=0;
None keeps the order given) in .h5ad, Zarr v2/v3, .zarr.zip, AnnData
and MuData: uns['cellucid']['priority'] = {seed, specVersion, nObs, levels}, obs['cellucid_original_position'] and, beside a sparse X,
varm['cellucid_prefix_counts']. Chunks are smaller for shares: per-cell
arrays 2^16 values, X data and indices 2^14. Preparing an object read
from such a file puts it back in its original order first.
cellucid.restore_original_order() puts an AnnData or MuData read from
such a file (also a backed one) back in the order its cells were given. A
record that no longer describes the cells (a subset, a reordering) is
never trusted.
show_anndata() returns a notebook widget (cellucid.Widget, an
anywidget; cellucid.Widget(adata) makes one without showing it). The page
and the viewer channel travel over the notebook's own connection: no port,
proxy, client_server_url or allowed_hosts. Tested end to end in
JupyterLab. Calls that wait for the viewer raise at once inside the cell
that would wait, saying what to do. anywidget is a dependency;
jupyter-server-proxy is no longer one. AnnDataViewer is the
browser-tab viewer (its unused height is gone).
Served in priority order. show_anndata(), serve_anndata(),
AnnDataViewer and cellucid serve serve an in-memory AnnData, and any
AnnData, MuData, Loom or 10x file prepare() did not write, as a Zarr store
with the cells in priority order, made chunk by chunk as the viewer asks: a
share transfers only its cells (1 % of a 1M-cell object: 0.18 MB, was
2.1–10.6 MB). The temporary .h5ad is gone. An in-memory object is
snapshotted before the call returns (memory-mapped, about 2 s per 100M
values), its matrices stored as prepare() stores them. Files are served
at disk speed (CSR X 7 → 109 MB/s). Each served file is listed with a
strong ETag, so the browser keeps what it read across visits and sees a
rewrite.
Round trip. viewer.pull(adata) / cellucid.pull(adata, viewer or paths) writes every highlight, label, score and gene set onto obs /
uns in one call (<NA> for cells a subsampled viewer did not open), from
a live viewer or from the web app's CSV, GMT and session files;
viewer.push(), push_gene_set(), subset(), gene_sets(), results,
selection (obs names), write_gmt() / read_gmt(). Pull and subset
refuse an object whose cells are in another order or that lacks the
viewer's cells.
One session API. request = viewer.request_session_bundle(), then
request.result() or viewer.apply_session_to_anndata(adata, request).
get_session_bundle() is removed. Sessions saved on a share apply (cells
not open are <NA>); set_data_mapping() no longer waits or returns.
Viewers of a prepare() file name cells correctly in Python (obs names in
original order); count messages read "1 cell".

Docs

Fourteen runnable round-trip notebooks, seven Python–R twins with
"Open in Colab" badges: show a dataset in any notebook, pull results back,
push clusters and gene sets, read the web app's exports, re-analyse an
exported .h5ad, subsampled views; a gallery page lists them side by side.
New pages: Subsampled loading (web app) and the Subsampling
specification; the Cellucid-ready AnnData specification gains Priority
order; the notebook pages describe the widget; the published-data
tutorials open CELLxGENE's 2.3M-cell Human Lung Cell Atlas.
Every page was checked against this release's behaviour (served stores,
kept stores, sessions, analyses on a share, rendering limits).

cellucid 0.9.6 cellucid 0.9.6 on Python PyPI