cellucid 0.9.4

cellucid 0.9.4

on Python PyPI

Version 0.9.4 is a patch release on 0.9.3. The export format stays 3, so
exports written by 0.9.3 open unchanged; prepare(obs_categorical_dtype="uint32")
is the one addition to what an export may hold. The web app, the Python
package and the R package now share one version and are released together.
The main changes are:

• Six new backgrounds (Atelier, Nebula, Abyss light and dark, Aurora light
  and dark), prepared in the background and sharpened while the GPU keeps up.
• No fabricated limits: any number of kept views, selections of any size,
  direct H5AD and Zarr files of any size, categorical fields with more than
  65,535 categories, and session files bounded only by their format.
• Analysis on the workers: summaries, tests, histograms, violins and
  density plans leave the main thread; gene downloads go through one scheduler
  with a payload cache; marker-gene clustering is bounded by memory, not a
  count.
• Jupyter selections of any size reach Python as binary indices.
• Sessions restore the vector-field controls after the embeddings they
  depend on, and keep every view's embedding.

Added

• Web app, version: the web app declares the release version it shares with the Python and R packages: version: in its CITATION.cff, a cellucid-version meta element and softwareVersion in the page's structured data, and the footer prints it before the build id. The three are released together under one version.
• Web app: Six new backgrounds, in light and dark pairs: Atelier (light) and Nebula (dark), Abyss (light) and Abyss (dark), Aurora (light) and Aurora (dark), listed under the Light and Dark groups of the Background menu; Studio (light) stays the default. Each is a painted stage that turns with the camera, with a nearer layer that slides against it for depth: Atelier's paper studio, where the cells cast a soft contact shadow that follows filters (and glows through the paper seen from below); Nebula's deep field, tinted by the colours of the field the cells are coloured by; Abyss's sunlit shallows and deep ocean with glowing marine snow; Aurora's dawn and night skies. A look is prepared in the background in idle time the first time it is chosen (its sky coarse first, then sharpened level by level while the GPU is measured to keep up, so it appears within a fraction of the full work on every machine and reaches 1024² per face on a capable one), costs nothing while the scene is still, keeps its large areas outside the lightness range of the category palettes, and appears without its fade under the system's reduce-motion setting. Only the look on screen keeps GPU memory (at most about 27 MB). Sessions and dataset presets can name the looks, and figure export redraws the look under the points in PNG and SVG (as an image) when the figure's background is Match viewer. (backgrounds)
• Web app: Keep any number of views: the four-view ceiling is gone. GPU memory decides instead: when an allocation is refused, the colour and visibility stores of kept views that are not on screen are released first and rebuilt, unchanged, when shown again. The benchmark harness's view-count axis takes any number too.
• Web app: Selections of any size: the annotation, lasso, KNN and proximity tools, category highlights and Jupyter commands hold selected cells as one bit a cell instead of a JavaScript Set, which Chromium refuses above 16,777,216 entries, so selecting a large cluster in a 30M+ cell dataset failed. Combining selections (union, intersection, difference) works a word at a time.
• Web app: Categorical fields with more than 65,535 categories (barcodes, free text) load: they use 32-bit codes (export format uint32, missing marker 4,294,967,295) in prepared exports, cellucid serve, Jupyter, direct H5AD/Zarr and saved sessions. A direct H5AD/Zarr or served AnnData with such a column used to be refused as a whole. Existing exports are unchanged. (limits)
• Web app: Direct H5AD files have no size ceiling: files above 512 MiB used to be refused before they were read. The file is read from disk as each view needs it instead of being copied into memory, in a background worker, so opening it no longer blocks the interface (a 250 MB, 1-million-cell CSR file: about 0.8 s of main-thread loader work and 1.1 GB peak before, about 0 s and 0.8 GB now; it opens in about 0.25 s instead of 0.7 s). A row-major (CSR) matrix is reorganised by gene in the background right after opening, with progress shown, and a gene asked for meanwhile is answered from the same pass; when the reorganised matrix does not fit the memory budget it is kept on disk (the browser's private file storage). The fixed working-set ceilings on sparse matrices, dense matrices and observation columns are removed. (limits)
• Web app: Zarr directories and Zarr ZIP archives load through the same background reader, with several chunks decoded at once, and without the fixed per-array and per-archive size ceilings; every chunk is still checked against the exact size its shape and dtype declare. (limits)
• Web app: Memory budgets for kept layouts, inactive fields, the AnnData cache, decoded blocks and the gene index scale with the memory the browser reports (Chromium) instead of fixed values, and they decide only what is kept for reuse: no dataset is refused for them any more. Browsers that report nothing keep the previous values. (limits)
• Web app: Latent-space outlier quantiles of a direct H5AD/Zarr file work for a latent space of any size: the 512 MiB latent-space budget is removed. The latent space is streamed from the file in row slabs in the reader worker, with the same bit-identical rule, and kept in memory only while it fits the reader's budget. (limits)
• Web app: Categorical fields made from highlight pages work at any cell count (they were refused above 16,777,216 cells, a JavaScript Set limit), an intersection field combines up to 32 pages (was 12), and the limits of 20 user-defined fields and of 256 characters for a field name or category label are removed. (limits)
• Web app: Session files have no 16 MiB manifest or 512 MiB chunk limit; the only limit is the format's 4 GiB − 1 per frame (its length is a u32), and every inflate is still bounded by the chunk's declared size. (limits)
• Web app: Prepared exports have no 512 MiB size ceiling per file: a 3D dataset above 44.7 million cells was refused. Every payload is checked against the exact size its manifest declares, and one the browser cannot allocate is reported as too large, naming the bytes. Compressed payloads above 4 GiB load too (their gzip trailer is compared modulo 2^32).
• Web app: A sample dataset's published starting view has no size limit (it was refused above 32 KiB, or 64 KiB decompressed): the catalog now declares the state file's exact length in state_bytes beside state_sha256 (exports-v3/datasets.json, and cellucid serve's dataset listing), and the viewer reads exactly that many bytes and checks the digest. A catalog entry advertising a state must carry all three fields. (limits)
• Web app: Camera Path: no limit of 100 keyframes, 60 seconds per transition or 40 characters per label; a transition still lasts at least 0.1 s. (limits)
• Web app: Highlight selection undo reaches every earlier step: the oldest steps go only when every tool's recorded selections together outgrow a share of the device's memory (was 5 steps per tool). (limits)
• Web app: The cell info card lists every loaded categorical field (was 4), adding columns when they do not fit the window's height. (limits)
• Web app: The gene search list lists every match (built 100 at a time as you scroll or arrow; Home and End jump to the ends) instead of the first 100 with "…and N more". (limits)
• Web app: Notifications retired early to keep five cards on screen are kept: a "Show N earlier messages" control under the stack lists every one of them. (limits)
• Web app: Community annotation: a Pull has no limit of 10,000 user files or 64 MB in total. Changed files are fetched and cached about 32 MiB at a time, and the merged view is compiled by streaming the cache one file at a time; an interrupted Pull keeps what it cached. CAP lookup results are kept whole (results with more than 100 synonyms, 200 markers or 512-character text were dropped), every fetched result is shown (was 20 of 25), and synonyms, markers and author handles are shown in full. (limits)
• Web app: Analysis: the gene list lists every matching gene (built a page at a time while scrolling or arrowing) instead of the first 50; analysis windows and saved plot options have no count limit. (analysis, limits)
• Web app: Marker genes: clustering is no longer limited to 500 genes or groups. It is admitted by the memory it needs (about 100 bytes per pair of genes: about 100 MB for 1,000 genes, 1.6 GB for 4,000) against the analysis memory budget in Performance settings. A request past the budget is refused with that size and what to change, instead of "maximum is 500". (analysis, limits)
• Web app: Jupyter: selections of any size reach Python (a million cells arrive in about 120–200 ms). A confirmed selection is posted to the notebook's server as binary cell indices (POST /_cellucid/selection, little-endian uint32, ascending) instead of a JSON event sent with keepalive, which the browser refused above 64 KiB (about 8,000 cells). Validating and encoding one million cells takes a few milliseconds instead of 150–300 ms, and no step builds a per-cell Set, so selections above 16,777,216 cells are no longer refused. No event is sent with keepalive any more. Works with the matching cellucid Python version, released next. (DX-17)
• Web app: Gene downloads (analysis and the sidebar) go through one download scheduler: a gene the user chose starts at once and no new download of a long run (DE, markers, heatmap) or of a prefetch starts until it has arrived; the browser gets the same order through the fetch priority hint, and a prefetched gene the user then chooses is raised, not downloaded twice. Observation-field loads count as foreground too. A finished or aborted run cancels its queued and in-flight downloads, and replacing the dataset ends every download of the old one, so a stalled read can never hold back the next dataset. (analysis, network)
• Web app: A dataset load keeps its downloaded gene payloads (compressed, up to 64 MiB) and decodes an evicted gene again from them: a second DE or marker run over the same genes makes no requests (pancreas over 4G: 3,753 revalidation round trips before). Long runs download the payloads of the genes after their decode window ahead, as many at once as the connection needs (the window starts at 8 and doubles while throughput grows, never more than half the cache), so a run is bound by bandwidth instead of round trips. (analysis, network)
• Web app: A gene the user waits for shows its download progress in its loading notification. (analysis, network)
• Web app: The documentation moved to https://docs.cellucid.com (the old Read the Docs addresses redirect there); the footer, Cite, info-tip and welcome links point at it. Search and sharing metadata: a description that names UMAP, 3D, Scanpy/AnnData and Seurat; og:image:alt, twitter:image:alt and og:site_name; corrected JSON-LD (WebApplication, operating system "Web browser", licence, documentation, sameAs for GitHub, PyPI and the R package), still allowed by its CSP hash; a crawlable "About Cellucid" text with links to the docs, GitHub and citing; a sitemap.xml whose <lastmod> is the build date, written by scripts/generate-web-asset-inventory.mjs; a custom 404.html; an llms.txt; the manifest name "Cellucid"; and a preload of the interface font.
• Web app: Data mapping, Embeddings: one compact row per array -- checkbox, name and dimension; an embedding that is on adds a second line with Default and guessed, one that is off stays one line. A long name wraps to a second line instead of losing its columns to an ellipsis. The dimension of an AnnData array is a chip with a chevron (3D ⌄) that opens the row's editor, closed until opened: Dimension, one labelled column menu per axis on one line, and, for an array wider than three columns, Add columns (it replaces the "+ Columns of …" pill under every wide array). Nothing in the panel overflows the sidebar on a desktop or a phone. (data mapping)
• Web app: Data mapping: every obsm array of two or more columns can be shown in any dimension its columns allow, at any of its columns -- a 2-column UMAP in 1D or 2D, a 3-column array in 1D, 2D or 3D, as a wide array already could. The guessed default is unchanged; views, sessions, highlights and figure export take the new selections like any other (UMAP 1, Spatial 1,3). (data mapping)
• Python: Jupyter: POST /_cellucid/selection receives selections as binary cell indices (little-endian uint32, strictly ascending, with no size limit; a body the server cannot hold is refused with HTTP 413), authenticating before it reads the body; @viewer.on_selection still receives cells as a list[int], now always ascending. POST /_cellucid/events refuses selection and session_bundle records (HTTP 400): only their own routes produce them. Selections were previously capped by the event endpoint's 1 MiB limit. (DX-17)
• Python: prepare(obs_categorical_dtype="uint32") writes 32-bit categorical codes (obs/{index}.codes.u32, missing marker 4,294,967,295) for fields with more than 65,535 categories, inside export format 3; existing exports are unchanged and an older web app refuses such an export by name. The width error now names the width that fits ("Pass obs_categorical_dtype='uint32'."). serve() and show_anndata() serve a string or boolean obs column with more than 65,535 distinct values with uint32 codes instead of refusing the whole AnnData, and session bundles carry uint32 user-defined codes (encodings 4 and 5). (limits)
• Python: cellucid serve lists a published default session with its exact length (state_bytes) beside state_manifest and state_sha256, and reads a bundle of any size (the 32 KiB limit is removed), hashing it as it streams. debug_connection() accepts the three fields together. (limits)
• Python: Documentation: served from https://docs.cellucid.com, the Read the Docs canonical domain; every link in the package, README, CITATION.cff and docs uses it. Every page declares <link rel="canonical"> on that host (html_baseurl), sphinx-sitemap (pinned in the docs extra) writes https://docs.cellucid.com/en/latest/sitemap.xml, and a robots.txt at the root of the default version advertises it.

Changed

• Web app: When several views' spatial trees land at once (four views of four representations), each view switches to its tree in its own frame instead of all in one long frame.
• Web app: Slider, legend and camera-path changes are drawn in the frame they happen in, not the next one: they used to wait for an animation frame that ran after the viewer's, which added one frame of latency to every slider step (about 50–90 ms at 10 million cells in four views).
• Web app: Starting a drag or a wheel zoom in another view no longer waits for the sidebar: the camera follows the view at once, and the sidebar switches to it (legend, counts, controls) when the gesture ends (on release, or once the wheel has rested for 150 ms). A wheel turn that cannot zoom further, or a wheel in free-fly mode, no longer redraws the scene.
• Web app: The full-window selection overlay (lasso, proximity and neighbour rings, the projectile charge ring) is hidden while nothing is drawn on it, so the browser no longer composites an empty window-sized layer over the plot every frame.
• Web app: Category counts in the legend are recounted only while the legend can be seen: with its panel collapsed or the sidebar hidden, a filter change no longer recounts every category; the counts update when the legend is shown.
• Web app: A camera drag captures the mouse until it is released, so dragging across the sidebar or an analysis plot no longer makes the page work out hover styles and plot tooltips under the pointer at every move.
• Web app: Shader programs are compiled in batches: the point programs compile 9 shaders instead of 14 (shared vertex shaders) and the velocity overlay 8 instead of 14, with every link issued before any status is read, so the driver overlaps the work and the page waits once per batch instead of once per compile and link. Turning velocity on and starting the app pause less.
• Web app: Multiview: a view whose camera and contents did not change keeps its pixels instead of being redrawn, so dragging, zooming or flinging the camera of one view of four redraws only that view, and the settle frame after a change in a single view draws nothing.
• Web app: Zooming through levels of detail no longer rebuilds the figure-export preview at every step; it rebuilds once the camera settles.
• Web app: An enabled velocity overlay with nothing to draw (no field for the layout, nothing visible, particles off when zoomed out, smoke mode) no longer keeps the scene redrawing every frame.
• Web app: Filtering no longer blocks on the GPU: writing alpha, colours and draw orders into storage the GPU already holds no longer waits for the previous frame to finish (a filter tick at 10M cells in four views spent about 40 ms waiting).
• Web app: Framed (cropped) figure exports are drawn at their full resolution: a frame's points are rasterized at the pixels they land on instead of being enlarged from the whole view, so a framed PNG, or the point layer of a Hybrid SVG, is as sharp as an unframed one.
• Web app: When a Remote server connection drops, the viewer says so at once, even while another dataset you chose is still loading; the notice is completed with "Reconnect to continue" once recovery has run.
• Web app: Filtering a kept view uploads only the rows of its visibility texture that changed, reuses its own transparency array instead of allocating a new one per update, and skips its highlight, edge and overlay work when nothing moved.
• Web app: Recolouring a kept view rewrites its colour store in place instead of allocating a new store and vertex array each time (about 70 MB transient a view at 10M cells).
• Web app: A kept view's colour store is RGBA8 on a 4-byte stride, like the live one: Metal (Chrome and Safari on macOS) needs that alignment, and a 3-byte RGB store made the browser keep and re-convert a hidden 4-byte copy.
• Web app: Figure export of the Studio background no longer has a ceiling on the dots it considers: a flat embedding's dot surface seen at a grazing angle is enumerated only where it shows, and the exported marks are unchanged.
• Web app: An idle page no longer wakes 60–120 times a second: the frame loop stops once the scene has settled and starts again when something asks for a frame (an interaction, a data or setting change, a resize, a held navigation key).
• Web app: The KNN selection tool builds its neighbour graph as compact index ranges instead of a JavaScript map per cell, so it works on graphs with more than 16.7 million cells and builds much faster.
• Web app: Spatial trees (level of detail, frustum culling, hover) are built in the background worker at every dataset size; above 512 MiB of positions (44.7 million cells in 3D) they were built on the main thread, freezing the page for about 15 s at 50 million cells. Positions go to the worker in 64 MiB slices, so the page never holds a second full copy.
• Web app: Analysis keeps the interface responsive on large pages: the Detailed and Gene signature summaries and tests (from 200,000 values), the Quick Insights median and quartiles and the signature-score median (from 500,000 finite values), histogram bins (from 200,000 values) and density-plot plans are computed on the analysis workers, with the same functions and the same numbers as before. Categorical counts are taken once while a page is built and shared by every consumer; gathering a page and aligning two variables for a scatter plot yield to the interface in 12 ms slices. (analysis)
• Web app: Gene search, in the sidebar and in analysis: a gene kept under the pointer or highlighted with the arrow keys for 150 ms is loaded in the background, so choosing it usually draws at once; moving on cancels the load and choosing the gene joins it. (analysis)
• Web app: Violin plots of large pages no longer freeze the interface: the box statistics Plotly's violin calc needs are computed on the analysis workers with the density, and the vendored Plotly takes that calcdata entry instead of building a point record per value. A ten-million-cell violin held the main thread for 1.5 s and 800 MB; it now holds it for under 50 ms and 90 MB, drawing the identical violin. Violin outliers, like box-plot outliers, draw at most 16 copies of an identical value. (analysis)
• Web app: Analysis worker dispatches copy at most 96 MiB per main-thread task; a ten-million-cell density plot used to copy about 1 GB in one 114 ms task. A density over more than 4,096 distinct values sends the page's own values with unit weights instead of a float64 copy and an array of ones (a quarter of the bytes). Quick Insights' category composition yields to the interface like its continuous summary. (analysis)
• Web app: Latent-space outlier quantiles are 4 to 6 times faster (10 million cells by 10 columns: 4.6 s to 0.8 s), bit-identical to cellucid-python and cellucid-r: the latent space is read row by row and large categories are ranked by radix ordering. (analysis)
• Web app: Removed the analysis bulk loaders that nothing called (DataLayer.fetchAnalysisData, fetchComprehensiveAnalysisData, loadAnalysisBulkData, loadLatentEmbeddings); every analysis gene load goes through the field loader and its cache. (analysis)
• Web app: The outlier filter control stays hidden for a field whose every outlier quantile is missing (no category reaches centroid_min_points), as for a field without quantiles. (limits)
• Python: prepare() exports a categorical field where no category reaches centroid_min_points with quantized outlier quantiles, publishing every quantile as missing, instead of refusing the export. The mapped latent-space route of serve()/show_anndata() has no 512 MiB latent-space budget any more and uses obsm without caching a copy. Session bundles accept field names and category labels of any length (the 256-character limit is removed). The release build's sdist limit is PyPI's 100 MB upload limit instead of 1 MiB, and the 4,096-member limit is removed. (limits)
• Python: .cellucid-session reading has no 16 MiB manifest or 512 MiB chunk limit, matching the web app: a frame holds up to 4 GiB − 1 and a length past the end of the file is refused. (limits)
• Python: Jupyter: get_session_bundle() uploads have no 512 MB limit; the upload streams to disk and is bounded by the free space of the temporary directory (HTTP 413 beyond it). wait_for_event() and set_data_mapping() are handed their event as it is recorded, so a burst of other events (thousands of hovers) can no longer push it out of the 512-event log they used to scan. (limits)

Fixed

• Web app: Sessions: the vector-field controls (overlay on or off, the field, its colour map and sliders) are restored after the embeddings, the kept views and the active view, because the field select lists the fields bound to the embedding the active view shows. A session saved while that view showed an embedding without vector fields was refused by a page whose view showed one (Session UI select option "" is not available in current control "velocity-field"), and the other way round the saved field was overwritten when the embedding came back. The block is marked data-state-serializer-after-embedding in index.html; any control whose choices follow the embedding can use the same marker. (K3)
• Web app: The page preconnects to the sample-catalog origin only when it will fetch from it: a page told another catalog (?exportsBaseUrl=, as cellucid serve, Jupyter and the browser tests do) no longer opens a connection to theislab.github.io it never uses. WebKit reported that unused connection's failed TLS handshake as a console error when the network was poor. (assets/js/app/ui/core/preconnect-init.js replaces the static <link rel="preconnect">.)
• Web app: Benchmark harness: a configuration waits until every presented view draws from its final geometry (its spatial tree, with levels of detail when LOD is on, built and adopted by its store) before a measurement window starts. On a slow host the tree built off the main thread landed inside the window's first frames, so the window measured the landing's uploads and error reads and reported sync stalls that no frame of the configuration makes. The viewer exposes this as isPresentedGeometrySettled().
• Web app: Direct H5AD/Zarr: the embedding payload is handed to the viewer instead of being shared with the reader's cache, so evicting a layout now frees its memory and a reload re-reads the file rather than renormalizing an array it already rewrote. Embeddings with a positive range below 1e-8 now get the export format's scale, the same one cellucid prepare and cellucid serve use, instead of a floor of 1. (DX-14)
• Web app: Firefox draws datasets above 30 million points. Every point draw (the point passes, the GPU gather, highlights and figure export) is now split into calls of at most 16,777,216 points; Firefox refused a single draw of more than 30,000,000 and showed nothing. The image is unchanged.
• Web app: Per-point textures (visibility, draw-order permutation, the GPU gather's source) are stacked in layers instead of one 2D texture, so no point count is refused for texture size. On GPUs that report a 8,192 texture limit (Firefox on macOS) a 3D dataset above 22.4 million points could not build its draw order, and a dataset above 67.1 million points (268 million elsewhere) was refused at load; the only device limit left is the texture layer count, at least 2^34 points.
• Web app: Connectivity edges load and draw for any graph the browser has memory for: a k = 15 neighbour graph was refused from about 12 million edges (1.1 million cells) and aborted the whole dataset. Only the edges a panel shows reach the GPU (the slider's K, 28 bytes each), so GPU memory no longer grows with the graph, and the edge shader reads no texture. The shown edges are the first K visible edges of one fixed random order computed on the fly (a keyed bijection), not a shuffled copy of the graph, so a different, equally random sample of edges is shown than before. Counting all visible edges for the slider no longer blocks a filter change. A graph the browser cannot allocate leaves the dataset loaded without edges.
• Web app: Switching back to a representation stays free whenever memory allows: geometry and colour stores kept for switching back are released only when the GPU refuses an allocation (then everything kept is released and the allocation runs again), never ahead of one. A spatial tree whose build failed for memory is built again on the next request once memory has been released, instead of staying off until level of detail or culling was toggled.
• Web app: Figure export draws its point passes on WebGL contexts kept for the page instead of creating and losing a context for every pass: exports start faster, and Linux WebKit (WebKitGTK/WPE) no longer crashes the page during an export (it aborted on a GL call made without a current context while a context was torn down).
• Web app: Figure export works on synthetic benchmark clouds (it refused them: "normalization for view live is incomplete").
• Web app: PNG figure export draws each plot's point raster at exactly the device pixels the plot covers: at 300 dpi a plot's edges fell between device pixels (for example at x = 256.25), so the raster was resampled with a quarter-pixel blur and its bytes depended on the browser's resampling. The raster is now rendered at the snapped size and copied one to one. (figure export)
• Web app: The Python documentation names POST /_cellucid/selection wherever it describes how the viewer reports back to Python; it had described /_cellucid/events alone, which carries every event except selections.
• Web app: Analysis loops that hand the event loop back (statistics, scoring, clustering) now yield through the app's one shared macrotask channel. Each yield used to create its own MessageChannel held only by its handler, which WebKit can garbage-collect with the message still pending, leaving the loop waiting forever. (analysis)
• Web app: Analysis reads a categorical field at the width it is stored in. A field with more than 65,535 categories (stored as uint32 codes) was copied into 16-bit codes, so every category above 65,535 was counted as another one, in Quick Insights, Detailed counts and marker discovery, without an error; marker discovery also refused more than 32,767 groups. Codes are no longer copied to another width, and a uint8 field is no longer widened for analysis. (analysis, uint32 categories)
• Web app: Sessions: restoring a session with several views keeps the live view on the embedding it was turned to. Restoring the views marked the live view as following its dimension's default, so when the session's mapping differed from the one the file opened with (for example a 3-column array saved in 2D), the live view was moved to the default. (data mapping)
• Web app: Data mapping: choosing something in the panel (an embedding, a column, Genes, Neighbour graph, Latent space, Vector fields) no longer scrolls the sidebar or drops the focus. The checklist was rebuilt on every change, and Safari's scroll anchoring then scrolled the sidebar by about a screen; rows are now updated in place, and unchanged text, options and lists are not rewritten. Pressing Default keeps the keyboard focus on it. (data mapping)
• Web app: The antialiasing tooltip and documentation now quote current measurements: in a dense 2D view antialiasing costs about twice the GPU time of a frame (it said about ten times).