annzarro 0.5.0 on Python PyPI

AnnZarro v0.5.0

This release adds an admin web interface to AnnZarro. On a shared server, an administrator can now add data directories, manage users, give an external collaborator only some of the datasets, switch panels and external hosts off, and read the server's health and an audit log, all in the browser. It also protects every state-changing request against forged requests, hardens the server against requests that hold a worker, and makes very large datasets cheaper to open and read.

Admin interface

/admin (login on) has six sections: Users, Data directories, Server health, Panel sets, Settings and Audit log. A change needs the admin's password entered within the last 30 minutes, and every change is written to the audit log. Behind a reverse proxy that terminates TLS, set server.proxy_count, or the page refuses changes.
Users: add and remove users, reset passwords (a generated password is shown once), grant or revoke admin, and clear a lockout. The last admin cannot be removed or demoted, on the web or on the command line. Everyone can change their own password at Change password in the header.
Data directories: with server.admin_data_roots set, an admin adds directories in the browser, below the parents the operator allows. They appear in the dataset picker at once.
Dataset access for external collaborators: restrict a user to the folders, stores and remote prefixes you grant. Everyone else keeps seeing everything. A restricted user gets the same "not found" for a dataset they may not read as for one that does not exist. Manage it in the Users section (Access), or with annzarro user access.
Settings: switch panel types off for everyone or one user, and control what Gene Set Analysis may send out (all external hosts, individual services, the species lookup, the Links). The server tells the browser which hosts it may contact and the browser refuses any other. Enrichr and Reactome are off by default on a shared server.
Server health: each worker's read budget in use, cache size, requests and refusals by reason, and who was seen recently. Panel sets: list by owner, reassign or delete. GET /status and GET /cache/info are now admin-only on a shared server.
Audit log: each record carries the hash of the one before it, so an edit in the middle is found (annzarro admin audit verify). The log is also written to the annzarro.audit logger.
A change by an admin, or with annzarro user ..., reaches every gunicorn worker on its next request, with no restart.

Security

Every request that changes something now needs the session's token plus a same-origin Origin (and Sec-Fetch-Site). Sign-in has its own one-time token, and signing out is a POST.
Pages carry a Content-Security-Policy that lists the external hosts the user's settings allow.
A request body over server.max_request_mb (64) is refused with 413: one with a length before it is read, a chunked one as soon as it passes the limit.

Performance and limits

Cell and gene names (/data/cells, /data/genes) are streamed a chunk at a time. For a 95.6 million cell store that is 0.42 GiB and 0.14 s to the first byte, where it was 15.2 GiB and 14.4 s. server.names_stream_max_concurrent (2) bounds the threads slow readers hold. Behind nginx, use proxy_http_version 1.1.
Sharded stores: a subset read takes a shard's rows in bounded batches instead of the whole shard. A 1,000-row read of a 5M x 50 store in one 850 MB shard peaks at 47 MiB instead of 757.
Colouring by a gene on a cell subset of a CSR X reads each chunk once: 10,000 cells of a 175,000-cell store took 70 to 118 s before.
Subsets share the cache and the compute fairly between users. The cell subsets a worker computes at once are charged by their cells against server.subset_budget_mb (15% of RAM by default); while others wait, one user holds at most server.read_user_share_percent (50) of it and of the pairwise read budget. A subset that cannot start within server.subset_wait_s (30) is answered 503 read_busy, and one larger than the whole budget runs alone.
Numeric subset filters compare the column as the array it is stored as. Six at once on a 95.6 million cell store take 1.8 s and 10.8 GB, where they took 18.8 s and 30.5 GB.
Regular-expression name search: a pattern that could hold a worker is refused (400 regex_too_complex, or regex_too_long over server.name_search_max_regex_chars), and every match runs under server.name_search_timeout_s (10 s) with the regex package, which stops inside a match. The picker says when a search ran out of time.
Remote stores: one request may spend at most server.remote_request_timeout_s on remote reads (120 s on a shared server, 600 s for one user). Past it the reply is 504 remote_deadline.
Other limits: panel-set quotas (max_sessions_per_user 500, max_sessions 5,000, max_sessions_mb 256), dataset_scan_max_entries (200,000) for the directory listings and /status, a uns member is sized against server.max_read_mb before it is read whole (strings by their real length), and gunicorn replaces a worker after max_requests (20,000) requests.
annzarro store check reports, from metadata only, what in a store's layout makes AnnZarro slow, with the cost in clicks and a fix for each (--json, --fail-on).
In large-plot mode, hovering finds the nearest point in 1 ms instead of 43 at 95.6 million points, skips categories hidden with the legend, and the large plot's graphics memory is freed when a subset makes the plot regular.

Fixes

Switching a plot axis between a text field and a number field draws the axis as what the new field is, and refits it. In large-plot mode the x and y pickers grey out the categorical columns, with the reason, until a subset is made.
A table column named like orig.ident or x.y.z shows its values (they showed N/A).
A wheel zoom of a 3D plot is kept when the focused cell changes or the plot is redrawn.
The open dataset list follows the window height, up to 70% of it or 640 px, with one scrollbar.
Center at 0 with Lock Range centres the colour scale on the locked range (±the larger of the two limits) and brings the typed limits back when it is turned off; a locked range of 0 to 0 disables it. Log with Lock Range keeps the typed limits too, so turning Log off restores a typed Min at or below 0. Unlocking a centred panel recentres it at once.
A panel opened from the welcome page before the first dataset had started loading no longer keeps its "No dataset loaded" screen and hidden controls.
Splitting a panel, choosing a panel type in a new section and closing a section ease to the new sizes instead of jumping (no animation with the system's reduced-motion setting).
A missing store is the same 404 not_found on every data route (five answered 404 or 500 in their own words).

Docs

New Python API reference (annzarro.__all__: readers, store check, the server factories, the figure recipe), with types linked to anndata, zarr and numpy.
Desktop app or server: the landing page, the README and a new getting-started page compare the two ways to run AnnZarro and say which to choose.
New pages: The admin interface and the Admin API; the policy registry; Resource limits; and a section on restricting a user in the authentication guide.
CONTRIBUTING.md and a code of conduct.
The README and CITATION.cff cite the bioRxiv preprint.

Desktop

The macOS app is signed with a Developer ID and notarized by Apple. Install it by opening the .dmg and dragging AnnZarro to Applications.
The desktop app bundles the regex package; its licence is in the app's third-party notices.

Upgrade notes

Forged-request checks. Scripts that sign in with a cookie and send POST, PUT or DELETE must now send the session's token in X-AnnZarro-CSRF (the page has it in <meta name="annzarro-csrf">). curl and annzarro export on a server with login off are not affected. Behind a proxy that terminates TLS or rewrites Host, set server.proxy_count, or every change is refused with 403 csrf_failed.
Sign-in and sign-out. The login form carries a one-time login_csrf field; a script that posts to /login must fetch the form first. /logout is POST only; GET /logout shows a button.
Content-Security-Policy. Every page now has a connect-src that lists only the external hosts the policy allows. A customised page or an embedding that contacts other hosts is refused by the browser.
External hosts. Enrichr and Reactome are off by default on a shared server; switch them on in Settings or with integrations.gene_set.services. integrations.external_requests accepts a new value, none. ANNZARRO_EXTERNAL_REQUESTS beats the YAML in both directions.
/status and /cache/info answer admins only on a shared server.
Remote deadline. A remote read that took longer than server.remote_request_timeout_s in total now fails with 504 remote_deadline. Keep the proxy's proxy_read_timeout at or above it (120 s). 0 turns it off.
Subset budget. Cell subsets now wait for, and are refused past, server.subset_budget_mb and server.subset_wait_s; raise them if users see 503 read_busy on subsets.
New limits and defaults: the keys under Performance and limits above, all in server.*; see Resource limits in the docs. The admin state lives in annzarro_admin/ beside the users file; back it up with the users file.
Command line. annzarro user commands must run as the account that owns the users file's directory.

Downloads

Desktop apps for macOS (Apple silicon and Intel), Windows and Linux are attached below. macOS: open the dmg, drag AnnZarro to Applications and open it. Windows: SmartScreen asks for More info > Run anyway. Details: https://annzarro.readthedocs.io/en/latest/getting-started/desktop-app.html. Verify downloads against SHA256SUMS.txt. The Python package: pip install annzarro.

annzarro 0.5.0 AnnZarro v0.5.0 on Python PyPI