dathere/qsv 22.0.0 on GitHub

[22.0.0] - 2026-08-06 📐 The "Data Schematic" Release 📊

qsv's biggest release ever with 530+ commits since v21.1.0. The headliner is viz — an entirely new command that turns a CSV into interactive plotly charts and maps, with viz smart auto-designing a Data Schematic. Schematics are self-contained, offline-capable HTML with static PNG/SVG/PDF export via viz_static. See the gallery.

A Data Schematic is our take on a modern, storytelling data dictionary for the Age of AI. The name is descriptive rather than decorative: a schematic is the drawing form of a schema, and viz smart renders the editable JSON Schema describegpt drafts, together with the statistics that back it. Where a dictionary lists fields, a schematic shows components and how they connect — correlation, process order, hierarchy, temporal pacing, spatial pairing — and every claim it makes is checkable against the data it describes.

It's neuro-symbolic by construction. Statistics, heuristics and algorithms are deterministic and reproducible, so they decide what gets drawn and what the numbers are. LLMs handle what computation cannot — classifying each field against a shared, catalog-wide concept vocabulary, world knowledge, translation — and because that drafted schema is saved as an editable sidecar (see example JSON Schema), a human in the loop (data steward, curator) can ratify or correct anything the LLM proposed and re-render from the corrected schema. The format is defined in docs/DATA_SCHEMATIC.md.

Three more new commands land alongside it: denull (detect the null sentinels that silently degrade typing), fixedwidth (convert fixed-width text to CSV) and clean (remove qsv-generated cache files).

Highlights

viz — a whole new visualization command. Interactive plotly charts and maps from CSV, with 20+ standalone chart subcommands and a viz smart mode that auto-designs an entire Data Schematic from qsv's existing stats & frequency caches. Output is self-contained, offline-capable HTML, with static PNG/SVG/PDF export via viz_static. See the gallery (#302; #4019).
The schematic is explorable, not just viewable. An embedded DataTables data viewer drawer puts the underlying rows beside the charts, cross-linked with map points in both directions, alongside a browsable Data Dictionary drawer — all in one shareable file (#4283; #4284, #4306).
denull — detect the null sentinels that silently corrupt typing. Literal NULL/N/A text makes stats type a numeric column as String, quietly degrading viz smart, schema and describegpt downstream (#4175).
fixedwidth — convert fixed-width text to CSV, with positions auto-detected from a header comment so qsv table --align leftfwf output round-trips (#4168).
clean — remove qsv-generated cache files, with verify-before-delete safety and --dry-run as the default (#3373; #4015).
Your schematic and dictionary speak your data's language. describegpt detects the dataset's content language locally with whatlang (zero tokens) and viz smart renders its entire UI, chart strings and coverage notes in it (#4301, #4310, #4313).
⚠️ Three breaking changes. minijinja 2.22 changes rendered template output (booleans now render True/False, none renders None) across template, apply, fetchpost, describegpt and profile. The cached 2 → 3 migration swaps the on-disk cache backend from sled to redb, invalidating existing on-disk caches and inverting the meaning of a TTL of 0 (was "immediately stale", now "cache indefinitely"). describegpt's bundled prompt file is bumped 8.0.0 → 9.0.0.

Detailed MCP Server and Cowork Plugin changes are documented in the MCP Server/Cowork Plugin CHANGELOG.

Added

viz: new command that generates interactive charts and maps from CSV using plotly — the headline feature of this release. Standalone subcommands cover bar, line, scatter, histogram, box, violin, pie, heatmap, candlestick/ohlc, sankey, radar, geo, map, choropleth, contour, scatter3d, treemap, sunburst, icicle, splom, parcats and bubble. viz smart auto-designs a Data Schematic — a whole one-page rendering of the dataset's schema and statistics — by reusing qsv's existing stats & frequency caches rather than re-scanning: continuous numeric columns become box plots drawn from precomputed quartiles, low-cardinality columns become frequency bars, and the panel set, chart types, axis scaling and ordering are all chosen statistically. Output is self-contained, offline-capable interactive HTML (no network needed to view), with static PNG/SVG/PDF/JPEG/WebP export via the viz_static feature and --open to launch a viewer. Built on the plotly crate with no polars dependency, sidestepping the plotlars/polars version conflict. Gated behind a new viz feature (+ viz_static sub-feature); included in distrib_features/all_features/qsvmcp, not in qsvlite or qsvdp. See the visualizations gallery (#302, #4019).
fixedwidth: new command that converts fixed-width text (fields at fixed byte-column positions, with no delimiters) to CSV. Column positions are given explicitly via --positions/--widths, or auto-detected from a #1,10,15-style header comment — the same format qsv table --align leftfwf emits, so fixed-width output round-trips back to CSV (#4168).
clean: new command that removes qsv-generated cache & sidecar files to reduce clutter and simplify data packaging. Cleans the index (<file>.idx), stats and frequency caches by default; user outputs (--schema, --validate, --moarstats, --all) are opt-in and never touched unless named. Every deletion is verified before it happens, never matched on filename alone: the stats family is anchored on its .stats.csv.json args sidecar (so your own report.stats.csv survives), the frequency cache is validated by its first-line qsv_version/arg_input metadata, and a .idx must pass a csv-index size-multiple-of-8 magic check and have a sibling source. --dry-run is the safe default (and wins over --force); --stale limits cleaning to caches older than their source or orphaned entirely, and --recursive walks a directory (#3373, #4015).
denull: new command that detects null sentinels — literal text like NULL or N/A standing in for a missing value, which makes stats type a numeric column as String (nullcount stays 0, no quartiles), silently degrading viz smart, schema and describegpt downstream. Scans each column once with bounded memory (a 414 MB, 86-column file peaks at ~39 MB). A column is CONFIRMED only when every non-numeric value it holds is a known sentinel and at least two distinct numeric values remain; otherwise it is REJECTED with the reason (off-vocab value, zero-padded codes, too many distinct non-numeric values). --apply rewrites the data, blanking sentinels per column — something a single qsv replace pass cannot do, since it takes one regex across all selected columns. Numeric sentinels (-999, 9999) are deliberately NOT detected: they parse as valid numbers and no scan can distinguish them from real data (#4175).
viz: viz smart now explains why it skipped a column instead of dropping it silently, pointing at qsv denull when the column looks like numeric data held back by a non-numeric token (#4175).
validate: new --split-ragged flag quarantines wrong-column-count rows instead of aborting. validate previously stopped dead on the first ragged row; --split-ragged streams well-formed rows to <input>.valid, diverts ragged rows to <input>.invalid, appends a report to <input>.validation-errors.tsv, and still exits non-zero. Works in both RFC 4180 and JSON Schema modes (#4231, #4237).
stats: new --jsonl and --pretty-json flags emit the per-column statistics to stdout (or --output) as JSON, instead of only to the --stats-jsonl sidecar. --jsonl writes NDJSON (one object per column, machine-friendly); --pretty-json writes a single pretty-printed JSON array of the same objects. Both are mutually exclusive with each other and with --stats-jsonl (#4112, #4195).
extsort & sortcheck: stats-cache aware short-circuit. When a valid stats cache exists, both consult its per-column sort_order statistic to skip work when the relevant single column is already sorted in the requested order — extsort (CSV mode) streams the input straight through and skips the external merge sort entirely (honoring --reverse), and sortcheck answers instantly on the exit-code path (#2116, #4006).
get: new cache-fetch subcommand exports an already-cached entry's decompressed contents to --output <file> or stdout. It is offline — it reads the content-addressed blob directly and never re-fetches the source — making it a first-class alternative to qsv input dc:<name>. Accepts and ignores a leading dc: prefix, and errors cleanly when the name is not cached (#4124).
describegpt: new --infer-null-values flag has the LLM propose each field's null sentinels, emitted into the JSON Schema dictionary's per-property x-qsv object. The LLM proposes a single list; qsv — not the model — splits it into null_values (independently confirmed to occur as a literal value in a String column, listing every observed casing, spelled as found in the data) and null_candidates (everything else, each stamped confirm_required: true). A numeric or date placeholder can never reach the confirmed list: -999 parses as a valid integer and no scan can distinguish it from a real reading, so an LLM is the only thing that can propose one — and a human the only thing that can confirm it. This is the complement of denull, which detects only sentinels in columns that would promote to a numeric type once blanked; a purely categorical column (status = ok/pending/NULL) is confirmed here and ignored there. Sentinels are reported, never applied (#4175).
describegpt: --context-file now accepts non-text files in addition to text/Markdown. The file type is sniffed from its contents (reusing sniff's detectors): CSV and Excel/ODS spreadsheets are extracted to text locally, while PDF and image files (JPEG/PNG/WebP/GIF) are base64-encoded and sent to the LLM as a multimodal Chat Completions attachment (needs a multimodal model & endpoint; ~32 MB max). Word/PowerPoint are rejected with guidance to convert to PDF/text. The context is now injected into the user message instead of the system prompt.
describegpt: new --format okf output emits the Data Dictionary as an Open Knowledge Format document — a leaner, vendor-neutral plain-markdown-plus-YAML-frontmatter sibling of SemanticMd (type/title/description/resource/timestamp/tags frontmatter + a Column | Type | Description Schema table). New --okf-type flag sets the required type key (default "CSV Table"); --ds-source/--ds-updated also map to OKF's resource/timestamp (#4018).
viz: new chart subcommands pie, heatmap (correlation matrix or category×category pivot), candlestick/ohlc, sankey, and radar (polar). viz smart now adds a correlation-heatmap panel when the dataset has 2+ numeric columns (#302).
viz: new chart subcommands geo (token-free projection point map via ScatterGeo, with --projection), contour (2D density of two numeric columns, binned via --bins), and scatter3d (3D scatter over --x/--y/--z). geo and scatter3d honor the --color/--size marker encodings and --series (#302).
viz smart now auto-wires the new chart types: a 3D scatter of the strongest-correlation triple when there are 3+ numeric columns; a 2D density contour instead of the correlated-pair scatter for large datasets (where a scatter overplots); and an offline ScatterGeo projection world-overview instead of mapbox tiles when the coordinates span a continental/global extent (#302).
viz smart box plots now overlay sample points via a size-based heuristic — all points for small data, Tukey outliers for medium, none for large (a fast cache-only quartile box) — overridable with --box-points (now accepted by smart, not just box) (#302).
viz smart frequency bar charts now show a (NULL) bar for empty cells and an Other (N) aggregate bar for the categories beyond --limit (N = the count of distinct categories rolled up), matching qsv frequency's default output. Both aggregate bars are drawn in a muted grey so they read as summaries rather than real categories. New --no-nulls and --no-other flags suppress them (#302).
viz choropleth & viz smart can now build a choropleth from a user-supplied GeoJSON by point-in-polygon binning: each row's --lat/--lon is tested directly against the GeoJSON polygons (even-odd ray casting, handling holes & MultiPolygon) and the matched feature id becomes the location — exact, works for any country/admin level, and needs no geocoding or GeoNames lookup. Points outside every region snap to the nearest feature by default (--no-snap drops them instead); either way a coverage note reports how many points missed every polygon. Wired into the viz smart dashboard as the "Regions" panel when a --geojson is supplied. Zero new dependencies (#302).
viz choropleth & viz smart choropleths now have richer hover tooltips. Instead of a bare feature id and value, each region shows a human-readable name + id (e.g. Kagoshima (JP46)), the value labeled with its measure (count: 65), the share of total (15.6% of total, for count/sum aggregations only), and the rank (rank 1 of 47). Region names are read from the GeoJSON via the new --feature-name-key flag, or auto-detected from common name properties (properties.name, etc.) when omitted. Applies to all paths (point-in-polygon, literal --locations, geocoded) and both the geo and MapLibre (--map) basemaps (#302).
viz smart now draws summary choropleth(s) from a region-code column — no --lat/--lon needed. When a --geojson is supplied and a column is tagged as a geographic region code (e.g. concept: geo.zip_code, from a describegpt/curated Data Dictionary), viz smart aggregates the rows by that code, auto-matches the codes to the GeoJSON feature ids (zero-padding short zips to 5 digits), and leads the dashboard with a count-per-region choropleth plus, when the dataset has a measure column, a median-of-measure-per-region choropleth (median resists the heavy right-skew of things like prices). The region column's frequency bar is suppressed in favor of the map. Tile vs. offline-projection basemap is chosen from the matched regions' geographic span. Zero new dependencies (#302).
viz smart now adds a judicious animated panel — at most one per dashboard, and only when the motion is genuinely informative. In priority order it picks: a Gapminder entity-bubble chart (one bubble per low-cardinality category, tracing a path through two measures over time, sized by a third least-redundant measure) when an entity column has dense enough per-time-bucket coverage; an animated geo map (dated points accumulating on a ScatterGeo projection) for continental/global extents; or an animated scatter pair whose per-time-bucket centroid path bends the most (a trailing-window reveal) — deliberately NOT the strongest-correlated pair, since a near-perfect line (e.g. open≈close, packages∝weight) has no 2-D shape to evolve and animating it adds nothing. --slider auto (default) fires only on a strong signal (a date column + enough time buckets); on lowers the bar; off suppresses. Reuses the native viz geo --slider animation core (#4212).
viz smart: HTML dashboards gain an interactive data viewer drawer — the data behind the charts, in the same file. Next to the rowcount in the metadata frontmatter, an "(Explore)" link (all rows embedded) or "(Preview)" link (first N rows) opens the underlying table in a bottom drawer, mirroring the Data Dictionary drawer. New --preview-threshold <n> (default 50,000; 0 disables) decides which: at or under the threshold every row embeds. Built on vendored DataTables running vanilla — no jQuery — with CSV export, correct date sorting, and per-column search widgets via DataTables ColumnControl (#4283, #4284, #4288, #4302, #4320).
viz smart: data viewer rows and map points are now cross-linked. Clicking a row highlights its point on the map and clicking a point reveals its row — each scrolling the counterpart into view — and rows with no plottable coordinates are pinned rather than silently unreachable. Region-click on a summary choropleth filters the data viewer to that region (#4306, #4324, #4338).
viz smart: dashboards are now localized. The UI renders in the dataset's language, resolved once with explicit intent winning over inference: --language <lang> > the dictionary's x-qsv.detected_language_code > English. --language accepts a BCP-47/ISO 639-1 tag, an ISO 639-3 code (what describegpt's detection pass emits), or an English language name. Coverage extends past the UI chrome to the chart strings and coverage notes, across seven languages (#4310, #4313, #4317).
describegpt: auto-detects the dataset's content language when compiling a data dictionary — locally via whatlang, deterministic and costing zero tokens (no LLM round trip). Emitted as detected_language/detected_language_code when confidence clears the threshold (default 80%, tunable via --language <float>), and the detected language then drives the LLM's output language in every inference phase, so a Spanish dataset gets a Spanish dictionary without being asked (#4301, #4305).
viz smart: new --photos renders a column of image URLs as hover-dwell map imagery — resting the pointer on a map point for two seconds opens that row's photo in a preview card beside the marker. Detection is stats-cache-only (each column's min/max are sniffed for an http(s) URL ending in a known image extension), so it works on a bare qsv viz smart data.csv with no Data Dictionary (#4271).
viz smart: skew-aware encoding feeds the stats cache's distribution-shape signals (skew, zero-share, Gini) into --smarter's axis and chart choices. Additive-amount measures with a high Gini get a Lorenz curve alongside the box, labeled with the cached gini_coefficient so curve and number always agree; heavily-skewed measures get log-scaled boxes with an explicit log-scale cue. Candidate detection blends the dictionary's additive-vs-intensive aggregation with the Gini gate, adding no extra data pass (#4219, #4224).
viz smart: a dictionary-declared pipeline funnel panel. A funding/processing pipeline (planned → committed → spent) was previously visible only as % zeros annotations scattered across three unrelated box titles; the funnel states it once, as the actionable signal it is. Detection requires both a vocabulary match and row-wise containment, and bridges declared pipelines whose stages do not strictly nest (#4222, #4281, #4282).
viz scatter: the animated Gapminder-style bubble chart is now directly requestable, not just auto-selected by viz smart. viz scatter --slider previously hard-errored on --color/--size; it now accepts them together with --series to animate bubbles through time (#4217, #4319).
viz: new QSV_VIZ_CDN environment variable loads the plotly.js runtime from its CDN (<script src>, pinned to the same version qsv would otherwise embed) instead of embedding it. This shrinks a chart to a few KB and a smart dashboard by ~1.9 MB, at the cost of requiring network access when the page is viewed — intended for charts published to the web. It governs only the plotly.js bundle; figure payloads still follow QSV_VIZ_NO_COMPRESS. Default output remains self-contained and offline-capable (#302).
viz smart --dict-info: the embedded Data Dictionary page now offers a download row — the dictionary as JSON Schema (as before), the frequency counts the dashboard actually charted, and every generated sidecar the run consumed: the stats cache (.stats.csv.data.jsonl), its parsing-options metadata (.stats.csv.json), the frequency cache (.freq.csv.data.jsonl) and the --bivariate stats CSV. Every file is bundled into the HTML as a base64 data: URI on a plain <a download> — send someone the dashboard and they can download the underlying files with no access to your machine, and the page stays a single self-contained, script-free file. A sidecar is offered only when it was genuinely USED, never merely present: the frequency cache appears only when viz actually read it (it silently falls back to a full recompute when the cache is stale or option-incompatible), the bivariate CSV only when this run freshly wrote it, and the stats pair is located the way get_stats_records locates it, dc: handles resolved. The human-readable <stem>.stats.csv is deliberately NOT offered: viz never reads it, so nothing can show it describes the same computation the dashboard used. The charted-frequency CSV (<stem>.viz-frequency.csv) is generated in memory — nothing is written to disk — and is always available, since it is exactly what the panels show (capped at --limit, aggregate (NULL)/Other (N) buckets flagged) whether the bars came from a cache hit or a recompute. Absolute local paths (canonical_input_path, canonical_stats_path, arg_input) are stripped to bare file names before embedding, so sharing a dashboard never discloses your directory layout. Sidecars over 4 MB are skipped with a note (#302).
viz: a --dict-info dashboard now bundles every sidecar the run consumed as an in-page download, not just the dictionary: the JSON Schema export, the frequency counts the panels actually charted (<stem>.viz-frequency.csv, generated in memory), and the stats/freq/bivariate sidecars. The HTML was self-contained; the data behind it wasn't — send someone the dashboard and they could read the charts but not get at the numbers (#4256).
viz smart: the correlation drill-down panels are now legibility- and skew-aware for heavy-tailed, zero-inflated numeric columns — the case where money-like measures (e.g. 45–60% zeros, Gini > 0.9) previously collapsed the 2D density panel into a single dark cell at the origin and the 3D scatter into an unreadable blob. The density contour is dropped when its mass concentrates into one bin, first retried as a log-scaled contour over the strictly-positive rows (with the omitted-zeros share stated in the title) when enough points survive; the strongest-pair scatter now scales each axis logarithmically when that column spans orders of magnitude and holds no zeros (the axis is named in the panel title); and the 3D scatter is skipped when its points collapse onto the origin in any 2D projection. Extends the skew-aware axis work from #4219 to the relationship panels; --log-scale on|off still overrides (#4223).
viz smart: now uses robust statistics on measures whose mean is set by their tail rather than a typical row. Such a column (mean at least 2x the median, or a zero-inflated non-negative column whose median is 0) is summed rather than averaged in the "measure by dimension" bar — "top 10 by mean" otherwise ranks whoever holds the single biggest item instead of the biggest total — unless its name marks it intensive (a rate, ratio, index, per-capita figure), which is never summed. When a majority of a correlation matrix's columns are tail-dominated, the matrix is computed with Spearman's rank rho instead of Pearson's r, whose covariance those same extreme rows would otherwise dominate. Both heatmaps now NAME their coefficient in the panel title (Correlation (Pearson r) / Correlation (Spearman ρ — rank, robust to outliers)) instead of a bare "Correlation", and the grouped bar states η² as a plain explained-variance share ("explains 17% of variance") rather than a bare coefficient. On the reporting dataset this flips the headline panel from (mean, η²=0.17) to (sum, explains 17% of variance) and lifts the two near-duplicate spend columns from an outlier-driven r=0.80 to their true rank agreement of ρ=0.94 (#4220).
viz smart: now detects 1:1 categorical columns from the data and charts only one of them. A code/label pair like magencyacro ↔ magencyname (DPR ↔ Department of Parks & Recreation) otherwise produces byte-identical frequency bars and spends two parcats axes on the same variable, with trivial straight-through ribbons between them. The shorter-valued member is kept — it fits a bar label or treemap tile where the long form truncates — and the drop is reported so the mapping stays discoverable. This is the data-driven counterpart of the existing name-based subject/subject_code rule, which needs both a <base>_code spelling and a data dictionary and so could not see this pair; both feed the same suppression set, so the per-column panels, the grouped bar, parcats and the hierarchy panel all inherit the verdict. Detected by partition refinement — two columns are 1:1 exactly when they induce the same grouping of rows, so candidate columns are split apart on the first row their groupings disagree, which also means detection stops reading as soon as no candidates remain. Strict 1:1 only, no "near-1:1" tolerance, and guarded on support: at least 4 populated rows per category, so a small-sample coincidence cannot delete a panel. An empty cell is its own category for this purpose, so columns blank on different rows are never treated as the same variable (#4221).
viz smart: the "measure by dimension" bar now picks its dimension deterministically. η² is accumulated over a HashMap's values, so its float rounding depended on iteration order and two mathematically tied dimensions (1:1 columns explain exactly the same variance) swapped places between runs of the same binary on the same file — the panel title flipped from by magencyname to by magencyacro and back. Ties now resolve to the lowest column index (#4221).
viz smart: the correlated-pair scatter now judges legibility on the axes it will actually render. It previously tested for a degenerate cloud on the raw values before resolving its log axes, so a strictly-positive heavy-tailed pair — squashed against the origin on linear axes but spread cleanly once logged — was dropped in exactly the case log scaling exists to rescue, and --log-scale on could not override the drop. The pair is now tested in the space it will be drawn in, mirroring the density contour's linear → log retry from #4223. --log-scale off still drops such a cloud: with log declined the linear axes are what will be drawn, and absent beats unreadable (#4276).

Changed

dependencies: bumped minijinja & minijinja-contrib from 2.21 to 2.22, which changes rendered template output in template, apply, fetchpost, describegpt and profile. Upstream #913 aligned scalar rendering with Jinja2: booleans now render as True/False (was true/false) and none renders as None (was an empty string). This reaches any template that interpolates a boolean directly — including qsv's own to_bool and regex_match filters, e.g. {{ active|to_bool }} — and any none-valued expression, most notably a JSON null reached through the fromjson filter ({{ (meta|fromjson).author }} now yields None where it previously yielded nothing). Templates that only test booleans ({% if active|to_bool %}) are unaffected. To restore the old rendering, pipe through a fallback: {{ value or "" }} (or {{ value|default("", true) }}) for none — note that a bare {{ value|default("") }} does not help, as default only substitutes for undefined, not none — and {% if b %}true{% else %}false{% endif %} for booleans. Separately, upstream #909 fixed the split filter to return a sequence instead of a lazy unsized iterator, so loop.last (and negative indexing/slicing) now work over split results — a {% if not loop.last %}, {% endif %} separator inside {% for x in col|split(',') %} no longer emits a spurious trailing separator.
describegpt: the bundled prompt file (resources/describegpt_defaults.toml) is bumped 8.0.0 → 9.0.0. dictionary_prompt now asks for the grain sentence in the --language target language and for a new top-level grain_unit string (the bare entity noun), emitted as x-qsv.grain_unit in the JSON Schema dictionary. If you pin a custom --prompt-file, it will not request grain_unit; the field is then simply absent and viz smart falls back to its previous grain-parsing behavior — nothing breaks, but localized chart titles keep the old English-leak issue until the prompt is updated (#4321).
dependencies: migrated the cached crate from 2.0.2 to 3.0.0-rc.7, which swaps the on-disk cache backend from sled to redb (DiskCache* → RedbCache*). This affects the persistent caches used by fetch/fetchpost (--disk-cache & Redis), geocode (the OpenCage result cache), apply summarize, and describegpt. One-time impact: existing on-disk caches are invalidated — redb cannot read sled's format, so entries are recomputed on first run after upgrade; Redis caches self-heal. Beyond mechanical renames: v3's #[concurrent_cached] auto-generates a nested {fn}_no_cache companion that collided with a hand-written geocode helper (renamed to search_index_uncached to fix an infinite recursion); RedbCache::cache_size() is Ok(None) by design, so geocode cache-info reports the entry count as "unknown" rather than a misleading 0; and the recomputable external-API caches (fetch/fetchpost disk cache & the geocode OpenCage lookup cache) keep sled's non-durable behavior via .durable(false) to avoid fsync-per-write on the per-row hot paths. A cache TTL of 0 (QSV_DISKCACHE_TTL_SECS, QSV_REDIS_TTL_SECS, geocode --cache-ttl, apply-summarize) now means "cache indefinitely" (time-based expiration disabled) — v3 rejects a zero TTL at build, and under sled a 0 TTL had instead meant every entry was immediately stale. Use --no-cache (geocode/apply) or omit the enable flags to disable caching entirely; sled stays in the lockfile as a separate direct dependency (#4194).
viz: single-chart HTML no longer embeds the ~2.1 MB MathJax (tex-svg) bundle that plotly ships alongside plotly.js — roughly halving a standalone chart (e.g. 4.86 MB → 1.97 MB). viz renders only plain-text titles and labels, and viz smart dashboards have always dropped it. Consequently, LaTeX in a --title (e.g. $\alpha$ ) is no longer typeset and renders as literal text, matching viz smart's long-standing behavior (#302).
viz choropleth & viz smart point-in-polygon binning: the default snap-distance cap is now context-sensitive and stats-aware instead of a fixed 10 km. When --snap-max-dist is omitted, the cap is auto-derived from the GeoJSON's median region size (10% of the median bbox diagonal, so ward-scale maps get sub-km caps and country-scale maps get larger ones) and the lat/lon coordinate precision (few-decimal coordinates raise the cap to cover their quantization error — from the stats cache's max_precision in viz smart, or counted from the raw values in viz choropleth), clamped to 0.1–100 km; the fixed 10 km remains only as the no-signal fallback. Whenever any points snap, the coverage note beneath the map now always reports the snap count, the cap applied, and how it was derived (the smart "Regions" panel title carries the count and cap; the derivation detail stays on the stderr note). Pass --snap-max-dist 10 to restore the previous behavior exactly (#302).
viz smart: the leading overview panels (map/geo, correlation heatmap and its scatter/contour/3D drill-downs, and the time-series trend) now each span the full dashboard width on their own row, instead of being squeezed into a half-width grid cell. The per-column box/bar/histogram panels still flow in the --grid-cols-wide grid below. Applies to all render paths (typed subplot grid, raw-JSON static export, and the inline-div HTML grid) (#302).
viz smart: static image export (PNG/SVG/PDF/…) now renders the geographic panel instead of dropping it. Since the Mapbox tile basemap can't be statically exported (it needs network tiles), the map is drawn as an offline ScatterGeo projection fit to the data's extent — with an albers-usa projection auto-selected when the coordinates span the US. The Mapbox tile map and 3D scene panels remain HTML-only (#302).
viz smart: dashboards now auto-fit the data for both HTML and static image export. --max-charts defaults to 0 (auto), drawing every eligible column (up to 64). Up to 8 cartesian panels render as a single typed subplot grid (plotly's typed subplot-axis limit); beyond 8, HTML switches to an inline-div grid of independent plots while static image export (PNG/SVG/PDF/…) assembles the grid as raw Plotly JSON with domain-positioned xaxis9+/yaxis9+ axes — so it's no longer capped at 8 panels. Rendered via the static exporter's JSON path (no new dependency; plotly re-exports plotly_static). Subplot gaps now scale with the grid size so tall dashboards (e.g. a 42-panel, 21-row export) lay out correctly instead of collapsing to negative cell heights. Set a positive --max-charts N to cap the count (#302).
viz geo maps are now theme-aware. The geo-subplot charts (choropleth, geo, and the viz smart map/region panels) previously always rendered light-gray land on a (possibly dark) page; under a dark --theme (e.g. plotly_dark) they now use a dark land fill and a dark subplot background (geo.bgcolor, newly exposed on the plotly fork's LayoutGeo) so the whole map — sea and the area outside the projection included — matches the page. The viz smart light/dark toggle also recolors the geo land/sea (not just the page) when switching modes live (#302).
viz smart: the footer qsv logo now stays legible under any --theme and after the in-browser light/dark toggle. The logo gains a theme-aware outline halo — a light halo under the dark theme so its navy shield reads against near-black paper (e.g. plotly_dark), and a faint dark halo on light themes — keyed on the same body.qsv-dark class the toggle drives (#302).
viz: viz smart --dict-info drawer and panel polish — Examples are annotated with their counts (Dog Lifetime Spayed Female (16,350), …), an Other (1) bucket is charted under the value's real name instead of an opaque label (the (N) counts distinct values rolled up, so 1 means nothing was aggregated), bar hovers carry the full category name rather than plotly's truncated ticktext echo, and a polar panel's title is no longer clipped by the panel edge. describegpt's jsonschema output gains untruncated examples_full feeding examples/x-qsv.example_counts, while --truncate-str keeps bounding the LLM prompt (#4262).
viz: the visualizations gallery gains a Boston 311 (2025) visual data dictionary (267,187 rows, 28 auto-chosen panels, a 25-neighborhood choropleth), and NYC 311 moves from an embedded iframe to a clickable screenshot link-out beside it — dropping the page's heaviest iframe (3.6 MB) while keeping the full dashboard one click away. All three link-out previews are now WebP (2.0 MB PNG → 0.37 MB). examples/viz/README.md corrected alongside: three datasets and five committed dictionaries that the gallery charts were undocumented, the NYC example still called for an LLM it no longer needs, and icicle/parcats/splom had no example though the intro claims every chart type is covered. The Pittsburgh link-out's copy-pasteable command now reproduces its artifact too — it named an input that does not exist (pitt311data.csv; the page is built from pittsburgh_311.tsv) and an output that is not the linked file (#4263, #4264, #4269).
prebuilt binaries now ship viz — and viz/viz_static no longer build on big-endian targets, by design. No release had ever shipped viz: it landed four days after 21.1.0 was tagged, and only publish.yml's three x86_64 entries were ever updated to enable it. The portable, qsvpy and both macOS publish workflows still built without it — so the "p for portable" binaries the README recommends as the SIGILL workaround silently dropped viz (and, on Windows, magika and color too). viz is now enabled across the little-endian publish targets. Separately, viz and viz_static do not work on big-endian targets (s390x, ppc64 BE); that limitation was previously documented only on viz_static and phrased as a browser/headless concern rather than an endianness one, so cargo build -F distrib_features on s390x compiled clean and produced a silently broken binary. A compile_error! on the viz feature (which viz_static enables, so both tiers are covered) now fails the build loudly instead — if you build for a big-endian target, drop viz/viz_static from your feature set. Note powerpc64LE is little-endian, so the guard deliberately does not fire there; viz is simply not enabled for the ppc64le prebuilt, which has no viz CI coverage. CI coverage was widened to match: Windows and macOS now compile the viz_static tier (as Linux already did), musl gains a base-viz step, and aarch64-unknown-linux-gnu — the only published target with no test workflow at all — gets one (#4350, #4351).
licensing & attribution: qsv vendors and redistributes several open source components but shipped no NOTICE/THIRD_PARTY/CREDITS file and had no license-checking automation. MIT, BSD-3-Clause and CC-BY all require the copyright notice and license text to travel with copies — in several places they weren't. Those notices are now attributed and shipped (#4297, #4161).
json: the flatten-to-CSV step is now inlined, dropping the unmaintained json-objects-to-csv (last published 2022) and its transitive flatten-json-object. qsv needed a preserve_key_order method upstream lacks, so Cargo.toml pinned a git-branch patch to a personal fork — and because that's a git branch rather than a crates.io release, cargo install qsv was fragile, which is exactly what #3523 reported. Both dependencies are gone (#3523, #4014).
self-update: background update checks are now throttled to roughly twice a month (~15-day interval) via a persisted last-check time, replacing a stateless 50% coin-flip that remembered nothing. Both background call sites route through the guard — the no-command screen (previously the coin-flip) and the help command (previously an unconditional check on every render) (#4246).

Fixed

frequency: duplicate-named columns are now classified by position, not name. Per-column cardinality and all-unique detection were keyed by header name, so every column sharing a name inherited the FIRST duplicate's statistics — all marked identically <ALL_UNIQUE>, <HIGH_CARDINALITY> or normal, regardless of their real data. A user-facing correctness bug in normal output, not merely a cache artifact (#4041).
viz: the dictionary's per-column date format is now honored for DMY/MDY. A --dictionary declares each column's own strftime format via x-qsv.content_type (e.g. date:%m/%d/%Y), but route_from_content_type split the base token off to pick a panel route and threw the suffix away; every date parse then fell back to the one process-wide QSV_PREFER_DMY. When the two disagreed qsv silently misread every date in the column — chart axis values and data-viewer sort keys — with no warning, and both readings look plausible (#4303, #4318).
viz: selecting a record in the data viewer could freeze or kill the browser tab on a large map dashboard. applySelection painted every selection with Plotly.restyle(gd, {selectedpoints: sel}, …), and plotly declares selectedpoints as editType: "calc" — so each click recalculated the entire trace (points and hover strings) and rebuilt MapLibre's GeoJSON, leaking ~44 MB per click on a 265k-point map. Above a 50k-point threshold the row pin now carries the selection and the dimming restyle is skipped (#4341).
viz: the data viewer's table collapsed instead of scrolling in Safari. Under scrollX, DataTables leaves the table at width: 100% and holds columns open by writing min-width onto each <col> — but CSS 2.1 §17.6.1 limits the properties that apply to column elements to border, background, width and visibility; min-width is not among them. Blink honors it as an extension, WebKit does not (a minimal probe renders 1206px in Chrome and 200px in Safari) (#4325).
get: --timeout is now an inactivity timeout rather than a total-request one. reqwest's .timeout() aborted a slow-but-steady download mid-stream (e.g. a large CSV trickling in over a slow CKAN endpoint or proxy), surfacing as a cryptic "error decoding response body". The shared async client now uses read_timeout(), which resets on every received chunk, so only a genuinely stalled connection trips it; connect_timeout is retained (#4123).
get/dc:: a dc: handle is now resolved exactly once per run instead of repeatedly (#4257, #4273).
self-update: added timeouts, retries and structured error messages (#4182).
describegpt: a single stray backslash in an LLM response no longer silently discards the whole dictionary. Models occasionally emit a backslash in structural position ("role":\ "measure", — observed from google/gemma-4-26b-a4b), which is invalid JSON. try_fix_json had no in_string guard, so it passed the backslash through, the whole object failed to parse, and extract_json_from_output fell through to its brace scan — which returned the first nested object that happened to parse (one field's {label, description, …} body) as if it were the entire dictionary. Every real column lookup then missed, so a single-pass run produced a dictionary with no LLM labels at all, a --two-pass run silently discarded its baseline pass, and grain/relationships vanished — all with no error anywhere, after paying for the inference. A bare backslash outside a string is never valid JSON, so it is now dropped (escapes inside strings are preserved); and parse_llm_dictionary_response now warns when a response parses but matches not one of the real column names, so this class of failure can never be silent again (#4321).
viz/describegpt: the dataset grain phrase no longer leaks English into a localized dashboard. describegpt --language xx emitted grain in English regardless (the --language directive in dictionary_prompt enumerated its scope as Labels/Descriptions/Content Types and never reached grain), and viz smart then parsed an entity noun out of that English sentence — splitting on the English literal " one " — and interpolated it into an otherwise fully-translated title template. A --language pt dashboard rendered product collection record per location ao longo de Data de Coleta, plus the same English noun in the axis title and the raw English sentence in the dictionary drawer's Granularidade: line. describegpt now writes grain in the target language AND emits a separate structured x-qsv.grain_unit — the bare entity noun — which viz uses verbatim, so no natural-language parsing is involved and languages with neither articles nor spaces (ja, zh-CN) work too. Dictionaries without grain_unit keep today's behavior exactly: viz falls back to the legacy grain parse, then to the localized viz.chart.records (#4321).
viz: the Data Dictionary drawer's Attribution block is now localized. describegpt's attribution footer (Generated by … describegpt / Command line: / Prompt file: / Model: / LLM API URL: / Language: / Timestamp: / the LLM WARNING:) is authored when the dictionary is generated and stored verbatim in the schema as x-qsv.generated_by, which viz rendered byte-for-byte — so it stayed English at the foot of an otherwise fully translated drawer. The dictionary is written once while viz --language is chosen later and may differ, so it is translated at render time instead, on the same contract as the "Notable Characteristics" heading: the match is exact and line-oriented, so a hand-written dictionary, an older describegpt layout, or an uncurated --language passes through unchanged. The stored schema is untouched — it stays a stable machine-readable record, and the drawer's "Export JSONSchema" button still serves the original bytes. Labels only: the command line, prompt-file kind, model, API URL, language code and RFC3339 timestamp all ride through verbatim, so the footer remains a faithful, copy-pasteable record of how the dictionary was built (which is why a dictionary built with describegpt --language Portuguese renders Idioma: Portuguese — that string is a value, not a label). All five of describegpt's WARNING: variants are covered, not just the two a self-inferred dictionary can produce, since --dictionary <file> accepts any describegpt output (#4345).
test(viz): the browser-gated viz_static tests now actually verify what they claim. viz_static_more_than_eight_panels asserted that panels cat09–cat12 appear in the exported image, but its fixture built all 12 columns from one shared modulus (v{(i + c) % 4}), making every pair a 1:1 bijection — so the twin-collapse from #4221 folded them into a single panel and the assertion could never hold. Its sibling viz_static_max_charts_caps_panels passed vacuously for the same reason: with one panel eligible, --max-charts 4 was never exercised and the test would have kept passing had the flag been deleted outright. The test was valid when written and was silently invalidated a month later — #4221's own fixture-migration pass updated six sibling tests but missed these two, because the whole family is doubly dead: #[ignore]d with no workflow passing --ignored, and viz_static absent from all 17 cargo test invocations, so not even cargo check type-checked them. Both fixtures now give each column its own modulus (12 distinct cardinalities), the vacuous !stderr.contains("limited to") assertion is replaced by one that pins the fixture's premise, and the --max-charts assertions are inverted to match actual behavior — the trim keeps the highest-interest panels, so cat12 survives and cat01 is dropped, the opposite of what was asserted. The Linux workflow now builds with viz_static so these tests are at least compiled, a new rust-viz-static.yml runs them against a matched Chrome/chromedriver pair, and a new non-ignored test pins the fixture premise on every PR without needing a browser (#4343).
viz: viz smart no longer overflows Windows' 1 MB main-thread stack in debug builds (STATUS_STACK_OVERFLOW, which failed all 266 test_viz tests when viz was enabled in Windows CI; release builds — including the shipped Windows binary — were never affected). At opt-level=0 rustc gives every local its own stack slot with no reuse across branches, and smart_inline_panel_plot built 14 Plots and ~42 multi-KB plotly Layout temporaries in one 1,100-line frame, with the 18-arm panel_trace nested inside it — needing 3–4 MB of stack. Both are now split into one #[inline(never)] function per panel kind behind a thin dispatcher (so only one small frame is ever live), and assign_typed_axis takes &mut Layout instead of moving the Layout through a per-arm builder chain. A debug viz smart now runs in under 512 KB (heaviest gallery invocation: under 768 KB). The Linux CI workflow gains a 1 MB-rlimit smoke test so a stack regression fails fast on every PR, and the viz feature is re-enabled in the Windows and macOS test workflows (#4328).
viz: 42 findings from a whole-file review of src/cmd/viz.rs. The headline correctness fixes: the viz smart KPI row silently disappeared on most real-world datasets (tiles were rejoined to their stats row by display title, which classification had already decorated — e.g. amount (right-skewed) — so the lookup missed and the whole row was dropped); the AnimatedGeo panel blew past its point cap by ~nb/2x, because cumulative frames re-serialize each point once per subsequent frame, turning a nominal 150k-point budget into tens of MB of HTML; build_heatmap_pivot allocated an unbounded dense y_cats x x_cats matrix (two 50k-cardinality key columns is ~20 GB) and is now capped like every other multi-category builder; and issue #4219's log-axis clamp did not cover the raw-value panels, so a single 0 still broke the whole box/violin trace on a log axis. Also: CSV headers and cell values reaching plotly's pseudo-HTML renderer are now escaped exactly once per sink (identities — animation frame keys and the dictionary anchor — deliberately left raw, since escaping them would break slider/frame matching and click-to-scroll); a --geojson spec is loaded once per run instead of 2-3 times (a URL spec was multiple HTTP GETs, and a non-deterministic endpoint produced a genuine TOCTOU where points were binned against one fetch and drawn from another), and remote fetches now have a timeout and a bounded read; is_intensive_measure matches whole tokens rather than substrings (ratio is inside duration/generation/operation, which silently flipped additive amounts from Sum to Mean); time.duration columns are charted as measures instead of being dropped as temporal; the map's fitted zoom is centered in Mercator space so high-latitude extents actually fit; and a 100%-outside point/boundary mismatch is now reported instead of silently producing an empty map. Plus pip_assign spatial pruning, a --slider frame cap, and several per-row allocation hot paths. The plotly.js CDN tag now carries an SRI hash verified against the embedded bundle, and gallery regeneration is byte-stable (#4247).
stats cache: a cache built with different parsing options is no longer reused. The cache is located by input path and validated by mtime, but was not keyed by --no-headers/--delimiter — so a cache describing a different logical CSV passed every freshness check. Reachable through viz smart, the one consumer that force-regenerates under non-default parsing: that force rewrote the shared sidecar, so the next default-parsing run read a headerless cache — field names became positional (0, 1, …), the header row was counted as data, and panels keyed off column names (the KPI row) vanished. With a wrong --delimiter it was worse than wrong output: the cache held one fused column and the next plain run failed outright on a valid file until the sidecar was deleted by hand. get_stats_records_readonly already validated this; its sibling get_stats_records — used by viz, schema, frequency, joinp, pivotp, sample, pragmastat, sortcheck, extsort and profile — wrote both fields and checked neither, so the check is now shared between them. Also keeps the canonical metadata sidecar in sync with the JSONL it describes, which had desynced for symlinked inputs (#4251).
pragmastat: fixed two intermittently flaky tests. pragmastat_onesample_basic/pragmastat_twosample_basic asserted that the randomized spread/disparity bounds contain the sample point estimate — not an invariant the algorithm provides. The bounds are a distribution-free interval for the TRUE parameter (order statistics of a random disjoint pairing), while spread is the Shamos estimator over all C(n,2) differences: two different estimators, so the interval need not straddle the point. Compounded by a time-seeded RNG that --seed does not reach. Replaced with invariants guaranteed by construction (lower <= upper, non-negative spread, finite), verified over 5,600 sampled runs. Test-only; no production behavior changed (#4250).
viz: plotly trace names are now escaped — 19 sites the #4247/#4254 sweeps left behind. plotly renders its markup subset in the legend and hover box, so a trace name resolved from a raw CSV header (panel.name, the histogram/box/violin y_label) reached that renderer as live markup. Two carve-outs are deliberate and were verified rather than assumed: the dictionary anchor (Annotation::name, matched by a qsvdict- prefix in the embedded JS) and the animation frame keys stay raw as identities, and the --series bubble-animation legend was already escaped at read time — re-escaping it would have shipped &amp; to the reader. Same spoofing class as #4247/#4254; script execution stays blocked by plotly’s tag whitelist. Note this also fixes a case with no untrusted input at all: a correlation panel’s own generated suffix (<1% of rows omitted) was escaped in the panel title but raw in the trace name, so the two disagreed (#4331).
viz: slider step labels and the current-value readout are now escaped — the animation surface #4331 deferred. Verified in-browser rather than assumed: plotly's drawLabel and drawCurrentValue both pipe their text through svgTextUtils.convertToTspans, and the data-notex attr on those nodes only suppresses MathJax, not the HTML subset. An <a href> in a slider frame cell drew two live anchors — one in the step label, one in the current-value readout — so this was clickable link injection, not just the bold-text spoofing the issue estimated. Only the two RENDERED strings are escaped: the step label, and the currentvalue.prefix built from a raw CSV header (the site the issue's own inventory missed). The step value, the animate target and Frame::name stay byte-identical raw, preserving the #4247 identity decision — and .value() must keep being set EXPLICITLY, since sliders/defaults.js does coerce('value', label) and would otherwise inherit the escaped label and desynchronize the slider. javascript: URLs stay blocked by plotly's sanitizeHref protocol whitelist, so this is spoofing/phishing, not XSS (#4333).
viz: axis and colorbar titles are now escaped on the standalone chart paths — 11 sites the #4247 escaping sweep left behind, each resolving its label from a raw CSV header. A header like <a href="https://evil.example">Amount</a> drew a live clickable link as the axis or colorbar title. Same spoofing class as the panel-title fix in #4247; script execution stays blocked by plotly's tag whitelist, so this is spoofing/phishing, not XSS (#4254).
viz: map cluster bubbles were unreadable and inert. plotly builds the cluster count layer with an empty paint (so MapLibre's default black text-color applies, with no trace attribute to override it) and defaults cluster.color to the trace's marker color — an identity color doing a magnitude job. Measured contrast of the count on those defaults was 2.55:1 white-on-orange and 4.08:1 on plotly blue, both under the 4.5:1 floor for 12px text. Bubbles are now a single-hue sequential ramp keyed to point_count, per-basemap (the dark-basemap ramp is paler — a step dark enough to vanish into a dark map fails the mark-vs-surface floor), with the label ink picked per step by measured contrast (worst case 6.59:1 light / 7.86:1 dark) and a surface ring so a bubble reads over a busy patch of basemap. Counts also stopped disappearing: MapLibre drops symbols that collide with basemap labels and plotly set no overlap policy, so the number — the entire point of the bubble — silently vanished at some zooms. Clusters additionally gained the hover plotly never gave them (it filters clustered features out of the hover-bearing layer): a tooltip with the point count, and click-to-expand to the zoom where the cluster breaks apart (#4266, #4268).

Full Changelog: 21.1.0...22.0.0