dathere/qsv 27.0.0

on GitHub

[27.0.0] - 2026-10-27 📡 The "Live Data Schematics" Release 📊

Highlights

~109 commits since 26.0.0, and the banner feature is Live Data Schematics.

Until now, a Data Schematic was a snapshot: a self-contained HTML page describing the data as it was when viz smart rendered it. A question the page didn't anticipate sent you back to the terminal, and new data meant rendering it again.

In 27.0.0, a schematic can stay connected to its data. Open it from disk with qsv-clientd running, approve a one-time pairing code in your shell, and the page itself can drill into the rows behind a frequency bar, refresh its stats, fetch the latest copy of the data, re-render itself and run read-only queries, with results of any size paged from the daemon. Results of up to 10,000 rows get the data viewer's Advanced Filter and an Export CSV button. qsv-clientd is a small, optional per-user daemon from qsv-client; without it, schematics work exactly as before. The bridge stays inert unless the page was opened from disk or is served from loopback, so a schematic published on the web never contacts its viewers' machines.

Public CKAN data is one command away: with qsv-client 0.6.0 or later, point qsv-clientd viz at any public CKAN resource or dataset page (or a ckan:// id) to get a live schematic of it. qsv-clientd remembers where the data came from, so Fetch latest gets the current version.

Also in this release:

• Polars 2.1.0, streamed. sqlp --streaming and joinp --streaming now write results batch by batch, Polars' query optimizations apply again, and pivotp handles wide pivots with a fraction of the memory. sqlp gains recursive CTEs, and has one breaking change: a FROM subquery alias can no longer be read as a table.
• A correctness sweep across validate, stats, frequency, schema, viz and writestat. validate now reads piped stdin, follows local $refs, honors every type in a type array and fixes dynamicEnum on numeric and nullable columns; stats fixes weighted quantiles and modes; frequency stops collapsing columns to <ALL_UNIQUE> when it shouldn't; writestat no longer writes wrong data to SPSS portable files, or numbers too large for SAS transport files as missing values.
• Inputs read as written. Compressed CSVs are decompressed byte-for-byte (leading zeros and 1.50 survive), and QSV_SNIFF_PREAMBLE now actually skips the title lines it finds.
• get ships in the prebuilt qsv binaries, so dc: inputs work out of the box.
• Private caches on shared machines. The get cache and the dc: copies other commands read are now readable by you only, and so is every file stats writes beside a private CSV. Before, other local users could read data qsv get had fetched, including private CKAN resources fetched with an API token.

Read Breaking Changes before upgrading: sqlp subquery aliases, three validate schema behaviors, and QSV_SNIFF_PREAMBLE. --low-memory is gone from sqlp, joinp and count. The two security fixes at the top of Fixed also change behavior: a get cache shared with a group on purpose becomes owner-only, and dc: inputs stop with an error if another user has taken their temp folder.

Breaking Changes

• sqlp: the alias of a subquery in FROM can no longer be read as a table (#4907, Polars #30207). Some queries that worked in qsv 26.0.0 now fail with relation '<alias>' was not found. The alias still qualifies the subquery's columns (a.x). Reading it as a table, whether in another subquery or in a later FROM item, now fails:

  -- now fails: `a` is read as a table in the IN subquery
  SELECT a.x FROM (SELECT * FROM t WHERE g = 1) AS a WHERE a.x IN (SELECT x FROM a)
  -- now fails: `a` is read as a table in a later FROM item
  SELECT * FROM (SELECT g FROM t) AS a, a AS b

  To fix a query, move the subquery into a WITH clause (a CTE), which still works and returns the same rows as before:

  WITH a AS (SELECT * FROM t WHERE g = 1) SELECT a.x FROM a WHERE a.x IN (SELECT x FROM a)
  WITH a AS (SELECT g FROM t) SELECT * FROM a, a AS b

  In a .sql script, CREATE TABLE a AS SELECT ... also keeps a result for later statements. This also fixes queries where the alias hid a loaded table of the same name: in SELECT x FROM (SELECT g AS x FROM u) AS t WHERE x IN (SELECT g FROM t), the inner t now reads the t input file, as in DuckDB and PostgreSQL, instead of failing.

• validate: some schemas give different results (#4911, #4913). Most of the validate fixes below only stop valid data being flagged. Three of them change results that schemas may rely on today:

  • Stricter: numeric text in a column that also allows strings is read as a number. For a column typed ["integer", "string"] (in either order) with "minimum": 0, the cell -3 used to be read as a string, so minimum was skipped and the row passed. It is now read as a number and fails. Text that doesn't parse as a number is still read as a string.
  • Looser: dynamicEnum no longer rejects empty cells. In a column whose type allows null (e.g. ["string", "null"]), an empty cell that dynamicEnum used to reject now passes, because no reference file can list null. To keep rejecting empty cells, remove "null" from the column's type.
  • Stricter: uniqueCombinedWith must name columns the CSV has. A column name missing from the header (a typo, or a column only some files have) used to be skipped silently, so uniqueness was checked on fewer columns. Now the schema fails to compile ('uniqueCombinedWith' column "x" is not in the CSV header) and no rows are validated. The same goes for a column index past the last column, and for an index whose header name a later column repeats. A schema shared across files with different columns needs a uniqueCombinedWith that fits each file.

• QSV_SNIFF_PREAMBLE now skips the preamble rows it finds, so commands can return fewer rows and a different header (#4916). It used to sniff the number of rows above the header and then ignore it (only input --auto-skip used it), so a title line above the header was read as the header. Now every command reads from the header, including count, stats, input, the Polars-based commands (sqlp, joinp, pivotp, to parquet, writestat) and the DuckDB queries of scoresql --duckdb and describegpt. A file with a preamble is read without its index, and qsv index still indexes the file as stored. Stats caches and Polars schemas are kept apart for the two modes, so turning the env var on or off never reuses one built the other way (a schema of the rows past an N-line preamble is <file>.skip<N>.pschema.json). QSV_SNIFF_DELIMITER alone now only sniffs the delimiter. Not covered: stdin and Polars SQL's read_csv() table function. If you set QSV_SNIFF_PREAMBLE, check your results: unset it to keep the old behavior, or use input --skip-lines to control the skip yourself.