This alpha includes security fixes for private stored queries and database downloads, streaming output renderers with CSV and TSV now implemented as renderers, a new JSON format for individual rows, and tables listed as cards on the homepage and database page.
See 0.65.6 for the database download fix backported to the stable 0.65.x branch.
Security fixes
- Fixed a security issue where the
?owner_id=filter on the/-/queriesand/<database>/-/queriespages, and their.jsonequivalents, returned private stored queries belonging to that owner - including their SQL - to any actor, including anonymous users. This affected Datasette 1.0a31 through 1.0a42. - Fixed a security issue where a database registered using
add_database(db, name=..., route=...)with a route that differs from its name could be downloaded from/<route>.dbby actors who were denied access to it. Permission rules for theview-database-downloadaction that were written against the route no longer apply, and should use the database name like rules for every other action. - The example
permission_resources_sql()hook in the upgrade guide for 1.0a20 was missing itsif action == "permissions-debug"check. A plugin that copied that example allowed every action, including write actions, for every actor including anonymous users. If you used that example in your own plugin, add the missing check.
Breaking changes
-
The JSON for an individual row, such as
/database/table/1.json, now returns{"row": {"id": 1, ...}}. It previously returned{"rows": [{"id": 1, ...}]}, a list that always contained exactly one row. Two other keys in that response changed to match:"truncated"is no longer included. That key says whether a list of rows was cut short at the page size limit, so for a single row it was alwaysfalse.?_extra=render_cell, which returns the HTML that plugins rendered for each column, now returns"render_cell": {"column": "<html>"}. It previously returned"render_cell": [{"column": "<html>"}], a list with one item for the one row.
Add
?_shape=objectsto the URL to get the previous format back. The other?_shape=options work as they do for table JSON. See Rows and the upgrade guide. (#2612, #2137) -
Output renderer
renderfunctions must now return a Response object, orNone. Returning a dictionary withbody,content_type,status_codeandheaderskeys, deprecated since Datasette 0.49, is now an error. The"callback"key, the pre-0.49 name for"render", has also been removed. See the upgrade guide. (#954) -
The
view_namepassed to plugin hooks for SQL query pages, both arbitrary SQL and stored queries, is now"query". Template hooks such as extra_template_vars(template, database, table, columns, view_name, request, datasette) were previously passed"database"for those pages. Output renderers were passed"table"byrenderand"database"bycan_render. See the upgrade guide. -
An "include all rows" export that fails part of the way through, or that reaches the max_stream_mb limit, now drops the connection so the download fails. Previously the error message was written as the last line of the file, which otherwise looked complete.
-
The
datasette_allow_facettemplate variable has been removed from the table page. Customtable.htmltemplates that use it should use{{ settings.default_allow_facet|tojson }}instead.
Output renderers
- New
Response.stream(body)method for returning a response a chunk at a time from an async iterator of strings or bytes, for example from a register_routes(datasette) view. See Streaming responses with Response.stream(). - Output renderers can now stream every row of a table or view. A
renderfunction that accepts the newrow_streamargument - an async iterator of rows - and returnsResponse.stream()will be sent every row when the URL includes?_stream=1. See Streaming every row. (#1101) - CSV and TSV export are now implemented as default output renderers using that mechanism, replacing the previous special case. (#1062)
- Output renderers registered with register_output_renderer(datasette) can now list the extras they need, such as
countorhuman_description_en, using a new"extras"key. Datasette calculates those and passes them to therenderfunction in a newpageargument, a dictionary which also includesnextandnext_urlon table pages. (#2770, #2186) - Output renderer
renderandcan_renderfunctions can now acceptpathandfull_patharguments: the path to the HTML version of the page without the.extension, with and without the query string. See register_output_renderer(datasette). (#1179, #2987) - Output renderers whose
renderfunction accepts the undocumenteddataorargsarguments now trigger aDeprecationWarningwhen Datasette starts. Those arguments will be removed before Datasette 1.0. Use thepageargument in place ofdata, andrequest.argsin place ofargs. (#2770) - The
allow_csv_streamandmax_csv_mbsettings have been renamed to default_allow_stream and max_stream_mb, since they now apply to every format that can stream. The previous names continue to work. ?_dl=1now adds acontent-disposition: attachmentheader for every export format, not only CSV and TSV.csvandtsvare now included in the?_extra=renderersextra.
Features
- Tables and views on the homepage and the database page are now listed as cards, each showing the row count, the column names and links to the tables it references. A "+N more" button reveals the rest of a table's columns. The database page now matches the design of the homepage. (#3067)
- JSON objects and arrays on the row page have a new button for switching between the value as stored and a pretty-printed version. The copy button copies whichever is displayed.
- Filtering, faceting, sorting and paginating a table now crossfades the table and facets in place, in browsers that support CSS cross-document view transitions. Suggested facet links no longer jump to a
#facet-...fragment. (#3061) - New
<datasette-copy-button>custom element that plugins and custom templates can use to add a copy to clipboard button. Datasette now uses it for the schema on the table and schema pages, as well as for values on the row page. See Copy to clipboard buttons. (#3063) - New
DatasetteModal.confirm()JavaScript method for asking a question using an in-page dialog, returning a promise. See Confirmation dialogs. - The
beforeCloseguard for a modal dialog can now return a promise. The dialog stays open until that promise resolves. - The list of columns in the Column statistics dialog now shows at most three rows of columns, with a "+N more" button to show the rest, and no longer truncates column names when there is room to show them. (#3064)
- The select boxes used to filter a table now look the same across different browsers. (#3060)
- New
?_extra=expanded_columnsextra for table JSON, listing the foreign key columns that were expanded using?_labels=on. - The total time spent calculating suggested facets for a page is now capped at ten times the facet_suggest_time_limit_ms setting, so tables with a large number of columns can no longer spend several seconds suggesting facets. (#862)
- The default max_post_body_bytes limit has been increased from 2MB to 10MB (
10485760bytes), allowing larger JSON API requests. - The
allow_facetandallow_downloadsettings have been renamed to default_allow_facet and default_allow_download, matching default_allow_sql. The previous names continue to work. (#1410) datasette publishanddatasette packagenow install the same version of Datasette as the one used to run the command, unless--branchis used or an--installoption names Datasette. Previously they installed the latest stable release, which does not accept the--configoption.datasette publish cloudrunnow deploys with a newcontent_length_workaroundoption enabled, which omitsContent-Lengthfrom responses of 32,000,000 bytes or more because Cloud Run rejects larger responses that declare a length. See Large responses on Cloud Run. (#749)
Bug fixes
- Fixed a bug where following
next_urlthrough a sorted table with a text or compound primary key could repeat some rows and skip others, if several rows shared the same value in the sort column. Thanks, ZIHAN DAI. (#3054, #3055) - Fixed a bug where following
next_urlthrough a table sorted by a column with no declared type, such as a column in a table created usingCREATE TABLE ... AS SELECT, could skip or repeat rows. The sort value in thenexttoken was always compared as text, so numbers in those columns were compared against it incorrectly. The token now records whether the value is an integer, a floating point number or a BLOB.nexttokens issued by previous versions are still accepted. Thanks, peterolkhov. (#3056, #3057) - The
gt,gte,ltandltetable filters no longer return a 500 error for an integer too large for SQLite's signed 64-bit integers, such as?id__gt=9223372036854775808. Thanks, IlyaKhalafi. (#3035, #3036) - The create table, alter table and row edit dialogs no longer use the browser's native
confirm()prompt when discarding unsaved changes or dropping a table. Browsers embedded in other applications can dismiss those prompts without showing them, which left the "Drop table" button doing nothing and the create table dialog impossible to close. - Database downloads now send a
Content-Lengthheader without also sendingTransfer-Encoding: chunked, so browsers can show download progress and clients that reject that invalid combination, such as Node.js, can download the file. (#2663, #2317) If-None-Matchheaders are now compared against theETagfor database downloads and static files using the weak comparison specified by HTTP: aW/prefix is ignored, the header can list several values and*matches anything. Proxies that compress responses often turn a strongETaginto a weak one, which previously caused a full download on every request. The304response for a database download now includes theETagand CORS headers.- A
pluginsblock in metadata is now merged with thepluginsblock in configuration, instead of replacing it. - OpenTelemetry spans for requests made using
datasette.clientare now nested under the span that was active when the request was made, sharing its trace ID. Thanks, AnkitaAdvitot. (#2969, #2980) - Long values on the row page are no longer collapsed behind a "Show all 2,099 characters" toggle if plugin JavaScript has replaced the text with something else, such as a map. See Replacing values on the row page.
- Fixed a bug where links generated by output renderers using
path_with_format(), and the URLs returned by?_extra=rendererson.jsonpages, used a?_format=parameter that Datasette no longer reads. - Plugins loaded using
--plugins-dirnow have__spec__and__loader__set on their module. Python 3.15 emitted aDeprecationWarningwhenever an exception traceback passed through one of those plugins. - Fixed a scrolling bug on the
/-/queriespage. (#3053) - Fixed a bug where a database, table or saved query with a name such as
count,wrapperortagcould break the layout of its pages, because the class added to the<body>element for that name matched a class used by Datasette's own CSS and JavaScript.