This is a monstrous👹 release with lots of new features and improvements!
The biggest new feature is the describegpt
command which allows you to use OpenAI's Large Language Models to generate extended metadata from a CSV. We created this command primarily for CKAN and Datapusher+ so we can infer descriptions, tags and to automatically created annotated data dictionaries using the CSV's summary statistics and frequency tables. In that way, it works even for very large CSV files without consuming too many Open AI tokens. This is a very powerful feature and we are looking forward to seeing what people do with it. Thanks @rzmk for all the work on this!
This release also features major improvements in the sqlp
and joinp
commands thanks to all the new capabilities of Polars 0.31.1.
Polars SQL's capabilities have been vastly improved in 0.31.1 with numerous new SQL functions and operators, and they're all available with the sqlp
command.
The joinp
command has several new options for CSV parsing, for pre-join filtering (--filter-left
and --filter-right
), and pre-join validation with the --validate
option. Two new asof join variants (--left_by
and --right_by
) were also added.
Added
describegpt
command by @rzmk in #1036describegpt
: minor refactoring in #1104describegpt
:--key
& QSV_OPENAI_API_KEY by @rzmk in #1105describegpt
: add--user-agent
in help message by @rzmk in #1095describegpt
: json output format for redirection by @rzmk in #1107describegpt
: add testing (resolves #1114) by @rzmk in #1115describegpt
: add--model
option (resolves #1101) by @rzmk in #1117describegpt
: polishing #1122describegpt
: add--jsonl
option (resolves #1086) by @rzmk in #1127describegpt
: add--prompt-file
option (resolves #1085) by @rzmk in #1120joinp
: addedasof_by
join variant; added CSV formatting options consistent with sqlp CSV format options #1090joinp
: add--filter-left
and--filter-right
options #1146joinp
: add--validate
option #1147fetch
&fetchpost
: add--no-cache
option #1112sniff
: detect file kind along with mime type #1137- user-agent metadata now contains the current command's name #1093
Changed
fetch
&fetchpost
: --redis and --no-cache are mutually exclusive #1113luau
: adapt to mlua 0.9.0-rc.1 API changes #1129- upgrade to Polars 0.31.1 #1139
- Bump MSRV to latest Rust stable (1.71.0)
- pin Rust nightly to 2023-07-15
- Bump uuid from 1.3.4 to 1.4.0 by @dependabot in #1073
- Bump tokio from 1.28.2 to 1.29.0 by @dependabot in #1077
- Bump tokio from 1.29.0 to 1.29.1 by @dependabot in #1087
- Bump sysinfo from 0.29.2 to 0.29.3 by @dependabot in #1088
- build(deps): bump sysinfo from 0.29.4 to 0.29.5 by @dependabot in #1148
- Bump jql-runner from 6.0.9 to 7.0.0 by @dependabot in #1092
- build(deps): bump jql-runner from 7.0.0 to 7.0.1 by @dependabot in #1132
- Bump itoa from 1.0.6 to 1.0.7 by @dependabot in #1091
- Bump itoa from 1.0.7 to 1.0.8 by @dependabot in #1098
- build(deps): bump itoa from 1.0.8 to 1.0.9 by @dependabot in #1142
- Bump serde from 1.0.164 to 1.0.165 by @dependabot in #1094
- Bump serde from 1.0.165 to 1.0.166 by @dependabot in #1100
- Bump serde from 1.0.166 to 1.0.167 by @dependabot in #1116
- build(deps): bump serde from 1.0.167 to 1.0.171 by @dependabot in #1118
- Bump pyo3 from 0.19.0 to 0.19.1 by @dependabot in #1099
- Bump ryu from 1.0.13 to 1.0.14 by @dependabot in #1096
- build(deps): bump ryu from 1.0.14 to 1.0.15 by @dependabot in #1144
- Bump strum_macros from 0.25.0 to 0.25.1 by @dependabot in #1097
- Bump serde_json from 1.0.99 to 1.0.100 by @dependabot in #1103
- build(deps): bump serde_json from 1.0.100 to 1.0.101 by @dependabot in #1123
- build(deps): bump serde_json from 1.0.101 to 1.0.102 by @dependabot in #1125
- build(deps): bump serde_json from 1.0.102 to 1.0.103 by @dependabot in #1143
- Bump serde_stacker from 0.1.8 to 0.1.9 by @dependabot in #1110
- Bump regex from 1.8.4 to 1.9.0 by @dependabot in #1109
- build(deps): bump regex from 1.9.0 to 1.9.1 by @dependabot in #1119
- Bump jsonschema from 0.17.0 to 0.17.1 by @dependabot in #1108
- build(deps): bump cpc from 1.9.1 to 1.9.2 by @dependabot in #1121
- build(deps): bump governor from 0.5.1 to 0.6.0 by @dependabot in #1128
- build(deps): bump actions/setup-python from 4.6.1 to 4.7.0 by @dependabot in #1134
- build(deps): bump file-format from 0.17.3 to 0.18.0 by @dependabot in #1136
- build(deps): bump serde_stacker from 0.1.9 to 0.1.10 by @dependabot in #1141
- build(deps): bump semver from 1.0.17 to 1.0.18 by @dependabot in #1140
- cargo update bump several indirect dependencies
Fixed
fmt
: Quote ASCII format differently by @LemmingAvalanche in #1075apply
: makedynfmt
subcommand case sensitive. Fixes #1126 #1130applydp
: makedynfmt
case-sensitive #1131describegpt
: docs/Describegpt.md: typo 'a' --> 'an' by @rzmk in #1135tojsonl
: support snappy-compressed input. Fixes #1133 #1145- security.md: fix mailto text by @rzmk in #1079
New Contributors
- @LemmingAvalanche made their first contribution in #1075
Full Changelog: 0.108.0...0.109.0