jqnatividad/qsv 0.134.0

on GitHub

qsv pro v1 is here! 🎉

If you've been using qsv for a while, even if you're a command-line ninja, you'll find a lot of new capabilities in qsv pro that can make your data wrangling experience even better!

Apart from making qsv easier to use, qsv pro has a multitude of features including: view interactive data tables; browse stats/frequency/metadata; run recipes and tools (scripts); run Polars SQL queries; use Natural Language queries (using Retrieval Augmented Generation (RAG) techniques); regular expression search; export to multiple file formats; download/upload from/to compatible CKAN instances; design custom node-based flows and data pipelines; interact with a local API from external programs including the qsv pro command; run various qsv commands in a graphical user interface; and the list goes on!

And that's just the beginning, there's more to come! You just have to try it!

Download qsv pro v1 now at qsvpro.dathere.com.

Other highlights include:

• pro: new command to allow qsv to interact with the qsv pro API to tap into qsv pro exclusive features.
• lens: new command to interactively view CSVs using the csvlens crate.
• The ludicrously fast diff command is now easier to use with its --drop-equal-fields option. @janriemer continues to work on his csv-diff crate, and there's more diff UX improvements coming soon!
• stats adds sum_length and avg_length "streaming" statistics in addition to the existing min_length and max_length metrics. These are especially useful for datasets with a lot of "free text" columns.
• stats also got "smarter" and "faster" by dog-fooding its own statistics to make it run faster!
  It's a little complicated, but the way stats works is that it compiles the "streaming" statistics on the fly first as it multiplex load the data across several threads, and the more expensive advanced statistics are "lazily" computed at the end.
  Since we now compile "sort order" in a streaming manner, we use this info when deriving cardinality at the end to see if we can skip sorting - an otherwise necessary step to get cardinality which is done by "scanning" all the sorted values of a column. Everytime two neighboring values differ in a sorted column, it increments the cardinality count.
  Apart from this "sort order" optimization, we also improved the "cardinality scan" algorithm - halving its memory footprint and making it faster still for larger datasets by parallelizing the computation. This in turn, makes the frequency command faster and more memory efficient.
  It's performance tweaks like these, that despite adding six metrics (is_ascii, sort_order, sum_length, avg_length, sem - standard error of the mean & cv - coefficient of variation) in recent releases, that stats is still able to compile 35 statistics and do GUARANTEED data type inferences of a million row, 41 column, 520 MB sample of NYC's 311 data in 1.327 seconds (753,580 records per second)!¹
• we now also use our own fork of the csv crate, featuring SIMD-accelerated UTF-8 validation and other minor perf tweaks, making the entire qsv suite faster still!

Added

• pro: add qsv pro command to interact with qsv pro API by @rzmk in #2039
• lens: new command to interactively view CSVs using the csvlens crate #2117
• apply: add crc32 operation #2121
• count: add --delimiter option #2120
• diff: add flag --drop-equal-fields by @janriemer in #2114
• stats: add sum_length and avg_length columns #2113
• stats: smarter cardinality computation - added new parallel algorithm for large datasets (10,000+ rows) and updated sequential algorithm for smaller datasets 4e63fec

Changed

• count: added comment to justify magic number 5241e39
• stats: use simdjson for faster JSONL parsing; micro-optimize compute hot loop 0e8b734
• stats: standardized OVERFLOW and UNDERFLOW messages 38c6128
• sort: renamed symbol so eliminate devskim lint false positive warning 12db739
• enable lens feature in GH workflows #2122
• deps: bump polars 0.42.0 to latest upstream at time of release 3c17ed1
• deps: use our own optimized fork of csv crate, with simdutf8 validation and other minor perf tweaks e4bcd71
• build(deps): bump serde from 1.0.209 to 1.0.210 by @dependabot in #2111
• build(deps): bump serde_json from 1.0.127 to 1.0.128 by @dependabot in #2106
• build(deps): bump qsv-stats from 0.19.0 to 0.22.0 #2107 #2112 cb1eb60
• apply select clippy lint suggestions
• updated several indirect dependencies
• made various doc and usage text improvements

Fixed

• schema: Print an error if the qsv stats invocation fails by @abrauchli in #2110

New Contributors

• @abrauchli made their first contribution in #2110

Full Changelog: 0.133.1...0.134.0