harelba/q v3.1.0-beta

q version 3.1.0-beta Release - Newer beta version exists, use it

on GitHub

There's a newer beta version available, please look at the releases page and take the latest beta version.

• Automatic Immutable Caching - Automatic caching of data files (into <my-csv-filename>.qsql files), with huge speedups for medium/large files. Enabled through -C readwrite or -C read
• Direct querying of standard sqlite databases - Just use it as a table name in the query. Format is select ... from <sqlitedb_filename>:::<table_name>, or just <sqlitedb_filename> if the database contains only one table. Multiple separate sqlite databases are fully supported in the same query.
• Direct querying of the qsql cache files - The user can query directly from the qsql files, removing the need for the original files. Just use select ... from <my-csv-filename>.qsql. Please wait until the non-beta version is out before thinking about deleting any of your original files...
• Revamped .qrc mechanism - allows opting-in to caching without specifying it in every query. By default, caching is disabled, for backward compatibility and for finding usability issues.
• Save-to-db is now reusable for queries - --save-db-to-disk option (-S) has been enhanced to match the new capabilities. You can query the resulting file directly through q, using the method mentioned above (it's just a standard sqlite database).
• Only python3 is supported from now on - Shouldn't be an issue, since q is a self-contained binary executable which has its own python embedded in it. Internally, q is now packaged with Python 3.8. After everything cools down, I'll probably bump this to 3.9/3.10.
• Minimal Linux Version Bumped - Works with CentOS 8, Ubuntu 18.04+, Debian 10+. Currently only for x86_64. Depends on glibc version 2.25+. Haven't tested it on other architectures. Issuing other architectures will be possible later on

For details on the changes and the new usage, see here

The version is still in early testing, for two reasons:

• Completely new build and packaging flow - Using pyoxidizer
• It's a very large change in functionality, which might surface issues, new and backward compatibility ones

Please don't use it for production, until the final non-beta version is out

If you're testing it out, I'd be more than happy to get any feedback. Please write all your feedback in this issue, instead of opening separate issues. That would really help me with managing this.