dbcrossbar/dbcrossbar v0.4.0

0.4.0 - 2020-06-02

on GitHub

This is a summary of all the major changes since the 0.3.3 release. For more details and minor changes, see the individual CHANGELOG entries for the 0.4.0 preleases.

Added

• dbcrossbar now supports "struct" types, which have a fixed set of named fields. These will be automatically translated to BigQuery STRUCT types or to JSON columns, depending on the destination database.
• We now support a CLI-editable config file using commands like dbcrossbar config add temporary s3://example/temp/.
• Parsing-related error messages should now include context.
• bigquery: Users can now specify billing labels for jobs.
• dbcrossbar license will display the licences for all dependencies.
• Unstable features can now be hidden behind the --enable-unstable flag, including two new drivers:
  • UNSTABLE: We now support specifying schemas using a subset of TypeScript.
  • UNSTABLE: We now support reading data from Shopify's REST API. This is a testbed for new struct and JSON-related features.

Changed

• dbcrossbar conv is now dbcrossbar schema conv.
• Because of the new STRUCT support, some corner cases involving struct types and JSON may have changed subtly.
• We replaced gcloud auth, gsutil and bq with native Rust. This simplifies installation and configuration substantially, and fixes a number of BigQuery-related issues.
• AWS credentials must now always be passed via AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN (optional) and AWS_DEFAULT_REGION (required). This lays the groundwork for replacing the aws CLI tool with native Rust code, so that we will someday be able to remove our last CLI dependency.

Fixed

• Lots of issues.

Removed

• The data type { "other": string } has been removed from the portable schema format. It was not actually generated by any of our drivers.
• bigquery: We now export ARRAY<STRUCT<...>> as { "array": { "struct": ... } }, instead of exporting it as as single "json" value.