PR #141 includes the following updates:
Schema/Data Changes
11 total changes • 11 possible breaking changes
| Data Model(s) | Change type | Old | New | Notes |
|---|---|---|---|---|
shopify_<gql>__customersshopify_<gql>__ordersshopify_<gql>__productsshopify_<gql>__inventory_levelsshopify_<gql>__daily_shopshopify_<gql>__collection_metafieldsshopify_<gql>__customer_metafieldsshopify_<gql>__order_metafieldsshopify_<gql>__product_metafieldsshopify_<gql>__product_variant_metafieldsshopify_<gql>__shop_metafields
| Metafield columns | Unlimited | 50 (configurable) | The 50 most commonly used metafields are pivoted into columns by default. You can adjust this number via the shopify_max_metafields variable.
|
Feature Updates
- Introduces the
shopify_max_metafieldsvariable to configure the number of metafields pivoted into columns. Configure it as follows (see README for details):
vars:
shopify_max_metafields: 200 # (Any positive integer) Default is 50Bug Fixes
- Deduplicates metafields with identical slugs (for example,
my-first-metafieldandmy_first_metafield) to prevent ambiguous column errors. - Ensures the
stg_shopify_gql__shop.enabled_presentment_currenciesandstg_shopify_gql__transaction.receiptfields are strings. - Applies the following warehouse-dependent limits on the number of metafields that are pivoted into columns, regardless of the value of
shopify_max_metafields. Metafields are included based on their prevalence in your ShopifyMETAFIELDtable:- BigQuery: 10,000
- Redshift: 1,600
- Postgres: 1,600
- Databricks: 32,768
- Snowflake: No limit
Under the Hood
- Adds data validation tests for metafield models.
- Adjusts metafield seed data to test for duplicate slugs/column names.
Full Changelog: v1.3.1...v1.4.0-a1