Introduces an event de-duplication process which runs on Hadoop, plus an important bug fix for our recent SendGrid webhook support
Scala Hadoop Enrich
- Bumped to 1.5.1
- Bumped Scala Common Enrich to 0.20.1 (#2338)
Scala Common Enrich
- Bumped to 0.20.0
- Now using only base MIME type in content-type check for SendGrid Adapter (#2328)
Scala Hadoop Shred
- Bumped to 0.7.0
- Fixed good tests' checks for empty paths (#2278)
- Now deduplicating event_id and event_fingerprint pairs (#2246)
- Fixed incorrect event in SchemaValidationFailed1Spec (#2355)
- Updated tests to check atomic-events output (#2264)
- Now only writes atomic-events if JSONs shred successfully (#2245)
- Removed empty SchemaValidationFailed2Spec (#2271)
- Fixed test suite issue with multiple input lines (#2270)
EmrEtlRunner
- Updated hadoop_enrich version in config.yml.sample to 1.5.1 (#2339)
- Changed in bucket example in config.yml.sample to s3://my-in-bucket (#2358)
- Updated archive bucket examples in config.yml (#2368)
- Updated hadoop_shred version in config.yml.sample to 0.7.0 (#2360)
StorageLoader
- Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/action (#2136)
- Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2135)
- Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2134)
- Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/productFieldObject (#2133)
- Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2132)
Redshift
- Added Redshift DDL for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2131)
- Added Redshift DDL for com.google.analytics.enhanced-ecommerce/productFieldObject (#2130)
- Added Redshift DDL for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2129)
- Added Redshift DDL for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2128)
- Added Redshift DDL for com.google.analytics.enhanced-ecommerce/action (#2127)