snowplow/snowplow r73-cuban-macaw

Release 73 Cuban Macaw

on GitHub

Loads bad rows in batch pipeline into Elasticsearch, and formally separates the Snowplow enriched event format from the TSV format used to load Redshift.

• Blog post

EmrEtlRunner

• Bumped to 0.19.0
• Added hadoop_elasticsearch to config.yml.sample (#2124)
• Added support for Elasticsearch in targets section of config (#826)
• Bumped Elasticity to 6.0.5 (#2026)
• Stopped skipping the whole job just because enrich and shred are being skipped (#2049)

Scala Common Enrich

• Bumped Iglu Scala Client to 0.3.1 (#2079)
• Bumped version to 0.18.0
• Moved ScalazArgs into shared library (#2010)
• Removed executable bit from Scala source files (#2022)
• Removed JSON length checks (#2041)
• Removed truncation code (#2044)
• Stopped attempting to catch fatal errors (#2045)

Scala Hadoop Enrich

• Bumped to 1.3.0
• Bumped Scala Common Enrich to 0.18.0 (#2015)
• Added Iglu Scala Client as an explicit dependency (#2115)
• Added .forceToDisk to speed up run (#859)
• Started using Scala Common Enrich's version of ScalazArgs (#2013)

Scala Hadoop Shred

• Bumped to 0.6.0
• Added .forceToDisk to common to speed up run (#2039)
• Bumped Iglu Scala Client to 0.3.1 (#2081)
• Bumped Scala Common Enrich to 0.18.0 (#2016)
• Applied truncation logic to atomic-events TSV (#2042)
• Processed enriched events for atomic.events removing JSON fields (#1731)
• Started using Scala Common Enrich's version of ScalazArgs (#2014)

Storage

• Fixed README's link to architecture image, thanks @miike! (#2156)

Hadoop Elasticsearch Sink

• Added. (#824)

StorageLoader

• Bumped to 0.6.0
• Added tcpKeepAlive=true to JDBC for long-running COPYs via NAT (#2145)
• Fixed setup guide link in README, thanks @diamondo25! (#2025)
• Loaded atomic.events from shredded folder (#1795)

Postgres

• Added migration script for 0.6.0 to 0.7.0 (#2047)
• Removed JSON fields from atomic.events (#1949)

Redshift

• Added migration script for 0.4.0 to 0.8.0 (#2155)
• Added migration script for 0.5.0 to 0.8.0 (#2119)
• Added migration script for 0.6.0 to 0.8.0 (#2120)
• Added migration script for 0.7.0 to 0.8.0 (#2048)
• Removed JSON fields from atomic.events (#1849)

Data Modeling

• Added separators to custom fingerprint in deduplication queries (#2198)
• Renamed dvce_tstamp to dvce_created_tstamp in basic recipes (#2166)
• Removed JSON fields from deduplication queries (#2197)