Upgrades our Hadoop Enrichment process to run on Hadoop 2.4, re-enables our Kinesis-Hadoop lambda architecture and also introduces a new scriptable enrichment powered by JavaScript.
Documentation
- Replaced Hive ETL references with Kinesis Enrich in Scala Hadoop Enrich's README (#1671)
- Fixed links in Scala Common Enrich's README.md, thanks @bigsnarfdude! (#1669)
Scala Tracker
- Added git submodule (#1724)
Scala Hadoop Enrich
- Bumped to 1.0.0
- Renamed build to snowplow-hadoop-enrich (#1718)
- Updated dependencies to Hadoop 2.4 (#1716)
- Bumped Scala Common Enrich to 0.14.0 (#1700)
- Updated Core2015RefreshSpec to include JavascriptScriptEnrichment (#1746)
Scala Common Enrich
- Bumped to 0.14.0
- Added JavaScript scripting enrichment (#378)
- Made IpLookupsEnrichment error message more informative (#1426)
- commons-codec dependency is no longer test-only (#1712)
- Bumped commons-lang3 to 3.4 (#1713)
- Made mkt_ and refr_ fields TSV safe, thanks @jasonbosco! (#1643)
- Updated JodaTime dependency to 2.2 (#1748)
- Now handles null message in stripInstanceEtc (#1622)
EmrEtlRunner
- Bumped to 0.15.0
- Now using new scala-hadoop-enrich jar path in Hosted Assets (#1719)
- Updated ami_version in config.yml to 3.6.0 (#1651)
- Added bootstrap action to prepare AMI 3.x for Snowplow (#1714)
- Now setting buffer for processing thrift in core-site.xml (#1715)
- Added S3DistpCp step for thrift files in processing (#1647)
- Added example javascript_script_config to enrichments folder (#1755)
StorageLoader
- Wrote JSON Path file for com.mparticle.snowplow/app_event (#1688)
- Wrote JSON Path file for com.mparticle.snowplow/social_event (#1690)
- Wrote JSON Path file for com.mparticle.snowplow/transaction_event (#1692)
- Wrote JSON Path file for a com.mparticle.snowplow/session_context (#1694)
Redshift
- Added Redshift DDL for a com.mparticle.snowplow/app_event (#1686)
- Added Redshift DDL for a com.mparticle.snowplow/social_event (#1689)
- Added Redshift DDL for a com.mparticle.snowplow/transaction_event (#1691)
- Added Redshift DDL for a com.mparticle.snowplow/session_context (#1693)
Data Modeling
- Removed restrictions in sessions and visitors-source (#1725)