A host of upgrades to the Kinesis pipeline as well as the embedding of Snowplow tracking into this pipeline.
Common
Common: added NFS and CORE configuration to Vagrantfile to enhance performance (#1831)
Scala Stream Collector
- Bumped to 0.5.0
- stdout bad sink now prints to stderr (#1799)
- Added splitter for large event arrays (#941)
- Increased maximum record size from 50kB to 1MB (#1753)
- Added tests for splitting large requests (#1683)
- Updated bad rows to include timestamp (#1681)
- Handled case where IP is not present (#1680)
- Did some reorganisation and refactoring of the project (#1678)
- Added json4s dependency (#1673)
- Added bad stream (#1502)
Scala Common Enrich
- Bumped to 0.15.0
- Fixed JavascriptScriptEnrichmentSpec test to pass openjdk7 (#1793)
- Bumped scala-maxmind-iplookups to 0.3.0 (#1771)
- Bumped Scala Forex to 0.3.0 (#1770)
- Updated bad rows to include timestamp (#1577)
Scala S3 Sink
- Removed project from repo (#1672)
Scala Kinesis Enrich
- Bumped to 0.6.0
- Bumped to Scala Common Enrich 0.15.0 (#1685)
- Tries to send 503 records (#1756)
- Made back-off fields macros (#1745)
- Increased maximum record size to 1MB (#1736)
- Logging all bad rows (#1722)
- Exception installing MaxMind file must terminate (#1711)
- Sending Snowplow hearbeat (#1406)
- Allowed records of over 1Mb when running in local mode (#1663)
- Fixed error when fetching MaxMind file from s3:// URI (#1645)
- Sending a warning via Snowplow if no enrichment JSONs are retrieved from DynamoDB (#1621)
- Sending failure to sink event to kinesis to Snowplow (#1798)
- etl_tstamp should be Redshift Formatted not raw (#1842)
Kinesis Elasticsearch Sink
- Bumped to 0.4.0
- Removed Scala Common Enrich as an assembly dependency (#1819)
- Bumped to Scala Common Enrich 0.15.0 (#1811)
- Allowed use of AWS creds instead of DefaultAWSCredentialsProviderChain (#1803)
- App no longer hangs without shutting down (#1743)
- Updated the Elasticsearch version (#1734)
- Sent event to Snowplow on heartbeat (#1706)
- Added Scala Tracker dependency (#1705)
- Sending event to Snowplow when unable to write to Elasticsearch (#1704)
- Sending event to Snowplow on shutdown (#1703)
- Sending event to Snowplow on initialization (#1702)
- Initialized bad stream eagerly rather than lazily (#1677)
- Updated amazon-kinesis-connectors to 1.1.2 (#1675)
- Specifying character encoding in SnowplowElasticsearchTransformer (#1654)
- Updated bad rows to include timestamp (#1578)
- Moved location fields into elasticsearch section (#1517)
- corrected shredding example in comment (#1276)
Redshift
- Added Redshift DDL for com.snowplowanalytics.monitoring/application_warning (#1809)
- Added Redshift DDL for com.snowplowanalytics.monitoring/heartbeat (#1764)
- Added Redshift DDL for com.snowplowanalytics.monitoring/sink_write_failed (#1763)
- Added Redshift DDL for com.snowplowanalytics.monitoring/application_initialized (#1762)
- Added Redshift DDL for com.snowplowanalytics.monitoring/application_shutdown (#1761)
- Added Redshift DDL for com.snowplowanalytics.monitoring/stream_write_failed (#1844)
- Added Redshift DDL for com.snowplowanalytics.snowplow/web_page (#1835)
- Added migration script for 0.3.0 to 0.6.0 (#1832)
- Added migration script for 0.4.0 to 0.6.0 (#1833)
StorageLoader
- Wrote JSON path file for com.snowplowanalytics.monitoring/application_warning (#1810)
- Wrote JSON path file for com.snowplowanalytics.monitoring/heartbeat (#1760)
- Wrote JSON path file for com.snowplowanalytics.monitoring/sink_write_failed (#1759)
- Wrote JSON path file for com.snowplowanalytics.monitoring/application_initialized (#1758)
- Wrote JSON path file for com.snowplowanalytics.monitoring/application_shutdown (#1757)
- Wrote JSON path file for com.snowplowanalytics.monitoring/stream_write_failed (#1843)
- Wrote JSON path file for com.snowplowanalytics.snowplow/web_page (#1836)