Brings our Kinesis pipeline functionally up-to-date with our Hadoop pipeline, and makes various further improvements to the Kinesis pipeline.
Common
- Removed openjdk7 from .travis.yml (#2533)
Scala Common Enrich
- Bumped to 0.22.0
- Added handling for bad rows which are too long to print in full (closes #2419)
Kinesis
- Updated publish-kinesis-release.bash (closes #2477)
Scala Stream Collector
- Bumped to 0.6.0
- Added Scala Common Enrich as a library dependency (closes #2153)
- Added click redirect mode (closes #549)
- Configured the ability to use IP address as partition key (closes #2331)
- Converted bad rows to new format (closes #2006)
- Shared a single thread pool for all writes to Kinesis (closes #2369)
- Specified UTF-8 encoding everywhere (closes #2147)
- Made cookie name customizable, thanks @kazjote! (closes #2474)
- Added boolean collector.cookie.enabled setting (closes #2488)
- Made backoffPolicy fields macros (closes #2518)
- Updated AWS credentials to support iam/env/default not cpf (closes #1518)
Scala Kinesis Enrich
- Bumped to 0.7.0
- Renamed to Stream Enrich (closes #2418)
- Bumped Kinesis Client Library to 1.6.1 (closes #1823)
- Bumped Scala Common Enrich to 0.21.0 (closes #2033)
- Bumped Iglu Scala Client to 0.3.1 (closes #2080)
- Configured the ability to use IP address as partition key (closes #2332)
- Started emitting KCL metrics to CloudWatch (closes #2357)
- Converted bad rows to new format (closes #1207)
- Removed outdated comment about ClasspathPropertiesFileCredentialsProvider from sample config file (closes #1519)
- Removed redundant documentation from README (closes #2032)
- Updated test suite with valid self-describing JSONs (closes #2151)
- Updated Scala Tracker to 0.2.0 and enabled EC2 context (closes #2109)
- Updated to use new EtlPipeline (closes #1933)
- Specified UTF-8 encoding everywhere (closes #2148)
Kinesis Elasticsearch Sink
- Bumped to 0.5.0
- Bumped Kinesis Client Library to 1.6.1 (closes #1824)
- Bumped Scala Common Enrich to 0.22.0 (closes #2152)
- Added mixed output mode (closes #2412)
- Added new canonical event fields (closes #2089)
- Moved the stream-type setting into the main sink configuration object (closes #2490)
- Made source and sink fields macros (closes #2519)
- Renamed Build object to match project (closes #2002)
- Converted bad rows to new format (closes #1208)
- Updated schema regular expression in line with Iglu Central (fixes #1998)
- Cached the mapping of field name to field type (closes #2090)
- Specified UTF-8 encoding everywhere (closes #2149)
- Stopped sending timestamp instead of failure count (fixes #1951)
- Made performance of conversion from TSV to JSON linear (closes #1847)
- Updated to latest version of EnrichedEvent (closes #2089)