A major release which adds two new enrichments, upgrades existing enrichments and significantly extends and improves our Canonical Event Model for loading into Redshift, Elasticsearch and Postgres.
Common
- Updated kinesis push to remove sub-folders from zipfile (#1378)
EmrEtlRunner
- Added example configuration JSONs for new enrichments (#1545)
Scala Common Enrich
- Bumped to 0.13.0
- Bumped referer-parser to 0.2.3 (#670)
- Converted transactions from given currency to base currency (#370)
- Bumped CampaignAttributionEnrichment version to 0.2.0 (#1338)
- Added mkt_clickid and mkt_network fields to POJO (#1073)
- Added derived_contexts field to POJO (#787)
- Added geo_timezone field to POJO (#787)
- Added etl_tags field to POJO (#1247)
- Added currency fields to POJO (#1316)
- Changed enrichment configuration to use SchemaCriterion rather than SchemaKey (#1353)
- Extracted original IP address from CollectorPayload headers (#1372)
- Extracted dvce_sent_tstamp from stm field (#1383)
- Added dvce_sent_tstamp to POJO (#1384)
- Added refr_domain_userid and refr_dvce_sent_tstamp to POJO (#1449)
- Added domain_sessionid field to POJO (#1538)
- Added derived_tstamp field to POJO (#1557)
- Populated refr_ fields based on page_url querystring (#1461)
- Populated domain_sessionid field based on "sid" parameter (#1541)
- Parsed the page URI in the EnrichmentManager (#1463)
- Added ua-parser enrichment (#62)
- Added ability to disable user-agent-utils enrichment (#792)
- Used Netaporter to parse querystrings if httpclient fails, thanks @danisola! (#1429)
Scala Hadoop Enrich
- Bumped to 0.14.0
- Bumped Scala Common Enrich to 0.13.0 (#1340)
- Added integration tests for currency conversion enrichment (#1430)
- Added tests for other new EnrichedEvent fields (#1337)
Scala Hadoop Shred
- Bumped to 0.4.0
- Bumped Scala Common Enrich to 0.13.0 (#1343)
- Bumped json4sJackson to 3.2.11 (#1344)
- Extracted JSONs from derived_contexts field (#786)
- Updated to reflect new enriched event format (#1332)
Scala Kinesis Enrich
- Bumped to 0.4.0
- Bumped Scala Common Enrich to 0.13.0 (#1369)
- Emitted updated EnrichedEvent (#1368)
- Unified logger configuration, thanks @kazjote! (#1367)
Redshift
- Added refr_domain_userid and refr_dvce_tstamp to atomic.events (#1450)
- Added dvce_sent_tstamp column (#1385)
- Added foreign key constraint to all Redshift shredded tables (#1365)
- Changed JSON field encodings to lzo (closes #1350)
- Added migration script for 0.4.0 to 0.5.0 (#1335)
- Added etl_tags column (#1245)
- Added column for mkt_clickid and mkt_network (#1093)
- Widened domain_userid column to hold UUID (#1090)
- Added Redshift DDL for ua_parser_context (#789)
- Added new derived_contexts field (#784)
- Updated ip_address to support IPv6 addresses (#656)
- Added new currency fields (#366)
- Added domain_sessionid column (#1539)
- Widened structured event, URL, and referer fields (#1553)
- Added derived_tstamp column (#1558)
Postgres
- Added refr_domain_userid and refr_dvce_tstamp to atomic.events (#1451)
- Added dvce_sent_tstamp column (#1386)
- Added migration script for 0.3.0 to 0.4.0 (#1347)
- Added column for geo_timezone (#1336)
- Added etl_tags column (#1246)
- Removed primary key constraint on event_id (#1187)
- Added column for mkt_clickid and mkt_network (#1092)
- Widened domain_userid column to hold UUID (#1091)
- Added new derived_contexts field (#785)
- Updated ip_address to support IPv6 addresses (#655)
- Added new currency fields (#365)
- Added domain_sessionid column (#1540)
- Widened structured event, URL, and referer fields (#1554)
- Added derived_tstamp column (#1559)
StorageLoader
- Wrote JSON Path file for ua_parser_context (#790)