New features, stability enhancements and performance improvements for EmrEtlRunner and StorageLoader. As of this release EmrEtlRunner lets you specify EBS volumes for your Hadoop worker nodes; meanwhile StorageLoader now writes to a dedicated manifest table to record each load
EmrEtlRunner
- Bump to 0.23.0 (#2960)
- Bump JRuby version to 9.1.6.0 (#3050)
- Bump Elasticity to 6.0.10 (#3013)
- Remove AnonIpHash from contracts.rb (#2523)
- Remove UnmatchedLzoFilesError check (#2740)
- Use S3DistCp not Sluice for archive_raw step (#1977)
- Add warning about the array of in buckets in config.yml (#2462)
- Add dedicated return code of 4 for DirectoryNotEmptyError (#2546)
- Add support for specifying EBS for Hadoop workers (#2950)
- Add example EBS configuration to config.yml.sample (#3012)
- Catch Elasticity ThrottlingExceptions while waiting for EMR (#3028)
- Catch Elasticity ArgumentErrors while waiting for EMR (#3027)
StorageLoader
- Bump to 0.9.0 (#2961)
- Bump JRuby version to 9.1.6.0 (#3051)
- Fix typo in S3Tasks.download_events (#2888)
- Update manifest table as part of Redshift load transaction (#2280)
Redshift
- Added manifest table (#2265)