snowplow/snowplow r91-stonehenge

Release 91 Stonehenge (2017-08-17)

on GitHub

EmrEtlRunner robustness.

• Blog post

EmrEtlRunner

• Use S3DistCp not Sluice for staging step (#276)
• Add an S3DistCp step for the _SUCCESS file produced by RDB Shredder (#3137)
• Add step to delete raw events from HDFS before shredding (#2545)
• Use S3DistCp to move raw files from S3 to HDFS for all collector formats (#3136)
• Add file- and Consul-based locking mechanism (#3352)
• Move current behavior into a run command (#3104)
• Add lint command which validates Iglu resolver and enrichments (#1946)
• Add backend for a generate command (#3105)
• Add --resume-from option (#3128)
• Remove support for --start and --end flags (#3132)
• Remove support for --process-enrich and --process-shred flags (#3365)
• Handle run= sub-folders if resuming from shred (#2693)
• Add "ongoing run" message on exit with return code 4 (#3129)
• Add "no logs to process" message on exit with return code 3 (#2644)
• Retrieve RDB loader logs only when it failed or the entire run was successful (#3361)
• Bump rspec to 3.5.0 (#3116)
• Bump to 0.27.0 (#3358)