github opensearch-project/data-prepper 2.7.0

26 days ago

2024-03-27 Version 2.7.0


Features

  • Add a GeoIP processor. (#253, #3941, #3942)
  • Flatten json processor (#4128)
  • Add select_entries processor (#4147)
  • Decompress processor (#4016)
  • Support parsing of XML fields in Events (#4165, #4024)
  • Processor for parsing Amazon Ion documents (#3730)
  • Append values to lists in an event (#4129)
  • MapToList processor (#3935)
  • Date processor to convert from epoch_second, epoch_milli, or epoch_nano (#2929, #4076)
  • Support reading of old image for delete events on DynamoDB source (#4261)
  • Add string truncate processor to the family of mutate string processor (#3925)
  • Add join function (#4075)

Enhancements

  • Support format expressions for routing in the opensearch sink (#3833)
  • Allow . and @ characters to be part of json pointer in expressions (#4130)
  • Support maximum request length configurations in the HTTP and OTel sources (#3931)
  • Provide a config option to do node local aggregation (#4306)
  • Allow peer forwarder to skip sending events to remote peer (#3996)
  • Include encrypted data key in Kafka buffer message. (#3655)
  • Support larger message sizes in Kafka Buffer (#3916)
  • Modify S3 Source to allow multiple SQS workers (#4239)
  • Add support for tracking performance of individual Events in the grok processor (#4196)
  • Support codec on the file source to help with testing (#4018)
  • Provide a delay processor to put a delay in the processor for debugging and testing (#3938)
  • Support ByteCount in plugin parser (#3191)
  • Add Buffer Latency Metric (#4237)
  • Adds an append mode to the file sink (#3687)

Bug Fixes

  • Attempting to evaluate if a key is null throws an Exception if the value is a List for conditional expressions (#4109)
  • Data Prepper process threads stop when processors throw exceptions (#4103)
  • Upsert action requires existing document in OpenSearch (#4036)
  • Many Grok failures do not tag events (#4031)
  • Using update, upsert, or delete actions without specifying document_id crashes the pipeline with NPE (#3988)
  • OpenSearch Sink upsert action fails to create new document if it doesn't exist already (#3934)
  • DynamoDb source global state not found for export (#3579)
  • Missing Configuration details in Kafka documentation (#3157)
  • File Source fails to process large files. (#707)
  • Add key_value_when conditional to key_value processor (#4246)
  • Adds Kafka producer metrics for buffer usage (#4139)
  • Throw a more useful error when the S3 source is unable to determine bucket ownership (#4021)
  • Add sts_header_overrides to s3 dlq configuration (#3845)
  • Delay reading from the Kafka buffer as long as the circuit breaker is open (#4135)
  • Use timer for sink latency metrics (#4174)
  • Fix bug where process worker would shut down if a processor drops all events (#4262)
  • Send acknowledgements to source when events are forwarded to remote peer (#4305)
  • Injecting timestamp in index name that is not a suffix throws IllegalArgumentException (#3957)

Security

Maintenance

  • Create Kafka buffer integration tests for KMS (#3980, #4040)
  • Fixes Dependabot updates are not configured for all projects (#3301)

Don't miss a new data-prepper release

NewReleases is sending notifications on new releases.