2025-04-24 Version 2.11.0
Features
- Support AWS Aurora/RDS PostgreSQL as source (#5309)
- Support SQS as a Source (#1049)
- Jira Connector - to seamlessly sync all the ticket details to OpenSearch (#4754)
- AWS Lambda as Processor and Sink (#4699)
- Support AWS Aurora/RDS MySQL as source (#4561)
- AWS Lambda as Sink (#4170)
- Integrate OpenSearch Ml-Commons into Data Prepper (#5509)
Enhancements
- Update OTel Protobuf specification (#5434)
- Data Prepper support for dynamic renaming of keys (#4849)
- S3 DLQ should pass expected bucket owner to PutObject request (#5498)
- AWS Secrets Manager Plugin does not support sts_header_overrides (#5475)
- Support configurable stream read constraints max length in the JSON input codec (#5466)
- Support reading S3 object metadata only (#5433)
- Kafka local AWS Glue registry support (#5377)
- Address Scale Items for Lambda Processor and Sink (#5031)
- Add support to skip remote peer forwarding based on configuration (#5127)
- Add index_types for OTel logs and metrics #3148 (#3929)
- Send RetryInfo on OTel timeouts to improve or clarify backpressure on OTel gRPC sources (#4294, #4119)
- Enhance Lambda processor to retry based on certain class of exception (#5340)
- Support multiple delete_when condition in delete_entries processor (#5315)
- Add additional index_types (#3148)
- Add experimental feature concept (#2695)
Bug Fixes
- NullPointerException on S3 Delete Event Due to Null Object Size (#5448)
- Index Template with flat_object type field fails during deserialization (#5425)
- DynamoDB source with acknowledgements expires frequently (#5412)
- Kinesis source doesn't pass the given polling retrieval config to underlying KCL (#5269)
- UTF-8 Character Encoding Issues in opensearchproject/data-prepper container (#5238)
- Validate that routes configured in the sink exist on startup of Data Prepper (#5106)
- Escaping of "/" in json pointers (#5101)
- DynamoDB Source doesn't support parsing data with Control Characters (#5027)
Security
- [docker] Run as non root (#5311)
- werkzeug-3.0.3-py3-none-any.whl: 2 vulnerabilities (highest severity is: 7.5) (#5122)