opensearch-project/data-prepper 2.0.0

on GitHub

2022-10-10 Version 2.0.0

Breaking Changes

• Replaced the data-prepper-tar-install.sh script with bin/data-prepper
• Replaced the single Jar file with a directory structure and the bin/data-prepper script.
• Data Prepper pipelines no longer support prepper:. Use processor: instead.
• Data Prepper now requires Java 11 or higher to run.
• Removed properties from the opensearch sink: trace_analytics_raw and trace_analytics_service_map. Use the index_type instead.
• Renamed two grok metric names. Renamed grokProcessingMatchSuccess to grokProcessingMatch and grokProcessingMatchFailure to grokProcessingMismatch.
• Removed record_type from otel_trace_source.
• Removed otel_trace_raw_prepper - use otel_trace_raw instead.
• Removed otel_trace_group_prepper - use otel_trace_group instead.
• Removed peer-forwarder processor plugin.
• Incorrect HTTP methods were removed from Data Prepper core HTTP APIs.
• All APIs have been renamed to use the org.opensearch.dataprepper package.

Features

• Support conditional routing in sinks (#1337)
• Support Core Peer Forwarder which replaces the peer forwarder processor plugin (#700)
• Support parsing CSV and TSV values in event fields using CSV processor (#1081)
• Support decoding CSV or TSV S3 objects using CSV codec in S3 source (#1644)
• Support processing JSON values in event fields using JSON processor (#831)
• Support multi node aggregation using Core Peer Forwarder (#978)
• Use peer forwarding in service_map_stateful processor using Core Peer Forwarder (#1765)
• Use peer forwarding in otel_trace_raw processor using Core Peer Forwarder (#1766)

Enhancements

• Updates default configurations for bounded_buffer - buffer_size is 12,800, batch_size is 200.
• Support concurrency in Data Prepper expression (#1189)
• Support bulk create option in OpnSearch sink (#1561)
• Added new metric to track utilization rate of buffer (#1817)
• Support publishing metrics for Data Prepper core (#1789)
• Support configurable timeouts for processors and sinks to flush the data downstream before shutdown (#1742)
• Moved the fields from S3 JSON objects up to the root level of the object to promote consistency with other S3 codecs (#1687)
• Support Duration in Data Prepper server configuration (#1623)
• Fixed health check bug when Auth is enabled for HTTP and OTel trace source (#1600)
• Enabled HTTP health check for OTel trace source and OTel metrics source (#1546)
• Made Java 11 to be the baseline version for Data Prepper (#1422)
• Updated HTTP source request timeout to use configured timeout (#975)
• Distribute Data Prepper Docker image with JDK 17 (#694)
• Support ACM and S3 for TLS/SSL in HTTP source (#365)
• Updated HTTP methods supported on core API endpoints (#313)

Bug Fixes

• Fix a bug where file sink fails to write to output file with multiple pipeline threads (#1843)
• Fixed a bug S3 source poll delay (#1550)
• Fixed a bug where Data Prepper stops if there's an error reading S3 object (#1544)

Infrastructure

• Improved Data Prepper assemble task to create a runnable distribution (#1762)
• Updated end-to-end tests with core peer forwarder for trace tests (#1866)
• Support directory structure for Data Prepper (#305)

Maintenance

• Removed support for OTLP protocol as internal data transfer in trace pipeline (#1272)
• Update Jackson to 2.13.4 (#1871)
• Updated Armeria to 1.19.0 (#1806)
• Removed Peer Forwarder processor plugin (#1874)
• Removed deprecated type property in DataPrepperPlugin annotation (#1657)
• Removed deprecated trace_analytics_raw and trace_analytics_service_map values, index_type replaces them (#1648)
• Removed deprecated PluginFactory (#1584)
• Removed AWS SDK v1 entirely from S3 source (#1562)
• Updated Antlr to 4.10.1 (#1513)
• Updated all existing prepper plugins to use only processor. (#647)
• Removed deprecated prepper plugin type from pipeline configuration. (#619)
• Updated package naming to org.opensearch from com.amazon (#344)