2.32.0
This release includes:
- Fluent Bit 1.9.10
- Amazon CloudWatch Logs for Fluent Bit 1.9.4
- Amazon Kinesis Streams for Fluent Bit 1.10.2
- Amazon Kinesis Firehose for Fluent Bit 1.7.2
Compared to 2.31.12 this release adds:
- Enhancement - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on
prometheus_remote_write. Refer to amazon-ecs-firelens-examples for information on how to export metrics to AMP on ECS aws-for-fluent-bit:256 - Enhancement - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file aws-for-fluent-bit:537
- Enhancement - Customers can route logs to CloudWatch Logs at higher throughputs by increasing number of output workers as
cloudwatch_logsoutput plugin removed sequence tokens from API requests aws-for-fluent-bit:526 - Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between fluent-bit:7469
- Fix - Fix networking edgecase causing data loss and OOM related issues via net timeout event injection resolution fluent-bit:7728
- Fix - Allow reduced filter_throttle log output with
print_statusfalseconfiguration option fix fluent-bit:7469
We’ve run the new released image in our ECS load testing framework and here is the result. This testing result provides benchmarks of aws-for-fluent-bit under different input load. Learn more about the load test.
| plugin | source | 20 MB/s | 25 MB/s | 30 MB/s | |
|---|---|---|---|---|---|
| kinesis_firehose | stdstream | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | 0%(4000) | ✅ | ||
| kinesis_firehose | tcp | Log Loss | ✅ | ✅ | 0%(10104) |
| Log Duplication | ✅ | 0%(4000) | 0%(6000) | ||
| kinesis_streams | stdstream | Log Loss | ✅ | ✅ | 0%(14642) |
| Log Duplication | 0%(30639) | 0%(11704) | 0%(147188) | ||
| kinesis_streams | tcp | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | ✅ | 0%(56814) | ||
| s3 | stdstream | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | ✅ | ✅ | ||
| s3 | tcp | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | ✅ | ✅ |
| plugin | source | 1 MB/s | 2 MB/s | 3 MB/s | |
|---|---|---|---|---|---|
| cloudwatch_logs | stdstream | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | ✅ | ✅ | ||
| cloudwatch_logs | tcp | Log Loss | ✅ | ✅ | ✅ |
| Log Duplication | ✅ | ✅ | ✅ |
Note:
- The green check ✅ in the table means no log loss or no log duplication.
- Number in parentheses means the number of records out of total records. For example, 0%(1064/1.8M) under 30Mb/s throughput means 1064 duplicate records out of 18M input records by which log duplication percentage is 0%.
- Log loss is the percentage of data lost and log duplication is the percentage of duplicate logs received at the destination. Your results may differ because they can be influenced by many factors like different configs and environment settings. Log duplication is exclusively caused by partially succeeded batches that were retried, which means it is random.