github aws-powertools/powertools-lambda-python v3.8.0
on GitHub

2 days ago

Summary

We are excited to announce a new feature in Logger: Logger buffering. This new feature allows you to buffer logs for a specific invocation, and flush them automatically on error or manually as needed.

A special thanks to Ollie Saul and James Saull from Dotelastic for their instrumental input on this new feature!

⭐ Also, huge thanks to our new contributors: @tiagohconte and @speshak.

New Log Buffering feature

Docs

You can now enable log buffering by passing buffer_config when initializing a new Logger instance. This feature allows you to:

  • Buffer logs at the WARNING, INFO, and DEBUG levels
  • Automatically flush logs on error or manually as needed
  • Reduce CloudWatch costs by decreasing the number of emitted log messages

buffer1

Configuration options

Option Description Default
max_bytes Maximum size of the buffer in bytes 20480
buffer_at_verbosity Minimum log level to buffer (more verbose levels are also buffered) DEBUG
flush_on_error_log Whether to flush buffer when an error is logged True

When log buffering is enabled, you can now pass a new opt-in flush_buffer_on_uncaught_error flag to the inject_lambda_context() decorator. When enabled, 1/ we'll intercept any raised exception, 2/ flush the buffer, and 3/ re-raise your original exception. This enables you to have detailed logs from your application when you need them the most.

buffer3

For detailed explanations with diagrams, please refer to our comprehensive documentation.

Buffering FAQs

Q: Does the buffer persist across Lambda invocations?
A: No, each Lambda invocation has its own buffer. The buffer is initialized when the Lambda function is invoked and is cleared after the function execution completes or when flushed manually.

Q: Are my logs buffered during cold starts?
A: No. We never buffer logs during cold starts to ensure all logs from this phase are immediately available for debugging.

Q: How can I prevent log buffering from consuming excessive memory?
A: You can limit the size of the buffer by setting the max_bytes option in the LoggerBufferConfig constructor parameter. This will ensure that the buffer does not grow indefinitely and consume excessive memory.

Q: What happens if the log buffer reaches its maximum size?
A: Older logs are removed from the buffer to make room for new logs. This means that if the buffer is full, you may lose some logs if they are not flushed before the buffer reaches its maximum size. When this happens, we emit a warning when flushing the buffer to indicate that some logs have been dropped.

Q: What timestamp is used when I flush the logs?
A: The timestamp preserves the original time when the log record was created. If you create a log record at 11:00:10 and flush it at 11:00:25, the log line will retain its original timestamp of 11:00:10.

Q: What happens if I try to add a log line that is bigger than max buffer size?
A: The log will be emitted directly to standard output and not buffered. When this happens, we emit a warning to indicate that the log line was too big to be buffered.

Q: What happens if Lambda times out without flushing the buffer?
A: Logs that are still in the buffer will be lost. If you are using the log buffer to log asynchronously, you should ensure that the buffer is flushed before the Lambda function times out. You can do this by calling the logger.flush_buffer() method at the end of your Lambda function.

Q: Do child loggers inherit the buffer?
A: No, child loggers do not inherit the buffer from their parent logger but only the buffer configuration. This means that if you create a child logger, it will have its own buffer and will not share the buffer with the parent logger.

Changes

  • refactor(tracer): fix capture_lambda_handler return type annotation (#6197) by @tiagohconte

🌟New features and non-breaking changes

📜 Documentation updates

  • docs(layer): Fix SSM parameter name for looking up layer ARN (#6221) by @speshak

🐛 Bug and hot fixes

🔧 Maintenance

This release was made possible by the following contributors:

@dependabot[bot], @github-actions[bot], @leandrodamascena, @speshak, @tiagohconte, dependabot[bot] and github-actions[bot]

Check out latest releases or
releases around aws-powertools/powertools-lambda-python v3.8.0

Don't miss a new powertools-lambda-python release

NewReleases is sending notifications on new releases.

Get notifications