Summary
This release introduces a brand new streaming utility to handle datasets larger than the available memory.
This addresses scenarios where you want to process Amazon S3 objects as seekable data streams, with minimal memory consumption.
Built-in data transformations
We've added support for popular data transformations to decompress and deserialize data (gzip, CSV and ZIP). This way you can transform the data while streaming, and only use the minimum amount of memory necessary.
Bring your own data transformation
You can also create your own data transformations and add them to the pipeline. For example, here's how to use the ijson library to parse JSON as a data stream:
We plan to support more data transformations in the future too!
🌟 Would you like to see a specific data transformation supported in Powertools? Drop us a line in the python channel of our Discord.
Changes
🌟New features and non-breaking changes
- feat(streaming): add new s3 streaming utility (#1719) by @rubenfonseca
📜 Documentation updates
- docs(idempotency): fix register_lambda_context order (#1747) by @heitorlessa
- feat(streaming): add new s3 streaming utility (#1719) by @rubenfonseca
🔧 Maintenance
- feat(streaming): add new s3 streaming utility (#1719) by @rubenfonseca
- chore(deps-dev): bump mypy-boto3-cloudformation from 1.26.0.post1 to 1.26.11.post1 (#1746) by @dependabot
- chore(deps-dev): bump mypy-boto3-lambda from 1.26.0.post1 to 1.26.12 (#1742) by @dependabot
- chore(deps-dev): bump aws-cdk-lib from 2.50.0 to 2.51.1 (#1741) by @dependabot
- chore(deps-dev): bump mypy-boto3-dynamodb from 1.26.0.post1 to 1.26.13.post16 (#1743) by @dependabot
- chore(deps-dev): bump mypy-boto3-secretsmanager from 1.26.0.post1 to 1.26.12 (#1744) by @dependabot
- chore(deps-dev): bump mypy-boto3-ssm from 1.26.4 to 1.26.11.post1 (#1740) by @dependabot
- chore(deps-dev): bump types-requests from 2.28.11.4 to 2.28.11.5 (#1729) by @dependabot
- chore(deps): bump aws-xray-sdk from 2.10.0 to 2.11.0 (#1730) by @dependabot
- chore(deps-dev): bump mkdocs-material from 8.5.9 to 8.5.10 (#1731) by @dependabot
This release was made possible by the following contributors:
@dependabot, @dependabot[bot], @heitorlessa, @rubenfonseca and Release bot