Summary
This release simplifies data transformation with Amazon Kinesis Data Firehose, and handling secret rotation events from Amazon Secrets Manager.
🌟 Huge welcome to our new contributor @TonySherman. Tony documented how to use Event Handler with micro Lambda functions.
Data transformation
When using Kinesis Firehose, you can use a Lambda function to perform data transformation. For each transformed record, you can choose to either:
- A) Put them back to the delivery stream (default)
- B) Drop them so consumers don't receive them (e.g., data validation)
- C) Indicate a record failed data transformation and should be retried
To make this process easier, you can now use KinesisFirehoseDataTransformationResponse and serialization functions to quickly encode payloads into base64 data for the stream.
Example where you might want to drop unwanted records from the stream.
from json import JSONDecodeError
from typing import Dict
from aws_lambda_powertools.utilities.data_classes import (
KinesisFirehoseDataTransformationRecord,
KinesisFirehoseDataTransformationResponse,
KinesisFirehoseEvent,
event_source,
)
from aws_lambda_powertools.utilities.serialization import base64_from_json
from aws_lambda_powertools.utilities.typing import LambdaContext
@event_source(data_class=KinesisFirehoseEvent)
def lambda_handler(event: KinesisFirehoseEvent, context: LambdaContext):
result = KinesisFirehoseDataTransformationResponse()
for record in event.records:
try:
payload: Dict = record.data_as_json # decodes and deserialize base64 JSON string
## generate data to return
transformed_data = {"tool_used": "powertools_dataclass", "original_payload": payload}
processed_record = KinesisFirehoseDataTransformationRecord(
record_id=record.record_id,
data=base64_from_json(transformed_data),
)
except JSONDecodeError:
# our producers ingest JSON payloads only; drop malformed records from the stream
processed_record = KinesisFirehoseDataTransformationRecord(
record_id=record.record_id,
data=record.data,
result="Dropped",
)
result.add_record(processed_record)
# return transformed records
return result.asdict()Rotating secrets
When rotating secrets with Secrets Manager, it invokes your Lambda function in four potential steps:
createSecret. Create a new version of the secret.setSecret. Change the credentials in the database or service.testSecret. Test the new secret version.finishSecret. Finish the rotation.
You can now use SecretsManagerEvent to more easily access the event structure, and combine Parameters to get secrets to perform secret operations.
from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.data_classes import SecretsManagerEvent, event_source
secrets_provider = parameters.SecretsProvider()
@event_source(data_class=SecretsManagerEvent)
def lambda_handler(event: SecretsManagerEvent, context):
# Getting secret value using Parameter utility
# See https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/
secret = secrets_provider.get(event.secret_id, VersionId=event.version_id, VersionStage="AWSCURRENT")
if event.step == "setSecret":
# Perform any secret rotation logic, e.g., change DB password
# Check more examples: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas
print("Rotating secret...")
return secretChanges
🌟New features and non-breaking changes
- feat(event_source): add Kinesis Firehose Data Transformation data class (#3029) by @roger-zhangg
- feat(event_sources): add Secrets Manager secret rotation event (#3061) by @roger-zhangg
📜 Documentation updates
- docs(event_handler): fix typing in micro function example (#3098) by @leandrodamascena
- docs(we-made-this): fix broken Twitch video embeds (#3096) by @leandrodamascena
- docs(event_handler): add micro function examples (#3056) by @TonySherman
- feat(event_source): add Kinesis Firehose Data Transformation data class (#3029) by @roger-zhangg
- feat(event_sources): add Secrets Manager secret rotation event (#3061) by @roger-zhangg
- chore(deps): bump squidfunk/mkdocs-material from
dd1770ctoc4890abin /docs (#3078) by @dependabot
🔧 Maintenance
- chore(deps-dev): bump aws-cdk from 2.96.0 to 2.96.1 (#3093) by @dependabot
- chore(typing): move backwards compat types to shared types (#3092) by @heitorlessa
- refactor(parameters): BaseProvider._get to also support Dict (#3090) by @leandrodamascena
- chore(deps): bump docker/setup-qemu-action from 2.2.0 to 3.0.0 (#3081) by @dependabot
- chore(deps): bump docker/setup-buildx-action from 2.10.0 to 3.0.0 (#3083) by @dependabot
- chore(deps-dev): bump cfn-lint from 0.79.10 to 0.79.11 (#3088) by @dependabot
- chore(deps-dev): bump sentry-sdk from 1.30.0 to 1.31.0 (#3086) by @dependabot
- chore(deps-dev): bump aws-cdk from 2.95.1 to 2.96.0 (#3087) by @dependabot
- chore(deps-dev): bump the boto-typing group with 1 update (#3085) by @dependabot
- chore(deps-dev): bump ruff from 0.0.288 to 0.0.289 (#3080) by @dependabot
- chore(deps-dev): bump cfn-lint from 0.79.9 to 0.79.10 (#3077) by @dependabot
- chore(deps-dev): bump hvac from 1.2.0 to 1.2.1 (#3075) by @dependabot
- chore(deps): bump squidfunk/mkdocs-material from
dd1770ctoc4890abin /docs (#3078) by @dependabot - chore(deps-dev): bump ruff from 0.0.287 to 0.0.288 (#3076) by @dependabot
- chore(deps-dev): bump aws-cdk from 2.95.0 to 2.95.1 (#3074) by @dependabot
- chore(deps): bump actions/dependency-review-action from 3.0.8 to 3.1.0 (#3071) by @dependabot
- chore(deps-dev): bump aws-cdk from 2.94.0 to 2.95.0 (#3070) by @dependabot
- chore(automation): remove previous labels when PR is updated (#3066) by @sthulb
This release was made possible by the following contributors:
@TonySherman, @dependabot, @dependabot[bot], @github-actions, @github-actions[bot], @heitorlessa, @leandrodamascena, @roger-zhangg and @sthulb