pypi aws-lambda-powertools 2.12.0
v2.12.0
on Python PyPI

latest releases: 3.0.2.dev1, 3.0.1.dev3, 3.0.1.dev2...
17 months ago

Summary

This release improves the authoring experience for Batch processing (-boilerplate), fix a bug between Batch and Pydantic integration for poison pills, and several documentation and static typing improvements!

⭐ Huge thanks to new contributors: @michael-k (docs) and @LuckIlNe (Batch pydantic bugfix)

New Batch experience

No more decorators for Batch Processing

Docs: Amazon SQS, Amazon Kinesis Data Streams, Amazon DynamoDB Streams

Batch feature is generally used for building more resilient data pipelines ranging from ETL to Change Data Capture to AWS automation.

This release significantly removes boilerplate and makes it less error prone to author batch processors. You can now use process_partial_response and async_process_partial_response within your Lambda handler to kick off processing, handle failures and return the expected response for partial failures in Lambda.

Before

image

After

New Batch experience

Addressing poison pill issue with Batch and Pydantic model validation

Docs

Pydantic provides parsing and deep data validation mechanisms. It's commonly used with Batch feature to author and validate data models.

Thanks to @LuckIlNe, we now correctly handle malformed messages that fail validation early (poison pills), thus not breaking the partial failure mechanism. Previously, any record that doesn't conform with an early model validation would not fail gracefully.

Documentation improvements

Idempotency

Docs: Idempotency new diagrams, terminology, and Idempotency record expiration vs DynamoDB time-to-live (TTL).

Thanks to a discussion with @pmarko1711, we revamped the Idempotency documentation to include more sequence diagrams to visually explain the different idempotent scenarios. We also clarified terminologies like Idempotency Record, and a confusion between Idempotency Record Expiration and DynamoDB TTL.

The latter can be used as a fine-grained concurrency mechanism.

image

Batch

Docs

Pydantic integration examples now use the new Json model from Pydantic to auto-deserialize JSON strings into Pydantic models. This removes excessive boilerplate when working with SQS and Kinesis Data Streams in Batch.

All examples now recommend use of process_partial_response over the legacy decorator, including a new question in the FAQ section to clarify them.

Static typing improvements in Parser

When using Mypy for applications using Pydantic models provided by Parser, you couldn't use nested models for SQS, Kinesis, and DynamoDB Streams.

This quality of life improvement also allows you to use higher-level models like Json to auto-deserialize JSON strings in common Lambda Event Source fields like SQS.body. 

from aws_lambda_powertools.utilities.parser import BaseModel, validator
from aws_lambda_powertools.utilities.parser.models import (
    DynamoDBStreamChangedRecordModel,
    DynamoDBStreamRecordModel,
    KinesisDataStreamRecord,
    KinesisDataStreamRecordPayload,
    SqsRecordModel,
)
from aws_lambda_powertools.utilities.parser.types import Json, Literal


class Order(BaseModel):
    item: dict


# SQS example
class OrderSqs(SqsRecordModel):
    body: Json[Order]


# Kinesis example
class OrderKinesisPayloadRecord(KinesisDataStreamRecordPayload):
    data: Json[Order]


class OrderKinesisRecord(KinesisDataStreamRecord):
    kinesis: OrderKinesisPayloadRecord

# DynamoDB Streams example
class OrderDynamoDBChangeRecord(DynamoDBStreamChangedRecordModel):
    NewImage: Optional[OrderDynamoDB]
    OldImage: Optional[OrderDynamoDB]


class OrderDynamoDBRecord(DynamoDBStreamRecordModel):
    dynamodb: OrderDynamoDBChangeRecord

Changes

🌟New features and non-breaking changes

  • feat(batch): reduce boilerplate with process_partial_response (#2090) by @heitorlessa
  • feat(idempotency): allow custom sdk clients in DynamoDBPersistenceLayer (#2087) by @heitorlessa

📜 Documentation updates

🐛 Bug and hot fixes

  • fix(batch): handle early validation errors for pydantic models (poison pill) #2091 (#2099) by @LuckIlNe

🔧 Maintenance

This release was made possible by the following contributors:

@LuckIlNe, @dependabot, @dependabot[bot], @heitorlessa, @leandrodamascena, @michael-k and Release bot

Check out latest releases or
releases around aws-lambda-powertools 2.12.0

Don't miss a new aws-lambda-powertools release

NewReleases is sending notifications on new releases.

Get notifications