aws-powertools/powertools-lambda-python v1.21.0 on GitHub

Summary

After some vacation period, we're back with a new minor release with major features:

Bring your own boto3 sessions for cross-account operations & snapshot testing
New features on Feature Flags
Idempotency unit testing made easier
JSON Schema Validation utility contains new data elements to more easily construct a validation error
New utility: we're now exposing our internal custom JMESPath Functions so you can easily decode and deserialize JSON objects found in various formats within Lambda Event Sources.

New Contributors

I'd like to personally thank our new contributors to the project :)

@Tankanow made their first contribution in #697
@DanyC97 made their first contribution in #716
@gwlester made their first contribution in #710

Detailed changes

Boto3 sessions

You can now pass in your own boto3 session when using Parameters, Batch, and Idempotency.

This is helpful in two typical scenarios: 1/ You want to run an operation in another account like fetching secrets/parameters somewhere else, 2/ Use snapshot testing tools like Placebo that will replay session data that happened earlier when doing unit testing.

from aws_lambda_powertools.utilities import parameters
import boto3

boto3_session = boto3.session.Session()
ssm_provider = parameters.SSMProvider(boto3_session=boto3_session)

def handler(event, context):
    # Retrieve a single parameter
    value = ssm_provider.get("/my/parameter")
    ...

Feature flags

There's been three main improvements in Feature flags utility as part of this release: New rule conditions, Bring your own Logger for debugging, and Getting a copy of fetched configuration from the store

New rule conditions

You can now use the following new rule conditions to evaluate your feature flags for inequality, comparison, and more explicit contains logic, where a is the key and b is the value passed as a context input for evaluation:

Action	Equivalent expression
KEY_GREATER_THAN_VALUE	`lambda a, b: a > b`
KEY_GREATER_THAN_OR_EQUAL_VALUE	`lambda a, b: a >= b`
KEY_LESS_THAN_VALUE	`lambda a, b: a < b`
KEY_LESS_THAN_OR_EQUAL_VALUE	`lambda a, b: a <= b`
KEY_IN_VALUE	`lambda a, b: a in b`
KEY_NOT_IN_VALUE	`lambda a, b: a not in b`
VALUE_IN_KEY	`lambda a, b: b in a`
VALUE_NOT_IN_KEY	`lambda a, b: b not in a`

Example

Feature flag schema

{
    "premium_features": {
        "default": false,
        "rules": {
            "customer tier equals premium": {
                "when_match": true,
                "conditions": [
                    {
                        "action": "VALUE_IN_KEY",
                        "key": "groups",
                        "value": "PAID_PREMIUM",
                    }
                ]
            }
        }
    },
    "ten_percent_off_campaign": {
        "default": false
    }
}

App

from aws_lambda_powertools.utilities.feature_flags import FeatureFlags, AppConfigStore

app_config = AppConfigStore(
    environment="dev",
    application="product-catalogue",
    name="features"
)

feature_flags = FeatureFlags(store=app_config)

def lambda_handler(event, context):
    # groups: ["FREE_TIER", "PAID_BASIC", "PAID_PREMIUM"]
    ctx={"tenant_id": "6", "username": "a", "groups": event.get("groups", [])}

    # Evaluate whether customer's tier has access to premium features
    # based on `has_premium_features` rules
    has_premium_features: bool = feature_flags.evaluate(name="premium_features",
                                                        context=ctx, default=False)
    if has_premium_features:
        # enable premium features
        ...

Accessing raw configuration fetched

Previously, if you were using a single application configuration and a feature schema in a single AppConfig key, we would only use the feature flags schema and discard the rest.

You can now access the raw configuration with a new property get_raw_configuration within AppConfig Store:

from aws_lambda_powertools.utilities.feature_flags import FeatureFlags, AppConfigStore

app_config = AppConfigStore(
    environment="dev",
    application="product-catalogue",
    name="configuration",
    envelope = "feature_flags"
)

feature_flags = FeatureFlags(store=app_config)

config = app_config.get_raw_configuration

Unit testing idempotency

We have improved how you can unit test your code when using @idempotent and @idempotent_function decorators.

You can now disable all interactions with the idempotence store using POWERTOOLS_IDEMPOTENCY_DISABLED environment variable, and monkeypatch the DynamoDB resource client Idempotency utility uses if you wish to either use DynamoDB Local or mock all I/O operations.

import boto3

import app

def test_idempotent_lambda():
    # Create our own Table resource using the endpoint for our DynamoDB Local instance
    resource = boto3.resource("dynamodb", endpoint_url='http://localhost:8000')
    table = resource.Table(app.persistence_layer.table_name)
    app.persistence_layer.table = table

    result = app.handler({'testkey': 'testvalue'}, {})
    assert result['payment_id'] == 12345

New data elements for JSON Schema validation errors

When validating input/output with the Validator, you can now access new properties in SchemaValidationError to more easily construct your custom errors based on what went wrong.

Property	Type	Description
`message`	str	Powertools formatted error message
`validation_message`	str, optional	Containing human-readable information what is wrong, e.g. `data.property[index] must be smaller than or equal to 42`
`name`	str, optional	name of a path in the data structure, e.g. `data.property[index]`
`path`	List, optional	`path` as an array in the data structure, e.g. `['data', 'property', 'index']`
`value`	Any, optional	The invalid value, e.g. `{"message": "hello world"}`
`definition`	Any, optional	JSON Schema definition
`rule`	str, optional	`rule` which the `data` is breaking (e.g. `maximum`, `required`)
`rule_definition`	Any, optional	The specific rule `definition` (e.g. `42`, `['message', 'username']`)

Sample

from aws_lambda_powertools.utilities.validation import validate
from aws_lambda_powertools.utilities.validation.exceptions import SchemaValidationError

import schemas

def handler(event, context):
    try:
        validate(event=event, schema=schemas.INPUT)
    except SchemaValidationError as e:
        message = "data must contain ['message', 'username'] properties"

        assert str(e.value) == e.value.message
        assert e.value.validation_message == message
        assert e.value.name == "data"
        assert e.value.path is not None
        assert e.value.value == data
        assert e.value.definition == schema
        assert e.value.rule == "required"
        assert e.value.rule_definition == schema.get("required")

        raise

    return event

New JMESPath Powertools functions

Last but not least, as part of a documentation revamp in Idempotency by @walmsles, we're now exposing an internal feature used by many Lambda Powertools utilities which is the ability to extract and decode JSON objects.

You can now use JMESPath (JSON Query language) Lambda Powertools functions to easily decode and deserialize JSON often found as compressed (Kinesis, CloudWatch Logs, etc), as strings (SNS, SQS, EventBridge, API Gateway, etc), or as base64 (Kinesis).

We're exposing three custom JMESPath functions you can use such as powertools_json, powertools_base64, powertools_base64_gzip, and a new standalone function that will use JMESPath to search and extract the data you want called extract_data_from_envelope.

Sample

from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope
from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope="powertools_json(body)")
    customer = payload.get("customerId")  # now deserialized
    ...

SNS sample using built-in envelope

from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, envelopes
from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope=envelopes.SNS)
    customer = payload.get("customerId")  # now deserialized
    ...

Sample event

{
    "Records": [
        {
            "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
            "receiptHandle": "MessageReceiptHandle",
            "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\",\"booking\":{\"id\":\"5b2c4803-330b-42b7-811a-c68689425de1\",\"reference\":\"ySz7oA\",\"outboundFlightId\":\"20c0d2f2-56a3-4068-bf20-ff7703db552d\"},\"payment\":{\"receipt\":\"https:\/\/pay.stripe.com\/receipts\/acct_1Dvn7pF4aIiftV70\/ch_3JTC14F4aIiftV700iFq2CHB\/rcpt_K7QsrFln9FgFnzUuBIiNdkkRYGxUL0X\",\"amount\":100}}",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1523232000000",
                "SenderId": "123456789012",
                "ApproximateFirstReceiveTimestamp": "1523232000001"
            },
            "messageAttributes": {},
            "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
            "awsRegion": "us-east-1"
        }
    ]
}

Changes

🌟New features and non-breaking changes

feat: expose jmespath powertools functions (#736) by @heitorlessa
feat(feature-flags): Added inequality conditions (#721) by @gwlester
feat(feature-flags): Bring your own logger for debug (#709) by @gwlester
feat(feature-flags): improve "IN/NOT_IN"; new rule actions (#710) by @gwlester
feat(idempotency): makes customers unit testing easier (#719) by @cakepietoast
feat(feature-flags): get_raw_configuration property in Store (#720) by @heitorlessa
feat: boto3 sessions in batch, parameters & idempotency (#717) by @cakepietoast
feat(validator): include missing data elements from a validation error (#686) by @michaelbrewer

🌟 Minor Changes

fix(idempotency): use ExpressionAttributeNames in DynamoDBPersistenceLayer _put_record (#697) by @Tankanow

📜 Documentation updates

chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 (#730) by @dependabot
chore(deps-dev): bump coverage from 5.5 to 6.0 (#732) by @dependabot
docs(parser): fix incorrect import in root_validator example (#735) by @heitorlessa
docs: Terraform reference for SAR Lambda Layer (#716) by @DanyC97
docs(idempotency): fix misleading idempotent examples (#661) by @walmsles
docs(event-handler): document catch-all routes (#705) by @heitorlessa
refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse (#643) by @michaelbrewer
fix(data-classes): use correct asdict funciton (#666) by @michaelbrewer

🐛 Bug and hot fixes

fix(feature-flags): rules should evaluate with an AND op (#724) by @risenberg-cyberark
fix(logger): push extra keys to the end (#722) by @heitorlessa
fix(mypy): a few return types, type signatures, and untyped areas (#718) by @heitorlessa

🔧 Maintenance

chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 (#730) by @dependabot
chore(deps-dev): bump coverage from 5.5 to 6.0 (#732) by @dependabot
chore(deps): bump boto3 from 1.18.51 to 1.18.54 (#733) by @dependabot
chore(deps-dev): bump flake8-bugbear from 21.9.1 to 21.9.2 (#712) by @dependabot
chore(deps): bump boto3 from 1.18.49 to 1.18.51 (#713) by @dependabot
chore(deps): bump boto3 from 1.18.41 to 1.18.49 (#703) by @dependabot
chore(deps): bump codecov/codecov-action from 2.0.2 to 2.1.0 (#675) by @dependabot
chore(deps-dev): bump mkdocs-material from 7.2.8 to 7.3.0 (#695) by @dependabot
chore(deps-dev): bump mkdocs-material from 7.2.6 to 7.2.8 (#682) by @dependabot
chore(deps-dev): bump flake8-bugbear from 21.4.3 to 21.9.1 (#676) by @dependabot
chore(deps): bump boto3 from 1.18.38 to 1.18.41 (#677) by @dependabot
chore(deps-dev): bump radon from 4.5.2 to 5.1.0 (#673) by @dependabot
chore(deps): bump boto3 from 1.18.32 to 1.18.38 (#671) by @dependabot
refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse (#643) by @michaelbrewer
chore(deps-dev): bump xenon from 0.7.3 to 0.8.0 (#669) by @dependabot

This release was made possible by the following contributors:

@DanyC97, @Tankanow, @cakepietoast, @dependabot, @dependabot[bot], @gwlester, @heitorlessa, @michaelbrewer, @risenberg-cyberark and @walmsles

Full Changelog: v1.20.2...v1.21.0