This is our second major release with many new features, enhancements, and improved parity for more extensive integration across the AWS tooling ecosystem. With LocalStack 2.0, we have significantly optimized the internals of the platform and moved to new service implementations, images, and internal toolings to make it easy for developers to build & test their cloud applications locally!
Several of these changes require a migration, and we have done our best to make the migration as smooth as possible. Please find more in the How to migrate section.
Summary
Community
- LocalStack community image (
localstack/localstack
) no longer contains LocalStack Pro (localstack/localstack-pro
) code, effectively cutting the community image size in half - Removal of
localstack-full
andlocalstack-light
since they have become obsolete thanks to our new dependency packaging system (simply uselocalstack
orlocalstack-pro
) - Completely new AWS Lambda provider with improved parity and performance
- Completely new S3 provider with improved parity and features
- Improved supports for community cloud pods
- Simplified host configuration and docker networking
- Internal endpoints have moved into
/_localstack
and/_aws
Pro
- Cross-service IAM enforcement
- Completely new snapshot persistence (
PERSISTENCE=1
) mechanism with more flexible load and save strategies - New simplified container setup for AWS Big Data Technologies (Athena, Glue, EMR, etc)
How to migrate
General
- ⚠️ If you are using LocalStack Pro with a
LOCALSTACK_API_KEY
, please change your container configuration to uselocalstack/localstack-pro
. More details can be found here: #7854 - LocalStack Pro will fail to start if the LocalStack Key activation fails. You can disable this behavior by setting
ACTIVATE_PRO=0
.
Networking
-
We are unifying the variables
EDGE_BIND_HOST
,EDGE_PORT
andEDGE_PORT_HTTP
intoGATEWAY_LISTEN
, which will allow configuration of the addresses and ports the LocalStack process listens on. It takes the form<ip address>[:port][,<ip address>:<port>...]
where multiple values can be given, separated by commas. LocalStack will listen on all interfaces, but will ask for superuser permission for privileged ports. We will still acceptEDGE_BIND_HOST
,EDGE_PORT
, andEDGE_PORT_HTTP
during a deprecation period following the release of v2, but they will not be used by LocalStack for configuration at some point in the future. Please migrate your use ofEDGE_BIND_HOST
,EDGE_PORT
orEDGE_PORT_HTTP
toGATEWAY_LISTEN
. If you experience issues, please try the deprecated configuration variables. -
For example, if you previously ran LocalStack with the command
EDGE_BIND_HOST=0.0.0.0 EDGE_PORT=5000 EDGE_PORT_HTTP=5001 localstack start
or used these configuration variables in yourdocker-compose.yml
, please replace it withGATEWAY_LISTEN=0.0.0:5000,0.0.0.0:5001 localstack start
. -
We are unifying the variables
HOSTNAME_EXTERNAL
andLOCALSTACK_HOSTNAME
intoLOCALSTACK_HOST
, which will allow configuration of hostnames returned by LocalStack in a more consistent way. If provided, this variable will be used in preference toHOSTNAME_EXTERNAL
andLOCALSTACK_HOSTNAME
, and will be used as the hostname returned in URLs for created resources such as OpenSearch clusters, SQS queues, or RDS databases. We will still acceptHOSTNAME_EXTERNAL
orLOCALSTACK_HOSTAME
during a deprecation period following the release of v2, but they will not be used by LocalStack for configuration at some point in the future. Please migrate your use ofHOSTNAME_EXTERNAL
orLOCALSTACK_HOSTNAME
toLOCALSTACK_HOST
. If you experience issues, please try the deprecated configuration variables. -
For example, if you previously ran LocalStack with the command
HOSTNAME_EXTERNAL=<hostname> localstack start
or usedHOSTNAME_EXTERNAL
in yourdocker-compose.yml
, please replace it withLOCALSTACK_HOST=<hostname>[:port]
. -
Starting LocalStack Pro using the CLI will no longer publish port
53
to the host if it is already bound by some other service likesystemd-resolved
on Linux, ormDNSResponder
on macOS.
Persistence & cloud pods
Using LocalStack Pro with PERSISTENCE=1
(which we term snapshot-based persistence) now has two behavioral changes with respect to when data is restored and saved. More information can be found in our docs.
- Restoring state/loading a snapshot (load): Previously, snapshots could only be loaded from disk per-service when services were first initialized, lazy-loading the state the first time a service was used. With the new persistence mechanism introduced in v2, persistent data can be loaded on LocalStack startup. You can configure this behavior by setting
SNAPSHOT_LOAD_STRATEGY
toon_request
oron_startup
. The default strategy currently is stillon_request
until we iron out some quirks. - Saving state/creating a snapshot (save): Previously, a snapshot was created for a particular service on each request. This was mainly to protect against potential data loss if LocalStack would suddenly terminate. We found that this approach leads to several problems, specifically related to concurrency and performance. An alternative approach we have introduced is to store snapshots on LocalStack shutdown, which produces no performance overhead during runtime, but will not protect you against data loss if LocalStack does not terminate correctly. The default strategy is on a scheduled basis, specifically, we take snapshots of services that have changed every 15 seconds. You can configure this behavior by setting
SNAPSHOT_SAVE_STRATEGY
toon_request
,on_shutdown
orscheduled
, respectively.
The layout of Cloud Pods has been changed and cloud pods created with v1.x.x may be incompatible with v2.0. Please re-create your cloud pods with the latest LocalStack version.
Lambda
Lambda has been completely re-written and the current documentation can be found here. There are several behavioral changes to lambda that mostly affect users of the LOCAL
executor mode. If you run into issues, you can, for now, use the legacy provider using PROVIDER_OVERRIDE_LAMBDA=legacy
.
- Mounting the Docker socket
"/var/run/docker.sock:/var/run/docker.sock"
into the LocalStack container is now required to run Lambdas. - The default behavior is now equivalent to the old
docker-reuse
executor, there are no longer multiple options. - Local executor: with
LAMBDA_EXECUTOR=local
, lambdas were executed within the LocalStack container. This was mostly used as a fallback if the docker socket could not be mounted into the LocalStack container. If you cannot mount the docker socket or don’t have an externalDOCKER_HOST
available, we provide a new way to run Lambda functions via static worker containers that can be configured manually. The only requirement here is connectivity between the static worker and the LocalStack instance. Predefined workers will be available via images hosted by us: e.g.localstack/lambda-worker:python3.9
. - There are several new ways to configure the lambda provider variables that can be found in our docs, which we will update in the upcoming weeks.
- We have migrated our Lambda Docker images from lambci to use the official AWS images, which will now be pulled by default from
public.ecr.aws/lambda/
- Functions were previously created synchronously, i.e.,
CreateFunction
calls would block until the function state was “Active”. Functions are now created asynchronously, and their state will move from “Pending” to “Active”, which you can check with aGetFunction
call. This behavior can be disabled withLAMBDA_SYNCHRONOUS_CREATE=1
(not recommended). - Stricter input validation: previously, when creating functions, the
Role
attribute could be any value, and many of our examples included something likeawslocal create-function --role r1
. This will no longer work, as roles now have to have an ARN format. We do not t validate whether the role exists, so you can use any ARN. - Hot reloading: previously, the magic S3 bucket name for lambda code hot reloading was
__local__
, which was changed tohot-reload
. Please change your deployment configs accordingly, or use theBUCKET_MARKER_LOCAL
configuration to customize the value. - Lambda in LocalStack Pro supports “transparent endpoint injection”, which allows Lambdas to resolve domains like
s3.amazonaws.com
to the LocalStack container. Previously, this was based on patching SDKs, but is now completely DNS-based, and will be disabled ifDNS_ADDRESS=0
is set.
S3
S3 has been completely re-written and its behavior is aligned with AWS. Users should be mostly unaffected, but may experience some breakage depending on previous behavior that was not aligned with AWS. Should you run into problems, you can activate the old provider with PROVIDER_OVERRIDE_S3=legacy
.
Big Data container
- The mono container mode (
BIGDATA_MONO_CONTAINER=1
) for BigData services (Glue, EMR, Athena, etc) is now the default, the previous implementation with a separate sidecarlocalstack_bigdata
container is deprecated and will be removed in a future release. - Please note that some of the required dependencies (Spark, Hive, etc) are lazily downloaded and installed at runtime, which increases the processing time on first load. The libraries are cached locally in the
var_libs
directory - please make sure to properly mount the LocalStack volume into your container. We also provide a separatelocalstack/localstack-pro:latest-bigdata
BigData mono container image which has the default dependencies pre-installed.
Other notable changes
- supervisord is no longer used to start LocalStack in the Docker container, instead we use our own init program.
- LocalStack no longer automatically restarts on failure
- /var/lib/localstack/logs/localstack_infra.log was removed
REQUIRE_PRO
has been replaced withACTIVATE_PRO
, which sets whether or not LocalStack pro should be activated when using the pro image.
New deprecations
- With all new provider implementations, some configuration variables have been deprecated with this release and will be removed in future releases:
HOSTNAME_EXTERNAL
- This configuration will be migrated toLOCALSTACK_HOST
LOCALSTACK_HOSTNAME
- This configuration will be migrated toLOCALSTACK_HOST
EDGE_BIND_HOST
- This configuration will be migrated toGATEWAY_LISTEN
EDGE_PORT
- This configuration will be migrated toGATEWAY_LISTEN
EDGE_PORT_HTTP
- This configuration will be migrated toGATEWAY_LISTEN
LAMBDA_EXECUTOR
- This configuration is obsolete with the new lambda providerLAMBDA_STAY_OPEN_MODE
- Stay open mode is the default behavior in the new lambda providerLAMBDA_REMOTE_DOCKER
- The new lambda provider copies zip files by default and automatically configures hot reloadingLAMBDA_CODE_EXTRACT_TIME
- Function creation now happens asynchronously in the new lambda providerLAMBDA_CONTAINER_REGISTRY
- The new lambda provider usesLAMBDA_RUNTIME_IMAGE_MAPPING
insteadLAMBDA_XRAY_INIT
- The X-Ray daemon is always initialized in the new lambda providerLAMBDA_DOCKER_DNS
- This feature is currently not supported in the new lambda providerHOSTNAME_FROM_LAMBDA
- This feature is currently not supported in the new lambda providerLAMBDA_FALLBACK_URL
- This feature is not supported in the new lambda providerLAMBDA_FORWARD_URL
- This feature is not supported in the new lambda provider
Deprecation removals
- Deprecation removals:
LEGACY_EDGE_PROXY
has been removedLEGACY_DIRECTORIES
has been removed, please migrate to the current Filesystem LayoutLEGACY_IAM_ENFORCEMENT
has been removed/docker-entrypoint-initaws.d
for initializing LocalStack has been removed, please migrate to the modern Initialization Hooks- please find other minor deprecation notices in previous release notes.
What's Changed
- Update ASF APIs by @localstack-bot in #7669
- [SFN] basic version of stepfunctions v2 by @MEPalma in #7464
- Avoid installing amazon_kclpy>=2.1.0 by @dominikschubert in #7676
- Fix CFn Route53 Record Set TTL by @pinzon in #7677
- use lifecycle hook for registering lambda function URL routes by @thrau in #7668
- add proper shutdown for the hypercorn server that serves the ASF gateway by @thrau in #7609
- fix encoding issues in router, proxy, partition rewriting by @alexrashed in #7682
- Release 1.4.0 by @viren-nadkarni in #7670
- fix SNS PublishBatch modifying batch context for all subscribers after filtering by @bentsku in #7674
- Use more informative container names for spawned Lambda containers by @dominikschubert in #7667
- Change behavior of VtlTemplate to modify variable values in-place by @whummer in #7687
- refactor state visitor code and lifecycle hooks by @thrau in #7650
- fix coveralls integration by @alexrashed in #7694
- Ignore dangling images in localstack update by @dominikschubert in #7691
- Allow configuration of docker image used for port checks by @dominikschubert in #7690
- guard process information pytest_unconfigure hook from exceptions by @thrau in #7697
- refactor community ext imports by @alexrashed in #7693
- Add support for route.destination-cidr-block filter in DescribeRouteTables by @dominikschubert in #7700
- Fix docker runtime executor flags by @joe4dev in #7675
- fix internet gateway attrs by @pinzon in #7679
- Feature CFn Macros/Transformations by @pinzon in #7509
- Install postgres dependencies on runner for redshift by @dominikschubert in #7708
- Update ASF APIs by @localstack-bot in #7715
- Use token in GitHubReleaseInstaller for increased rate-limits by @dominikschubert in #7716
- extend persistence api by @thrau in #7713
- Allow container port binding for privileged ports by @dfangl in #7719
- fix cfn stack events resource types by @pinzon in #7599
- Don't require CAPABILITY_AUTO_EXPAND for change sets by @dominikschubert in #7718
- Add retry cycles for deleting stack resources by @dominikschubert in #7717
- Re-enable cli container lifecycle test by @joe4dev in #7702
- fix S3 GetObjectAttributes and GetObject to return Checksum if requested by @bentsku in #7704
- fix 'unexpected keyword argument' error by @steffyP in #7686
- add rest api fixture by @calvernaz in #7724
- Check original environment copy for deprecations by @dominikschubert in #7685
- Fix apigateway authorizer path suffix by @calvernaz in #7557
- Add initial support for CFn Fn:Transform intrinsic functions by @whummer in #7711
- [SFN] base implementation of intrinsic functions by @MEPalma in #7526
- [SFN] [IntrinsicFunctions]: snapshot tests, Array functions, String functions, fixes by @MEPalma in #7655
- Gather outdated snapshots by @baermat in #7739
- Fix apigateway path encoding by @joe4dev in #7728
- Refactor authorizer info in API Gateway invocation context by @whummer in #7740
- Improve support for request integration parameters by @calvernaz in #7580
- [SFN] [IntrinsicFunctions] base support for JsonMerge and States.Hash, boolean literals support, snap test by @MEPalma in #7671
- Feat/api gateway dynamodb query by @amitassaraf in #7572
- Enhance parity for API Gateway DynamoDB integrations by @whummer in #7753
- Update ASF APIs by @localstack-bot in #7758
- Enable generic actions and enhance parity for DynamoDB API GW integration by @whummer in #7755
- restructure python distributions by @alexrashed in #7730
- Minimal implementation for continuous backup by @giograno in #7735
- fix event target id validation by @pinzon in #7688
- validate StartingPosition during CreateEventSourceMapping operation by @pinzon in #7696
- improve parity for APIGW Resource by @bentsku in #7738
- Update Kinesis Mock to 0.3.10 by @etspaceman in #7748
- Skip lambda nodejs18 extra cert snapshot verify by @joe4dev in #7765
- [SFN] [IntrinsicFunctions] Base implementation of States.Math functions by @MEPalma in #7689
- [SFN] [IntrinsicFunctions] base implementation Encode-Decode functions by @MEPalma in #7709
- Add lambda model and api updates by @joe4dev in #7762
- Catch Route53 Record Sets without TTL by @QuinnyPig in #7767
- fix SNS storing lambda invocation logs + improve parity by @bentsku in #7771
- improve apigw parity work for Resource proxy path by @bentsku in #7746
- Enhance CloudFormation parity for policy of API Gateway REST APIs by @whummer in #7777
- Update apt repositories when installing system dependencies by @simonrw in #7794
- add runtime hook that runs on pro infra startup by @thrau in #7789
- add cwd parameter to ShellCommandThread by @thrau in #7800
- add persistence API integration for community services by @thrau in #7785
- test: list object version with prefix and delimiters by @macnev2013 in #7760
- Update ASF APIs by @localstack-bot in #7805
- Refactor machine arch name helper by @viren-nadkarni in #7818
- add validation of KMS key state in S3 by @bentsku in #7786
- Add Lambda init debug mode by @dominikschubert in #7809
- Add scenario tests with Fundamental level for Stepfunctions by @dominikschubert in #7793
- Add DELETE to the _localstack/ses/ endpoint by @justinwyer in #7816
- apigateway - refactor store by @bentsku in #7750
- Fix error with threads not having _target attributes by @dfangl in #7829
- refactor kinesis state lifecycle hook by @thrau in #7825
- DDB - persistence plugin and server rework by @giograno in #7827
- Fix multiple AWS::Lambda::Permission for single function by @dominikschubert in #7832
- Lambda runtime parity by @joe4dev in #7824
- fix openapi 3.0 security schemes import by @calvernaz in #7822
- Remove extra slash in SQS path strategy by @baermat in #7837
- refactor SNS store to facilitate lookups by @bentsku in #7803
- fix APIGW import RestAPI version type mismatch by @bentsku in #7787
- Update ASF APIs by @localstack-bot in #7844
- Bump moto ext to 4.1.4.post1 by @viren-nadkarni in #7813
- Fix: Alias name validator by @deepak2431 in #7826
- apigw parity work resource method by @bentsku in #7749
- update s3 tests location in CODEOWNERS by @bentsku in #7845
- New AWS client by @dfangl in #7240
- add logging for basic container information by @thrau in #7855
- ignore scheduled rules when processing put_events by @dominikschubert in #7848
- Fix transformation of falsy values in KeyValueBasedTransformer by @dominikschubert in #7852
- fix APIGW UpdateMethod authorizer validation by @bentsku in #7860
- Fix openapi authorization header for type token by @calvernaz in #7851
- Allow privileged mode to be passed as additional flag by @viren-nadkarni in #7835
- Added hooks to the SFN provider by @giograno in #7834
- fix NotFound exception for GetTopicAttributes by @bentsku in #7862
- Return RSA public key in SPKI format by @laszlovandenhoek in #7856
- Ignore AWS::SecretsManager-2020-07-03 transform by @silv-io in #7863
- Fix S3 routing in moto to allow upload of favicon.ico files by @whummer in #7868
- Stop tracking the values of HOST* related envars by @simonrw in #7877
- Add v1 provider aliases for legacy providers by @dominikschubert in #7875
- DynamoDB: Fix unprocessed item not added to response by @viren-nadkarni in #7879
- Refactor API GW tests, assert correct JSON content-type for DDB integration by @whummer in #7843
- Track additional host variables by @simonrw in #7883
- apigw parity work on Model resource by @bentsku in #7828
- add ListBuckets into general CORS handling by @bentsku in #7889
- fix scheduler start/stop for persistence by @steffyP in #7885
- KMS: Fix double hashing during sign/verify operations by @viren-nadkarni in #7878
- fix validator and model check during invocation by @bentsku in #7846
- fix validation in test for put-subscription-filter with lambda by @steffyP in #7894
- Fix SortingTransformer for nested input by @dominikschubert in #7895
- Fix deletion of file content on snapshot error by @dominikschubert in #7898
- add an exception handler to serialize werkzeug HTTPException by @thrau in #7790
- implement apigw Model validation if in use by a Method by @bentsku in #7872
- fix sns unsubscribe to be idempotent by @thrau in #7901
- add apigatewayv2 transformers by @calvernaz in #7903
- Update ASF APIs by @localstack-bot in #7909
- Unify hostnames in returned URLs by @simonrw in #7774
- Add lambda config to ignore architecture by @joe4dev in #7890
- Add Lambda runtime parity configuration for ulimit by @joe4dev in #7871
- fix APIGW request validation with empty request body by @bentsku in #7916
- add tracing for handler chain by @thrau in #7908
- 7910 support having a grant with the same name on different keys by @BlueDragon23 in #7911
- Implement basic persistence for lambda v2 by @dominikschubert in #7913
- Add foundation for basic xray integration by @dominikschubert in #7892
- Disable moto S3 bucket policy enforcement to unblock CDK bootstraps by @whummer in #7907
- Add support for provisioned and reserved concurrency by @dominikschubert in #7881
- Add support for VPC endpoints for private API Gateway REST APIs by @whummer in #7905
- Added test cases and fixed issues for RequestValidator apigateway by @sannya-singal in #7917
- Fix lambda destination default retries by @dominikschubert in #7933
- update the dockerhub action to update README file by @HarshCasper in #7925
- Add pro clients, allow setting of request params through a "proxy" by @dfangl in #7923
- v2 pipeline changes by @alexrashed in #7927
- SES: Fix sent email counter double incrementing by @viren-nadkarni in #7919
- remove code for LEGACY_DIRECTORIES by in @thrau in #7865
- remove /docker-entrypoint-initaws.d init scripts by @thrau in #7864
- remove LEGACY_EDGE_PROXY by @thrau in #7886
- fix config.dirs usage in the CLI by @thrau in #7906
- Update PRO image warning by @silv-io in #7928
- S3 migration - set s3 v2 provider as default @steffyP and @bentsku in #6827
- Removing state serializer interfaces by @giograno in #7921
- Configure cosmetic and functional variables with LOCALSTACK_HOST and GATEWAY_LISTEN by @simonrw in #7893
- add fix for CLI tests to properly set CLI context by @thrau in #7939
- Switch to new Lambda provider implementation by @dominikschubert and @dfangl in #6724
- Dynamically determine ports check Docker image to avoid duplicate download by @whummer in #7931
- re-add mkdirs call before starting localstack docker container by @thrau in #7944
- Fix test failures for community tests in PRO by @dominikschubert in #7950
- Add cache for common functions by @dfangl in #7942
- Added test cases and fixed issues for DocumentationPart apigateway by @sannya-singal in #7938
- Remove pro check and conditional client config for endpoint tests by @dominikschubert in #7953
- Fix lambda urls when persistence is enabled by @dfangl in #7954
- Remove mention of deprecated LAMBDA_EXECUTOR in docker-compose files by @dominikschubert in #7951
- Fix issues with LOCALSTACK_HOST parsing by @simonrw in #7945
- add check for FakeDeleteMarker in case of versioned bucket by @bentsku in #7948
- fix client metadata to always return an empty string for api_key by @thrau in #7960
- skip test_kinesis_firehose_opensearch_s3_backup + es equivalent + fix s3 flake by @bentsku in #7962
- Add back s3 v2 provider alias by @dominikschubert in #7943
- fix: dynamodb restart by @giograno in #7965
- fix CORS issue for CreateBucket and ListBuckets by @bentsku in #7961
- fix race condition in service plugin loading by @thrau in #7968
- Fix websocket template expression by @calvernaz in #7778
- Implement reset hooks for lambda provider by @dominikschubert in #7967
- skip test_package_installer_custom_lock to unblock CI pipeline by @thrau in #7959
- Update ASF APIs by @localstack-bot in #7974
- small pipeline improvements by @alexrashed in #7976
- Don't source (LOCALSTACK_)HOST in docker entrypoint by @simonrw in #7978
- fixed too many arguments for local lambda executor by @AndrewChubatiuk in #7568
- Fix cloud formation S3 CorsConfig by @deepak2431 in #7866
- Add GATEWAY_LISTEN to analytics events by @simonrw in #7977
- fix typo in CODEOWNERS for S3 by @bentsku in #7982
- Adds support for arrays in event bridge rule pattern by @bruno-imx in #7947
- Extend docker/net utils to allow port checks for UDP ports by @whummer in #7955
- add test for ListObjectsV2 + fix config flag by @bentsku in #7987
- add test for ListMultipartUploads by @bentsku in #7981
- rework init system that starts localstack by @thrau in #7970
- Improve Lambda deprecation warnings by @joe4dev in #7992
- fix CLI tests after reworking init process by @alexrashed in #7995
- Ignore empty provider overrides by @silv-io in #7998
- Bump moto-ext to 4.1.5.post2 by @viren-nadkarni in #7952
- remove DISABLE_TERM_HANDLER in docker entrypoint by @alexrashed in #7999
- move feature requests to discussion pages by @HarshCasper in #7304
- fix flaky CLI tests after startup time improved by @alexrashed in #7996
- fix response structure for S3 ListMultipartUploads by @bentsku in #7983
- add dynamic check of origin for CORS by @bentsku in #7904
- Fix length check for CloudFormation TemplateBody by @whummer in #8001
- Add utility for creating a client assuming a role, migrate some inter-service communication to new client by @dfangl in #7990
- Support non-standard AWS regions by @viren-nadkarni in #7997
- create separate cache directory for cli by @thrau in #8005
- Ignore publish version code hash if hot-reloading is active by @dfangl in #8004
- fix signal handling in localstack-supervisor by @thrau in #8007
- Add basic cloudwatch metrics for new lambda provider by @dominikschubert in #8006
- Log reported runtime init error by @dominikschubert in #8009
- Remove unneeded defaults for envars by @simonrw in #8012
- use GitHub auth token in download util if available by @alexrashed in #8014
New Contributors
- @justinwyer made their first contribution in #7816
- @deepak2431 made their first contribution in #7826
- @laszlovandenhoek made their first contribution in #7856
- @BlueDragon23 made their first contribution in #7911
- @AndrewChubatiuk made their first contribution in #7568
- @bruno-imx made their first contribution in #7947
Full Changelog: v1.4.0...v2.0.0