Celery v5.5.0 Beta 3 is now available for testing.
Please help us test this version and report any issues.
Key Highlights
Soft Shutdown
The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.
The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.
The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.
After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.
Relevant Issues: #9213, #9231, #9238
- New documentation for each shutdown type.
- New worker_soft_shutdown_timeout configuration option.
- New worker_enable_soft_shutdown_on_idle configuration option.
REMAP_SIGTERM
The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.
Previous Pre-release Highlights
Pydantic Support
This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.
After upgrading to this version, please share your feedback on the new Pydantic support.
Redis Broker Stability Improvements
The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.
After upgrading to this version, please share your feedback on the Redis broker stability.
Relevant Issues: #7276, #8091, #8030, #8384
Quorum Queues Initial Support
This release introduces the initial support for Quorum Queues with Celery.
See new configuration options for more details:
After upgrading to this version, please share your feedback on the Quorum Queues support.
What's Changed
- Added SQS (localstack) broker to canvas smoke tests by @Nusnus in #9179
- Pin elastic-transport to <= latest version 8.15.0 by @pyup-bot in #9182
- Update elasticsearch requirement from <=8.14.0 to <=8.15.0 by @dependabot in #9186
- improve formatting by @Bonifacio2 in #9188
- Add basic helm chart for celery by @necromancerthedark in #9181
- Update kafka.rst by @lokot0k in #9194
- Update pytest-order to 1.3.0 by @pyup-bot in #9198
- Update mypy to 1.11.2 by @pyup-bot in #9206
- all added to routes by @dhruvji in #9204
- Fix typos discovered by codespell by @cclauss in #9212
- Use tzdata extras with zoneinfo backports by @auvipy in #8286
- Use
docker composein Contributing's doc build section by @KeisukeYamashita in #9219 - Failing test for issue #9119 by @mgedmin in #9215
- Fix date_done timezone issue by @FKgk in #8385
- CI Fixes to smoke tests by @Nusnus in #9223
- fix: passes current request context when pushing to request_stack by @nikatlas in #9208
- Fix broken link in the Using RabbitMQ docs page by @thedrow in #9226
- Added Soft Shutdown Mechanism by @Nusnus in #9213
- Added worker_enable_soft_shutdown_on_idle by @Nusnus in #9231
- Bump cryptography from 43.0.0 to 43.0.1 by @dependabot in #9233
- Added docs regarding the relevancy of soft shutdown and ETA tasks by @Nusnus in #9238
- Show broker_connection_retry_on_startup warning only if it evaluates as False by @serl in #9227
- Fixed docker-docs CI failure by @Nusnus in #9240
- Added docker cleanup auto-fixture to improve smoke tests stability by @Nusnus in #9243
- print is not thread-safe, so should not be used in signal handler by @Zhong-z in #9222
- Prepare for (pre) release: v5.5.0b3 by @Nusnus in #9244
New Contributors
- @Bonifacio2 made their first contribution in #9188
- @necromancerthedark made their first contribution in #9181
- @lokot0k made their first contribution in #9194
- @dhruvji made their first contribution in #9204
- @KeisukeYamashita made their first contribution in #9219
- @mgedmin made their first contribution in #9215
- @FKgk made their first contribution in #8385
- @nikatlas made their first contribution in #9208
- @serl made their first contribution in #9227
Full Changelog: v5.5.0b2...v5.5.0b3