Celery v5.5.0 Beta 4 is now available for testing.
Please help us test this version and report any issues.
Key Highlights
Python 3.13 Initial Support
This release introduces the initial support for Python 3.13 with Celery.
After upgrading to this version, please share your feedback on the Python 3.13 support.
Previous Pre-release Highlights
Soft Shutdown
The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.
The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.
The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.
After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.
Relevant Issues: #9213, #9231, #9238
- New documentation for each shutdown type.
- New worker_soft_shutdown_timeout configuration option.
- New worker_enable_soft_shutdown_on_idle configuration option.
REMAP_SIGTERM
The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.
Pydantic Support
This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.
After upgrading to this version, please share your feedback on the new Pydantic support.
Redis Broker Stability Improvements
The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.
After upgrading to this version, please share your feedback on the Redis broker stability.
Relevant Issues: #7276, #8091, #8030, #8384
Quorum Queues Initial Support
This release introduces the initial support for Quorum Queues with Celery.
See new configuration options for more details:
After upgrading to this version, please share your feedback on the Quorum Queues support.
What's Changed
- Correct the error description in exception message when validate soft_time_limit by @narasux in #9246
- Update msgpack to 1.1.0 by @pyup-bot in #9249
- chore(utils/time.py): rename
_is_ambigious->_is_ambiguousby @pachewise in #9248 - Reduced Smoke Tests to min/max supported python (3.8/3.12) by @Nusnus in #9252
- Update pytest to 8.3.3 by @pyup-bot in #9253
- Update elasticsearch requirement from <=8.15.0 to <=8.15.1 by @dependabot in #9255
- update mongodb without deprecated
[srv]extra requirement by @fmigneault in #9258 - blacksmith.sh: Migrate workflows to Blacksmith by @blacksmith-sh in #9261
- Fixes #9119: inject dispatch_uid for retry-wrapped receivers by @pachewise in #9247
- Run all smoke tests CI jobs together by @Nusnus in #9263
- Improve documentation on visibility timeout by @kylez-ithaka in #9264
- Bump pytest-celery to 1.1.2 by @Nusnus in #9267
- Added missing "app.conf.visibility_timeout" in smoke tests by @Nusnus in #9266
- Improved stability with t/smoke/tests/test_consumer.py by @Nusnus in #9268
- Improved Redis container stability in the smoke tests by @Nusnus in #9271
- Disabled EXHAUST_MEMORY tests in Smoke-tasks by @Nusnus in #9272
- Marked xfail for test_reducing_prefetch_count with Redis - flaky test by @Nusnus in #9273
- Fixed pypy unit tests random failures in the CI by @Nusnus in #9275
- Fixed more pypy unit tests random failures in the CI by @Nusnus in #9278
- Fix Redis container from aborting randomly by @Nusnus in #9276
- Run Integration & Smoke CI tests together after unit tests passes by @Nusnus in #9280
- Added "loglevel verbose" to Redis containers in smoke tests by @Nusnus in #9282
- Fixed Redis error in the smoke tests: "Possible SECURITY ATTACK detected" by @Nusnus in #9284
- Refactored the smoke tests github workflow by @Nusnus in #9285
- Increased --reruns 3->4 in smoke tests by @Nusnus in #9286
- Improve stability of smoke tests (CI and Local) by @Nusnus in #9287
- Fixed Smoke tests CI "test-case" lables (specific instead of general) by @Nusnus in #9288
- Use assert_log_exists instead of wait_for_log in worker smoke tests by @Nusnus in #9290
- Optimized t/smoke/tests/test_worker.py by @Nusnus in #9291
- Enable smoke tests dockers check before each test starts by @Nusnus in #9292
- Relaxed smoke tests flaky tests mechanism by @Nusnus in #9293
- Updated quorum queue detection to handle multiple broker instances by @bkienker in #9294
- Non-lazy table creation for database backend by @MarcBresson in #9228
- Pin pymongo to latest version 4.9 by @pyup-bot in #9297
- Bump pymongo from 4.9 to 4.9.1 by @dependabot in #9298
- Bump Kombu to v5.4.2 by @Nusnus in #9304
- Use rabbitmq:3 in stamping smoke tests by @Nusnus in #9307
- Bump pytest-celery to 1.1.3 by @Nusnus in #9308
- Added Python 3.13 Support by @Nusnus in #9309
- Add log when global qos is disabled by @thedrow in #9296
- Added official release docs (whatsnew) for v5.5 by @Nusnus in #9312
- Enable Codespell autofix by @Nusnus in #9313
- Pydantic typehints: Fix optional, allow generics by @mathiasertl in #9319
- Prepare for (pre) release: v5.5.0b4 by @Nusnus in #9322
New Contributors
- @narasux made their first contribution in #9246
- @fmigneault made their first contribution in #9258
- @blacksmith-sh made their first contribution in #9261
- @kylez-ithaka made their first contribution in #9264
- @bkienker made their first contribution in #9294
- @MarcBresson made their first contribution in #9228
Full Changelog: v5.5.0b3...v5.5.0b4