We have had some problems with concurrency for a hefty load (primarily for Postiz cloud).
Here are the changes we have implemented:
- Workers can now reach a maximum of 300 concurrent jobs.
- Bottleneck JS timeout (expiration) is set to 60 seconds for a single HTTP request; if it fails, it returns to the user "Something is wrong with ${identifier}"
- Max Stalled jobs changed to 10.
- Cron for missing jobs - fixed.
Ideology behind the change:
Until recently, Postiz added posts to the queues and processed them without any concurrency, which works great with open-source instances that are not automated.
However, the problem was the restrictions of social media platforms - what if I want to post 10 posts to Instagram at the same time? If Instagram has a concurrency limit per second, what is the maximum number of requests that I can make?
Instagram will throw you off with a rate limit error.
So we needed a better solution to make "http" requests, wait between multiple workers.
So only X requests could run concurrently, and wait for the other requests.
Since you may want to run multiple workers simultaneously, the solution must be Redis-supported and not limited to local execution.
So libraries like p-queue would not be sufficient.
Another thought we have is to move to bullmq-pro, which would make concurrency of the entire job (not HTTP request)
The main problem was that it costs money, so it's hard to push it into the open-source, although it would probably be the easiest solution.
Bullmq-pro allows you to create concurrent jobs based on groups - so a group can be, for example, "LinkedIn" or "Instagram". So, as promised, only one concurrent job will be run at a time on the social platform.
In any case, I hope these transparent updates provide you with some clarity.
Full Changelog: v2.5.2...v2.4.3