I'm running (Django) cerillium to schedule tasks for remote workers w1, w2, w3. Each of these workers has their own queue, from which they are consuming the tasks kept by "scheduler", which is another celery work by defeating on the master server:
w1: q1 w2: Q2 w3: Q3
Scheduler scheduling work based on a DB check, i.e. it will replace one task with the same parameter if DB does not update according to the operation . So if one or more queues are accumulating, then many tasks ("duplicate" from my app's perspective) with the same parameter can be in multiple queues at the same time.
I have some weird behavior with it: When there are duplicate functions in multiple queues, if the queue runs its own instance of a task, only a few milliseconds ago, the second line of "duplicate" functions is executed is. So suddenly all the tasks are executed at the same time, even if they keep pressing on different minutes from each other.
Is there a document or other reference that tells this behavior? Is this behavior known, if so, how do I change it? I only have to do an example of this work to run.
Comments
Post a Comment