Menu
Can someone tell me whether Celery executes a task in a thread or in a separate child process? The documentation doesn't seem to explain it (read it like 3 times). If it is a thread, how does it get pass the GIL (particularly whom and how an event is notified)?
How would you compare celery's async with Twisted's reactor model? Is celery using reactor model after all?
Thanks,
350
When you start Celery ( celery -A tasks worker ) 1 worker is created. This worker is actually a supervisor process that will spawn child-processes or threads which will execute the tasks.
Celery itself is using billiard (a multiprocessing fork) to run your tasks in separate processes.
delay() has comes preconfigured and only requires arguments to be passed to the task — that's sufficient for most basic needs. Apply_async is more complex, but also more powerful then preconfigured delay. It is always better to use apply_async with specifically set options for maximum flexibility.
Celery is an asynchronous task queue framework written in Python. Essentially, you can think of it as a queue data structure with the FIFO (first in, first out) principle. So the first task you put into the queue gets processes first. This makes it possible to schedule tasks.
Can someone tell me whether Celery executes a task in a thread or in a separate child process?
Neither, the task will be executed in a separate process possibly on a different machine. It is not a child process of the thread where you call 'delay'. The -C and -P options control how the worker process manages it's own threading. The worker processes get tasks through a message service which is also completely independent.
How would you compare celery's async with Twisted's reactor model? Is celery using reactor model after all?
Twisted is an event queue. It is asynchronous but it's not designed for parallel processing.
184
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
