Menu
I am creating a django webserver that allows the user to run some "executables" on a local machine and to analyse their output through a webpage.
I have previously used a Celery tasks queue in order to run "executables" in similar situations. However, after reading up on Python concurrent.futures, I am beginning to wonder if I should use ThreadPoolExecutor, or ProcessPoolExecutor (or ThreadPoolExecutor inside a ProcessPoolExecutor :D) instead?
Googling I could only find one relevant question comparing Celery to Tornado, and it steered to using Tornado alone.
So should I use Celery or a PoolExecutor for my simple webserver, and why?
412
Perhaps the most important difference is the type of workers used by each class. As their names suggest, the ThreadPoolExecutor uses threads internally, whereas the ProcessPoolExecutor uses processes. A process has a main thread and may have additional threads. A thread belongs to a process.
Use the ThreadPoolExecutor class when you need to execute tasks that may or may not take arguments and may or may not return a result once the tasks are complete. Use the ThreadPoolExecutor class when you need to execute different types of ad hoc tasks, such as calling different target task functions.
The ThreadPoolExecutor Python class is used to create and manage thread pools and is provided in the concurrent. futures module.
A thread pool is a pattern for managing multiple threads efficiently. Use ThreadPoolExecutor class to manage a thread pool in Python. Call the submit() method of the ThreadPoolExecutor to submit a task to the thread pool for execution. The submit() method returns a Future object.
You need to use celery if:
In addition to this is a very mature library with side projects that helps you also on UI presentation side, have a look at Jobtastic.
If you don't need any of the listed point and you just need to execute this task without caring to much about status and without particular needs of scalability than just keep it simple.
About using ThreadPoolExecutor or ProcessPoolExecutor just keep in mind that the second will be able to receive and return only pickable objects and that the first will spawn child thread attached to your main process (probably your webserver if you are not using it inside another detached process) so the approach of mix them can make sense depending on the details of your implementation.
122
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
