Menu
The command celery worker -A proj --autoscale=10,1 --loglevel=info starts workers with auto scaling.
When creating workers with multi,
me@mypc:~/projects/x$ celery multi start myworker --autoscale=10,1
celery multi v4.1.0 (latentcall)
> Starting nodes...
> myworker@mypc: OK
me@mypc:~/projects/kate$ celery multi show myworker
/home/me/.virtualenvs/kate/bin/python2.7 -m celery worker --detach -n myworker@mypc --pidfile=myworker.pid --logfile=myworker%I.log --executable=/home/me/.virtualenvs/x/bin/python2.7
How do I enable multi to autoscale?
226
Celery allows for many, many long-running tasks to run at the same time. For this reason, Celery helps make systems more robust and supports high availability and horizontal scaling.
So Celery (and other queue frameworks) has other benefits as well - Think of it as a 'task/function manager' rather then just a way of multithreading.
As for --concurrency celery by default uses multiprocessing to perform concurrent execution of tasks. The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of available CPU's if not set.
Autoscaling will start new processes when the work load increases. In your case you have set the min number to 1, so you will only see 1 process, unless you start putting more tasks onto your queues. You can change the way the autoscaler works. See the link below for more details:
http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling
155
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
