I have a celery setup with rabbitmq. The issue is that celery is moving tasks to reserved state while running a long task, and do not execute them until the long running task is completed.
I want to accomplish that without using routing, and enabling "-Ofair" flag does the job. Prefork pool prefetch settings
How to enable the flag in celery python? Thanks
I am using celery 3.1.19
$ celery report
software -> celery:3.1.19 (Cipater) kombu:3.0.32 py:3.4.3
billiard:3.3.0.22 py-amqp:1.4.8
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
I am using Celery as follows and concurrency is set to 4:
app = celery.Celery()
app.conf.update(
BROKER_URL=broker,
CELERY_RESULT_BACKEND=backend,
CELERY_TASK_SERIALIZER='json',
CELERY_IMPORTS=imports or [],
CELERYD_CONCURRENCY=concurrency,
CELERYD_HIJACK_ROOT_LOGGER=False
)
Here is how I start the worker:
worker = app.Worker(
hostname=hostname,
queues=[hostname]
)
worker.start()
You should be able to run it this way.
worker = app.Worker(
hostname=hostname,
queues=[hostname],
optimization='fair'
)
worker.start()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With