Using maxtasksperchild with eventlet

Question

We have a python application with some celery workers.

We use the next command to start celery worker:

python celery -A proj worker --queue=myqueue -P prefork --maxtasksperchild=500

We have two issues with our celery workers.

1. We have a memory leak
2. We have pretty big load and we need a lot of workers to process everything fast

We're still looking into memory leak, but since it's legacy code it's pretty hard to find a cause and it will take some time to resolve this issue. To prevent leaks we're using --maxtasksperchild, so each worker after processing 500 events restarts itself. And it works ok, memory grows just to some level.

Second issue is a bit harder. To process all events from our celery queue we have to start more workers. But with prefork each process eats a lot of memory (about 110M in our case) so we either need a lot of servers to start right number of workers or we have to switch from prefork to eventlet:

python celery -A proj worker --queue=myqueue -P eventlet --concurrency=10

In this case we'll use the same amount of memory (about 110M per process) but each process will have 10 workers which is much more memory efficient. But the issue with this is that we still have issue #1 (memory leak), and we can't use --maxtasksperchild because it doesn't work with eventlet.

Any thoughts how can use something like --maxtasksperchild with eventlet?

Answer 1

• Upgrade Celery, I've just quick scanned master code, they promise max-memory-per-child. Hope it would work with all concurrency models. I haven't tried it yet.
• Set up process monitoring, send graceful terminate signal to workers above memory threshold. Works for me.
• Run Celery in control group with limited memory. Works for me.