Celery immediately exceeds memory on Heroku

Question

I'm deploying a Celery process to Heroku and every time it starts, it immediately starts to rack up memory usage and crash after it exceeds the maximum.

I only have one task called "test_task" that prints once per minute.

This is Django app using Celery with a Redis backend hosted on Heroku.

Proc file:

web: daphne chatbot.asgi:channel_layer --port $PORT --bind 0.0.0.0 --verbosity 1
chatworker: python manage.py runworker --verbosity 1
celeryworker: celery -A chatbot worker -l info

Heroku logs:

app[celeryworker.1]: [INFO] 2018-09-11 23:06:36,710 : Scheduling celery jobs...
app[celeryworker.1]: [INFO] 2018-09-11 23:06:36,880 : adding minute task
app[celeryworker.1]: 
app[celeryworker.1]: -------------- celery@f46a12fb-1666-449a-b287-2ce90f95cf2c v4.2.0 (windowlicker)
app[celeryworker.1]: --- * ***  * -- Linux-4.4.0-1027-aws-x86_64-with-debian-jessie-sid 2018-09-11 23:06:37
app[celeryworker.1]: -- * - **** ---
app[celeryworker.1]: ---- **** -----
app[celeryworker.1]: - ** ---------- [config]
app[celeryworker.1]: - ** ---------- .> app:         chatbot:0x7f1c2a5ad5d0
app[celeryworker.1]: - ** ---------- .> transport:   redis://east-1-4.ec2.cloud.redislabs.com:13275//
app[celeryworker.1]: - ** ---------- .> results:     disabled://
app[celeryworker.1]: - *** --- * --- .> concurrency: 8 (prefork)
app[celeryworker.1]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
app[celeryworker.1]: --- ***** -----
app[celeryworker.1]: -------------- [queues]
app[celeryworker.1]: .> celery           exchange=celery(direct) key=celery
app[celeryworker.1]: 
app[celeryworker.1]: 
app[celeryworker.1]: [tasks]
app[celeryworker.1]: . companies.tasks.test_task
app[celeryworker.1]: 
heroku[celeryworker.1]: Process running mem=616M(120.4%)
heroku[celeryworker.1]: Error R14 (Memory quota exceeded)
app[api]: Build succeeded
app[celeryworker.1]: [2018-09-11 23:06:55,033: INFO/MainProcess] Connected to redis://rediscloud:**@redis-13275.c12.us-east-1-4.ec2.cloud.redislabs.com:13275//
app[celeryworker.1]: [2018-09-11 23:06:55,201: INFO/MainProcess] mingle: searching for neighbors
app[celeryworker.1]: [2018-09-11 23:06:56,462: INFO/MainProcess] mingle: all alone
heroku[celeryworker.1]: Process running mem=1063M(207.8%)
heroku[celeryworker.1]: Error R15 (Memory quota vastly exceeded)
heroku[celeryworker.1]: Stopping process with SIGKILL
heroku[celeryworker.1]: State changed from up to crashed
heroku[celeryworker.1]: State changed from crashed to starting
heroku[celeryworker.1]: Process exited with status 137

Is there something obvious I'm doing wrong?

Answer 1

Turns out changing the --concurrency argument in the celery worker process to a lower number worked.

I set it to celery -A chatbot worker -l info --concurrency 2 and it seems to be working!