Error with celery worker in elastic beanstalk (using django and SQS) [ImportError: The curl client requires the pycurl library.]

Question

I'm trying to deploy to elastic beanstalk a django project that uses celery periodic tasks, using SQS.

I've been more or less following the instructions here:

How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

When I deploy to eb, the periodic tasks are not being executed.

Checking the celery-beat log, everything seems right:

celery beat v4.2.1 (windowlicker) is starting.
__    -    ... __   -        _
LocalTime -> 2019-01-27 09:48:16
Configuration ->
    . broker -> sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:48:44,659: INFO/MainProcess] beat: Starting...
[2019-01-27 09:48:44,660: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:44,809: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2019-01-27 09:48:44,809: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:49,865: INFO/MainProcess] Writing entries...
[2019-01-27 09:49:00,409: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:50:00,050: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:50,543: INFO/MainProcess] Writing entries...
[2019-01-27 09:52:00,048: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:53:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)

but there seems to be some problem with the celery worker. This is the error I get in celery-worker.log:

/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:49:12,513: INFO/MainProcess] Connected to sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
[2019-01-27 09:49:12,950: WARNING/MainProcess] /opt/python/run/venv/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-01-27 09:49:12,951: INFO/MainProcess] celery@ip-172-31-23-20 ready.
[2019-01-27 09:49:12,952: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 291, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 163, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 160, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
    qname, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
    region=self.region
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
    **kwargs
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 178, in __init__
    **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

 -------------- celery@ip-172-31-23-20 v4.2.1 (windowlicker)
---- **** -----
--- * ***  * -- Linux-4.14.88-72.76.amzn1.x86_64-x86_64-with-glibc2.3.4 2019-01-27 09:49:12
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         raiseflags:0x7fbf06309f28
- ** ---------- .> transport:   sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . check_users_inboxes
  . periodic_send
  . raiseflags.celery.debug_task
  . scrapeURLs
  . spreadsheets_update
  . sum_two_numbers

I've been searching for answers and some people seem to have the same problem. I've tried deploying different versions of celery and kombu but can't solve it.

This is my requirements.txt file:

amqp==2.4.0
anyjson==0.3.3
beautifulsoup4==4.7.1
billiard==3.5.0.5
boto==2.49.0
bs4==0.0.1
cachetools==3.0.0
celery==4.2.1
certifi==2018.11.29
chardet==3.0.4
Django==2.1.5
django-celery-beat==1.4.0
django-celery-results==1.0.4
django-timezone-field==3.0
google-api-python-client==1.7.7
google-auth==1.6.2
google-auth-httplib2==0.0.3
httplib2==0.12.0
idna==2.8
kombu==4.2.2.post1
oauth2client==4.1.3
psycopg2==2.7.7
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.7.5
pycurl==7.43.0 --global-option="--with-nss"
python-crontab==2.3.6
pytz==2018.9
requests==2.21.0
rsa==4.0
six==1.12.0
soupsieve==1.7.3
stripe==2.18.1
uritemplate==3.0.0
urllib3==1.24.1
vine==1.2.0

If I ssh connect to the instance and type "python -c 'import pycurl'" everything seems right, no problem there.

Does someone know how to solve it? Any help would be appreciated.

Thank you very much.

Answer 1

when you do

python -c 'import pycurl'

make sure that the python version is the same as python3.6, to be sure, try:

python3.6 -m pip install pycurl