Celery error 'connection reset by peer'

Question

Celery's connection is reset periodically with the error message:

[Errno 104] Connection reset by peer

Currently, I lose the task and anything triggered afterwards until I reset the server. I have 2 questions: (1) What is causing the error/how can I prevent it? (2) How can I catch the error when it occurs and retry my tasks?

I'm running on Heroku. Celery is configured with AMQP (CloudAMQP) as the broker and redis (REDIS_TO_GO) as the results backend.

Stacktrace:

File "celery/task/trace.py", line 224, in trace_task
  R = retval = fun(*args, **kwargs)
File "myapp/tasks.py", line 310, in mytask1
  mytask2.delay(message)
File "celery/app/task.py", line 343, in delay
  return self.apply_async(args, kwargs)
File "celery/app/task.py", line 458, in apply_async
  with app.producer_or_acquire(producer) as P:
File "python2.7/contextlib.py", line 17, in __enter__
  return self.gen.next()
File "celery/app/base.py", line 256, in producer_or_acquire
  with self.amqp.producer_pool.acquire(block=True) as producer:
File "kombu/connection.py", line 712, in acquire
  R = self.prepare(R)
File "kombu/pools.py", line 57, in prepare
  p.revive(connection.default_channel)
File "kombu/connection.py", line 626, in default_channel
  self._default_channel = self.channel()
File "kombu/connection.py", line 167, in channel
  chan = self.transport.create_channel(self.connection)
File "kombu/transport/amqplib.py", line 324, in create_channel
  return connection.channel()
File "kombu/transport/amqplib.py", line 243, in channel
  return Channel(self, channel_id)
File "kombu/transport/amqplib.py", line 267, in __init__
  super(Channel, self).__init__(*args, **kwargs)
File "amqplib/client_0_8/channel.py", line 82, in __init__
  self._x_open()
File "amqplib/client_0_8/channel.py", line 469, in _x_open
  self._send_method((20, 10), args)
File "amqplib/client_0_8/abstract_channel.py", line 76, in _send_method
  method_sig, args, content)
File "amqplib/client_0_8/method_framing.py", line 252, in write_method
  self.dest.write_frame(1, channel, payload)
File "amqplib/client_0_8/transport.py", line 165, in write_frame
  frame_type, channel, size, payload, 0xce))
File "python2.7/socket.py", line 224, in meth
  return getattr(self._sock,name)(*args)

Versions:

Django==1.4.1
django-celery==3.0.9
celery==3.0.9
celery-with-redis==3.0
amqplib==1.0.2
kombu==2.4.7

Answer 1

It looks like you are running out of connections. When you look at CloudAMQP Console Open connections you can verify it.

Maybe take a look at celery BROKER_POOL_LIMIT