Pika blocking_connection.py random timeout connecting to RabbitMQ

Question

i have a rabbit mq running on machine

• both client and rabbitMQ are running on the same network
• rabbitMQ has many clients
• i can ping client from rabbitMQ and back
• longest latency measured between the machine is 12.1 ms
• network details : Standard Switch network (network of virtual machines running on single physical machine - using vmware VC)

im getting random timeouts when initializing RPC connection

/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py

problem is that the timeout isn't consistent and happens from time to time.

when manually testing this issue and running the blocking_connection.py 1000 times from the same machine that it fails no timeout accrue.

this is the error i get when failing :

2013-04-23 08:24:23,396 runtest-trigger.24397 24397 DEBUG      producer_rabbit initiate_rpc_connection Connecting to RabbitMQ RPC queue rpcqueue_java on host: auto-db1
2013-04-23 08:24:25,350 runtest-trigger.24397 24397 ERROR      testrunner go   Run 1354: cought exception: timed out
Traceback (most recent call last):
  File "/testrunner.py", line 193, in go
    self.set_runparams(jobid)
  File "/testrunner.py", line 483, in set_runparams
    self.runparams.producers_testrun = self.initialize_producers_testrun(self.runparams)
  File "/basehandler.py", line 114, in initialize_producers_testrun
    producer.set_testcase_checkout()
  File "/baseproducer.py", line 73, in set_testcase_checkout
    self.checkout_handler = pm_checkout.get_producer(self.testcasecheckout)
  File "/producer_manager.py", line 101, in get_producer
    producer = self.load_producer(plugin_dir, producer_name)
  File "/producer_manager.py", line 20, in load_producer
    producer = getattr(producer_module, 'Producer')(producer_name, self.runparams)
  File "/producer_rabbit.py", line 13, in __init__
    self.initiate_rpc_connection()
  File "/producer_rabbit.py", line 67, in initiate_rpc_connection
    self.connection = pika.BlockingConnection(pika.ConnectionParameters( host=self.conf.rpc_proxy))
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 32, in __init__
    BaseConnection.__init__(self, parameters, None, reconnection_strategy)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 50, in __init__
    reconnection_strategy)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 170, in __init__
    self._connect()
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/connection.py", line 228, in _connect
    self.parameters.port or  spec.PORT)
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/blocking_connection.py", line 44, in _adapter_connect
    self._handle_read()
  File "/usr/lib/python2.6/site-packages/pika-0.9.5-py2.6.egg/pika/adapters/base_connection.py", line 151, in _handle_read
    data = self.socket.recv(self._suggested_buffer_size)
timeout: timed out

please assist

Answer 1

I had a similar issue. If everything looks fine, then you most likely have some sort of miss configuration, e.g. bad binding. If miss configured, then you'll get a timeout because the script can't reach where it thinks it needs to go, so the error can be miss leading in this case.

For my problem, I specifically had issues with both my rabbitmq.config file and my bindings and had to use my python solution shown in: RabbitMQ creating queues and bindings from a command line over the command line example I showed. Once updated and configured properly, everything worked fine. Hopefully this gets you in the right direction.

Answer 2

Pika provides some time out issue when connecting different hosts.Solution is to pass a socket_timeout argument in connection parameter.Pika should upgrade to >=0.9.14

credentials = pika.PlainCredentials(RABBITMQ_USER, RABBITMQ_PASS)
connection = pika.BlockingConnection(pika.ConnectionParameters(
    credentials=credentials,
        host=RABBITMQ_HOST,
         socket_timeout=300))
channel = connection.channel()