Run pika ioloop in background or use custom ioloop

Question

I have a feeling that this should really not be all that difficult, yet I have had little success so far.

Say I have a class called PikaClass that wraps pika and provides some business methods.

def PikaClass(object):
  def __init__(self):
     # connect to the broker
     self.connection = pika.SelectConnection(<connection parameters>, self.on_connect)
     # ..other init stuff..

  def on_connect(self, connection):
     # called when the connection has been established 
     # ..open a channel, declare some queues, etc.

  def start(self):
     # start the polling loop 
     self.connection.ioloop.start()

  def foo(self, **kwargs):
     # do some business logic, e.g., send messages to particular queues

Intuitively, this is what I would like to achieve: a user creates an instance of PikaClass, sets the loop going in the background, and then interacts with the object by calling certain business methods

p = PikaClass()
p.start()
bar = p.foo(..)

The problem is that p.start() blocks and prevents the main code from interacting with the object once start() has been called. My first thought was to wrap the call in a thread:

Thread(target=p.start()).start()
bar = p.foo(..)

But that still blocks and you never get to p.foo(..). The docs mention that you shouldn't share a connection between threads so that may cause a problem somewhere.

I have also tried using AsyncoreConnection instead of SelectConnection, and calling _connect() directly (instead of using the ioloop) but that does not have any effect (nothing happens).

So how can I run the ioloop in the background, or at least run my own ioloop?

Note: This is Python 2.6 on win64 (xp) with the latest pika 0.9.4

Answer 1

You are calling 'p.start' instead of passing it as a parameter. The code should be:

Thread(target=p.start).start()

Thread will call p.start when Thread.start is executed.

I'm not sure if this will solve your problem, but it may help you reach the solution.

Answer 2

The GIL is not a problem here, because the ioloop spends nearly all of its time in the select(2) system call, during which time the GIL is released and other Python threads can run off and perform other tasks.

The simplest approach would be to set up and tear down a queue connection for every request. You might think this would be too expensive — since it requires re-authentication and (possibly) repeat SSL negotiation with every connection — but it should be the simplest and most robust and easiest to write, which should be your controlling factors unless you know that setup and teardown will in fact hurt performance in your application overall (which is best measured by testing).

Another approach would be to only start() the ioloop once you had a message to send, and have the method that receives the reply stop the ioloop so that your program gets control again. You can make the ioloop return early with:

    connection.ioloop.poller.open = False

and then remembering to set it back to True before you call start() again to wait for another reply.