Lite Python AMQP server implementation for dev and mock? [closed]

Question

In Django, running ./manage.py runserver is really nice for dev, avoiding the hassle to setup and start a real webserver.

If you are not running Django, you can still setup a gunicorn server very easily.

Is there something similar for AMQP?

I don't need a full implementation nor something robust, just something that is easy to install and run for dev. PyPi package would be great.

Celery is not the answer. I don't want a client, I want a server. Like a mini python RabbitMq.

Answer 1

I'm not aware of any AMQP broker implemented in Python. And I am not aware of a 'lite' implementation in general; I think that implementing an AMQP broker is sufficiently complex that those that try it either aim to be close to one of the versions of the AMQP specification, or don't bother at all. :)

I also don't quite see how running a broker presents the same problems as running a test web server for your web application.

The web server does nothing useful without your application to run inside it, and while you're developing your application it makes sense to be able to run it without needing to do a full deployment.

But you are not implementing the internals of the broker, and you can configure it dynamically so (unlike the web server) it doesn't need to restart itself every time you change your code. Exchanges, bindings and queues can be declared by the application under test and then automatically deleted afterwards.

Installing RabbitMQ is not difficult at all, and it should need hardly any configuration, if any, because it comes with a default vhost and guest user account which are fine for use in an isolated test environment. So I have never had a problem with having RabbitMQ simply running on my test machine.

Maybe you've had some particular issue that I've not thought of; if that's the case, please do leave a comment (or expand your question) to explain it.

---

Edit: Recently I've been doing quite a lot of testing of AMQP-based applications, and I've found RabbitMQ's Management Plugin very useful. It includes an HTTP API which I'm using to do things like create a new vhost for each test run, and destroy it afterwards to clean up the broker's state. This makes running tests on a shared broker much less intrusive. Using the HTTP API to manage this, rather than the AMQP client being tested, avoids the tests becoming somewhat circular.

Answer 2

I had your same question and was shocked to see how dry the space is. Even after all this time, there's hardly a lightweight AMQP server out there. I couldn't even find toy implementations on Github. AMQP seems like a beast of a protocol. I also found that RabbitMQ is probably about as light as it gets.

I ended up going with a Docker based solution for my integration tests that automatically starts and kills a RabbitMQ container. You need to install the Docker Python library and (of course) have a Docker daemon running on your machine. I was already using Docker for other things so it wasn't a biggie for my setup; YMMV. After that, basically I do:

import docker

client = docker.from_env()
c = client.containers.run(
    'rabbitmq:alpine',                       # use Alpine Linux build (smaller)
    auto_remove=True,                        # Remove automatically when stopped
    detach=True,                             # Run in daemon mode  
    ports={'5672/tcp' : ('127.0.0.1', None)} # Bind to a random localhost port
)

container = client.containers.get(c.id)      # Re-fetch container for port
port = container.attrs['NetworkSettings']['Ports']['5672/tcp'][0]['HostPort']

# ... Do any set up of the RabbitMQ instance needed on (127.0.0.1:<port>)

# ... Run tests against (127.0.0.1:<port>)

container.kill()   # Faster than 'stop'. Will also delete it so no need to be nice

It sucks to have to wait for a whole server startup in tests but I suppose if you wanted to get really clever you could cache the instance and just purge it before each test and maybe have it start once at the beginning of your dev session and then be refreshed at the beginning of the next one. I might end up doing that.

If you wanted something longer lived that isn't necessarily programmatically started and killed for persistent dev then the normal Docker route would probably serve you better.