Looking for persistent, distributed, worker queue for erlang [closed]

Question

Before re-inventing the wheel, I'm looking for pointers to open source projects that meet these requirements.

• implemented in erlang though go or C are possible, if there isn't too much baggage (eg: twisting maze of dependencies.)
• endpoint or client in erlang (Eg: I want erlang code to be run when I do the jobs.)
• distribute jobs to nodes, and call some erlang function to do the jobs.
• persist the jobs somehow
• no master nodes, no single point of failure
• homogenous architecture
• manage a queue of jobs that may get backed up, without dropping jobs on the floor
• a job being done more than once is ok
• operational profile like Riak or Couchbase (Eg: start one node, then start others and point them to it.)

Strong preference for something lightweight. There's a lot of overwrought enterprise grade solutions out there in erlang that seem like they would take as long to learn as it would be for me to recreate this from scratch (in fact, I basically architected a solution to this very problem in answer to someone else's question here on stackoverflow. I can build what I described, but this seems like one of those needs that is right in the middle of what erlang was designed for.)

What I've considered: - ejabbered - more of a messaging framework - rabitmq - theoretically does this but every time I go to their website I drown in a sea of abstractions. It seems ready to do everything. I can't even tell if it has any sort of persistence.

Edit to add: Here's a slide deck on distributed locking using locker. Seems like it solves a key part of the problem (if one wants to roll their own.) http://www.slideshare.net/knutnesheim/locker-distributed-consistent-locking

Further Edit: I really am looking for something more lightweight than RabbitMQ. I know it can do what I want, but it seems like the cost in learning it is comparable to the cost of doing it myself, where at the end the custom solution would be closer to what I really need.

Answer 1

We use RABBITMQ to bind up all our applications into a complete set of things. Within the whole setup is a central RABBITMQ server to which systems create queues , whether persistent or temporary. Because of RABBITMQ availability, our entire distribution system runs on top of it. Systems built using different technologies send and receive tasks from other systems via RABBITMQ.

We came up with a Message Format, which can be in JSON or XML in which systems communicate to each other. It is so fast. However, there are so many details i wont go into here but, i had to write an OTP application on top RABBITMQ Client to abstract all AMQP stuff from erlang programmers. All a programmer knows is an APi, say i am sending a request to System A, just prepare the message format, M and call the API: zeenode_amqp:req(SystemA,Message). Systems can send synchronous or asynchronous requests.

What should be your take from this: RABBITMQ is very good for queuing systems. Infact in our setup, RABBITMQ pushes the messages straight to the Servers as soon as they hit RABBITMQ from clients. By using a carefully designed naming convention of queues and exchanges and carefully understanding the various AMQP use-case models, it will be perfect fit for you.

i think it is possible to roll your own, using Process dictionaries like Gproc in conjunction with Erlang Queue Module, and Poolboy. Anyways, i would recommend RABBITMQ. let me know, i could send you some libraries, so you study them and see if they work for you. once you have a good RABBITMQ setup, well configured following the docs on their site, and then, you have the Erlang amqp client as well, installed, then it is possible to create queues and exchanges dynamically (whether you want them persistent or not, depends on what you are doing). You can even cluster RABBITMQ servers so that there is guaranteed availability.

Answer 2

We use RabbitMQ for tasks like this. RabbitMQ exchange-queue-binding model supports flexible configuration. You publish to an exchange, the binding makes sure the messages arrive in the queue(s). One or more worker processes can subscribe to a queue. Queues and exchanges can be persistent. Some people say it takes about half a year to fully understand RabbitMQ.