What available message solutions are there for inter-process communication in ruby?

Question

I have a rails app using delayed_job. I need my jobs to communicate with each other for things like "task 5 is done" or "this is the list of things that need to be processed for task 5".

Right now I have a special table just for this, and I always access the table inside a transaction. It's working fine. I want to build out a cleaner api/dsl for it, but first wanted to check if there were existing solutions for this already. Weirdly I haven't found a single things, I'm either googling completely wrong, or the task is so simple (set and get values inside a transaction) that no one has abstracted it out yet.

Am I missing something?

clarification: I'm not looking for a new queueing system, I'm looking for a way for background tasks to communicate with one another. Basically just safely shared variables. Do the below frameworks offer this facility? It's a shame that delayed job does not.

use case: "do these 5 tasks in parallel, and then when they are all done, do this 1 final task." So, each of the 5 tasks checks to see if it's the last one, and if it is, it fires off the final task.

Answer 1

I use resque. Also there are lots of plugins, which should make inter-process comms easier.

Using redis has another advantage: you can use the pub-sub channels for communication between workers/services.

Another approach (but untested by me): http://www.zeromq.org/, which also has ruby bindings. If you like to test new stuff, then try zeromq.

---

Update

To clarify/explain/extend my comments above:

Why I should switch from DelayedJobs to Resque is the mentioned advantage that I have queue and messages in one system because Redis offers this.

Further sources:

• https://github.com/blog/542-introducing-resque
• https://github.com/defunkt/resque#readme

If I had to stay on DJ I would extend the worker classes with redis or zeromq/0mq (only examples here) to get the messaging in my extisting background jobs.

I would not try messaging with ActiveRecord/MySQL (not even queueing actually!) because this DB isn't the best performing system for this use case especially if the application has too many background workers and huge queues and uncountable message exchanges in short times.

If it is a small app with less workers you also could implement a simple messaging via DB, but also here I would prefer memcache instead; messages are short living data chunk which can be handled in-memory only.

Shared variables will never be a good solution. Think of multiple machines where your application and your workers can live on. How you would ensure a save variable transfer between them?

Okay, someone could mention DRb (distributed ruby) but it seems not really used anymore. (never seen a real world example so far)

If you want to play around with DRb however, read this short introduction.

My personal preference order: Messaging (real) > Database driven messaging > Variable sharing

Answer 2

• memcached
• rabbitmq