Pushing operations to a message queue for processing when able to

Question

My title isn't great because I'm finding it hard to explain what I want in a small snippet...soo I'll try in depth!

I'm writing an Elixir app using the Phoenix web framework.

What I want to do is this:

• Call a web route, which phoenix handles and passes off to a controller
• The controller to initiate a process that is to be handed off to a message queue
• The latest item in the queue to be processed when resource is available to do so (i.e. CPU/Memory/etc)

The process does not need to return back to the web route (i.e. its Async)

I know how I would handle this in other languages by using a message queue (Beanstalk, RabbitMQ, sidekiq) and have something that hands the "job" off to a worker and then write a worker to handle the jobs on the queue.

But....I know in Elixir it has queues by default as everything gets pushed into a "mailbox" (saw it in a youtube video :p)

But I cant find any decent code examples that show how to have a controller (in phoenix) spawn an async process that will be processed off the message queue.

Does anyone know of any decent resources that show what I'm looking for, or does anyone have any code snippets that would give me a basic description of how to do it.

Thanks :)

EDIT:

As mentioned in the comments, I've created a basic tutorial for how to get Poolboy up and running in an Elixir app. I hope it is helpful (and correct!) for people :)

https://github.com/thestonefox/elixir_poolboy_example

Answer 1

If the jobs are "important", you still want to use some storage to persist them. Otherwise, if the node goes down for any reason, you lose all jobs that are in the system. If this is the case, you can use something like RabbitMQ and create a bunch of workers in your application:

children = [
  worker(QueueConsumer, [], id: :consumer_1),
  worker(QueueConsumer, [], id: :consumer_2),
  worker(QueueConsumer, [], id: :consumer_3),
  worker(QueueConsumer, [], id: :consumer_4)
]

The consumers would get the next message from RabbitMQ and process them as they come.

If you don't care about persisting them or if you want to regulate them based on other criteria like CPU usage, then I recommend you to look into the jobs library (in Erlang): https://github.com/uwiger/jobs

Answer 2

Sounds like you are describing Poolboy.

http://hashnuke.com/2013/10/03/managing-processes-with-poolboy-in-elixir.html