Getting result of a long running task with RabbitMQ

Question

I have a scenario where a client sends an http request to download a file. The file needs to be dynamically generated and typically takes 5-15 seconds. Therefore I am looking into a solution that splits this operation in 3 http requests.

1. First request triggers the generation of the file.
2. The client polls the server every 5 seconds to check if file is ready to download
3. When the response to the poll request is positive, the client starts downloading the file

To implement this I am looking into Message Queue solutions like RabbitMQ. They seem to provide a reliable framework to run long running tasks asynchronously. However after reading the tutorials on RabbitMQ, I am not sure how will I receive the result of the operation.

Here is what I've in mind:

A front end server receives requests from clients and it posts messages to RabbitMQ as required. This front end server will have 3 endpoints

/generate
/poll
/download

When client invokes /generate with a GET parameter say request_uid=AAA, the front end server will post a message to RabbitMQ with the request_uid in the payload. Any free worker will subsequently receive this message and start generating the file corresponding to AAA.

Client will keep polling /poll with request_uid=AAA to check if task was complete.

When task is complete client will call /download with request_uid=AAA expecting to download the file.

The question is how will the /poll and /download handlers of the front end server will come to know about the status of the file generation job? How can RabbitMQ communicate the result of the task back to the producer. Or do I have to implement such mechanism outside RabbitMQ? (Consumer putting its results in a file /var/completed/AAA)

Answer 1

The easiest way to get started with AMQP, is to use a topic exchange, and to create queues which carry control messages. For instance you could have a file.ready queue and send messages with the file pathname when it is ready to pickup, and a file.error queue to report when you were unable to create a file for some reason. Then the client could use a file.generate queue to send the GET information to the server.