Message queue architecture (client to web server to worker and back)

Question

I have a web server written in Node JS running on Heroku. The server has a web server process and a worker process. The web server successfully sends messages to the worker via a RabbitMQ queue; the worker successfully returns the processed data to the web server. I use a randomly generated Uuid to track the messages and ensure the right message is paired up with the right original message.

In a separate project, I have the client (website) successfully communicating with the web server. Now, I need to put the two together.

How can I make it so:

1. Client sends a HTTP POST.
2. Web server receives the request and passes the request into the message queue.
3. Worker processes the request and returns to the web server.
4. Web server returns the correct data to the client so the client can associate the response with the original request.

Step 4 is where I'm stuck. I think I read somewhere that the client should continuously poll (HTTP POSTs?) the web server until it's data is ready. I think I need to respond to the client after step 1 so the request doesn't time out. Any ideas/advice appreciated!

Block diagram:

Answer 1

The short version of what you need to do is two-way messaging. Your web app needs to be a message producer and a message consumer. The same goes for your back-end service.

When the HTTP request comes in, the web server sends a message through RabbitMQ. The back-end picks it up at some point in the future. Meanwhile, the web server sends a response back through the HTTP request saying something is happening and the user will be notified later.

If you're using express, it would look something like this:

var router = express.Router();
router.post("/", postJob);

function postJob(req, res, next){
  req.session.inProgress = true;

  var msg = {
    job: "do some work"
  };

  jobSender.sendJobRequest(msg, function(err){
    if (err) { return next(err); }
    res.render("some-response");
  });
}

This code makes a lot of assumptions, like jobSender being some kind of encapsulated object with a method to send a message across RabbitMQ. I'm sure you can fill in the details of sending a message, based on what you've said already.

The important thing, is that the HTTP request handler sends the message across RabbitMQ and then sends an HTTP response back to the web browser.

At this point, the browser can do whatever it needs to do.

On the back-end, when the other service has completed it's work, it will need to do one of two things:

1) update a shared database somewhere, so that your web server knows what work has been done (and can read a status)

or

2) send a message back to the web server, via rabbitmq

option #1 might not always be a good option, for various reasons. and from your question, you want option #2 anyways.

you'll need a second queue - one that the web server is listening to. when the web server receives a message from this queue, it will update it's own database with the status that is receives.

this status may be "complete" or "in progress" or "error" or something else that you see fit.

For example, if you have a "job status" message, you may have an abstraction called "JobStatusReceiver" to receive the status message.

A simple module like this could receive messages from your job status queue, and update a local database with the status

var JobStatusReceiver = require("./jobStatusReceiver");
var someDataObject = require("someDataObject");

var jobStatus = {

  listen: function(){
    var receiver = new JobStatusReceiver();
    receiver.receive(function(statusMessage){

      someDataObject.status = statusMessage.status;
      someDataObject.data = statusMessage.data;


      someDataObject.save();
    });
  }

};

module.exports = jobStatus;

note that this may be happening in the web server, but it's not part of an HTTP request. the message came in through RabbitMQ with teh JobStatusReceiver, and not part of an HTTP request.

the someDataObject object is most likely going to be an object from yoru database, so it can be saved back to the database.

Lastly, the part where you need to notify the user of the completed action, with the data, can happen in a number of ways.

Generally speaking, it's fairly easy to make an AJAX call to an HTTP API on your web server every few seconds, to look for a valid response.

On the browser side, this could be as simple as:

var timer = setInterval(function(){

  $.ajax({
    url: "/api/check-status",
    success: function(data){
      if (data.complete){
        clearInterval(timer);
        doSomeOtherWork(data);
      }
    })
  });

});

and again in the Express app, handling the "/api/check-status", you would use the same "someDataObject" model to check the status:

var someDataObject = require("someDataObject");

var router = new express.Router();
router.get("/", checkStatus);

function checkStatus(req, res, next){

  someDataObject.load(function(err, someData){
    if (err) { return next(err); }

    res.json({
      complete: (someData.status === "complete"),
      data: someData.data
    });

  });

}

This should hopefully get you down the right path. There are a lot of details I've left out, of course, but hopefully you'll be able to fill in the missing parts.

...

P.S.: I cover all of this, except for the browser checking for status updates on a timer, in my RabbitMQ 4 Devs training course. It's a complete package to get up and running with RabbitMQ and Node.js