Make Meteor.methods Synchronous and Asynchronous

Question

Sometimes I need the Meteor.call to writeMeLater to be queued and executed synchronously(block other calls to writeMeLater from the same client).

Other times the calls to writeMeLater should be executed as soon as possible without queueing up behind all the calls currently queued.

Below is my attempt by using this.unblock() if a async parameter is true. Cases 1 and 2 works fine. But in Case 3, calls with async=true is being queued behind calls with async=false! How can we make calls with async=true skip the queue? This will be similar to how calls from a second client are not queued after calls from the first client,

All Meteor.call() are made from client

Case 1 (correctly synchronous):

Meteor.call('writeMeLater', 's', false)
Meteor.call('writeMeLater', 's', false)
Meteor.call('writeMeLater', 's', false)

Case 2 (correctly asynchronous):

Meteor.call('writeMeLater', 'a', true)
Meteor.call('writeMeLater', 'a', true)
Meteor.call('writeMeLater', 'a', true)

Case 3 (not the desired behavior)

Meteor.call('writeMeLater', 's', false)
Meteor.call('writeMeLater', 's', false)
Meteor.call('writeMeLater', 's', false)

Meteor.call('writeMeLater', 'a', true)
Meteor.call('writeMeLater', 'a', true)
Meteor.call('writeMeLater', 'a', true)

server/main.js

writeMeLater = function(data, callback) {
    console.log('writeMeLater: ', data)

    // simulate taking 3 second to complete
    Meteor.setTimeout(function() {
        Logs.insert({data: data, timestamp: new Date().getTime()})
        console.log('Log.insert: ', data)
        callback(null, 'done')
    }, 3 * 1000)
}



writeMeLaterSync = Meteor._wrapAsync(writeMeLater)



Meteor.methods({

    writeMeLater: function(data, async) {
        if(async)
            this.unblock()

        writeMeLaterSync(data)
    }

})

Answer 1

My first thought is, why not actually create a queue? Instead of relying on the JavaScript event loop as your queue. Then insert documents into that queue, like:

WritesQueue = new Meteor.Collection("WritesQueue");
WritesQueue.insert({data: 'a', prioritize: true, inserted: new Date()});

And maybe every time you insert a high-priority write, trigger Meteor.call("write") which processes the queue, with the prioritized (non-async) ones going first:

Meteor.methods({
  write: function () {
    WritesQueue.find({prioritize: true}, {sort: {inserted: 1}})
    .forEach(function (doc) {
      console.log(doc.data);
      WritesQueue.remove(doc._id);
    });
    WritesQueue.find({prioritize: false}, {sort: {inserted: 1}})
    .forEach(function (doc) {
      console.log(doc.data);
      WritesQueue.remove(doc._id);
    });
  }
});

Or if you want the queue processed every time you insert a high- or low-priority write, either call the write method either time or put the insert inside the write method itself. This solves the jump-to-the-head-of-the-line problem, though the writes are still processed synchronously per client.

As for trying to achieve parallel processing for a single client, @imslavko (in the comments to the question, above) is correct, in that one way to achieve this is for the client to establish multiple DDP connections. There's a relatively simple, albeit hacky and non-Meteoric, way to do that:

Install Iron Router and in your server code, define a server-side route:

Router.map(function () {
  this.route('writeMeLater', {
    where: 'server',
    action: function () {
      Meteor.call('writeMeLater',
        this.request.query.data, this.request.query.async);
    }
  });
});

The this.request.query above is an object with key-value pairs that you submitted with the request. For example:

HTTP.post("http://yoursite.com/writeMeLater",
  {params: {data: 'a', async: true}});

As far as the server knows, this request is coming from a new client, so it will be processed in a new Fiber (i.e. thread). If the writeMeLater method knows not to wait, many instances of it can start running concurrently. Now the issue becomes keeping the requests in order, if it's important to you that the order of execution on the server is the same as on the client, since the HTTP POST requests might not necessarily arrive at the server in the same order as they were sent. But there are various ways of dealing with that too (send them in batches, or include a counter and have the server wait a few seconds if it detects a request out of sequence, etc.).