Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Queue tasks in Node

I'm building a node application that will act as a "worker" to scaffold out a new application and then upload the application to AWS. There are 5 tasks that complete the scaffold cycle.

I'd like to know if it's possible in Node/Express to queue incoming requests and then start the scaffold cycle for requests in the queue when the cycle has successfully completed. There should only be one scaffold cycle running at one time.

like image 786
leaksterrr Avatar asked Aug 08 '15 19:08

leaksterrr


People also ask

What is job queue in NodeJS?

A job queue is similar to a message queue and contains an ordered list of jobs to be performed by a separate subsystem. In this post, we will use Kue, a Node. js module for creating a job queue and processing jobs in the background. Kue is backed by Redis and it is designed to be simple to use.

What is the difference between a message queue and a task queue?

A Message Queue is a mechanism for sharing information, between processes, threads, systems. An AppEngine task Queue is a way for an AppEngine application to say to itself, I need to do this, but I am going to do it later, outside of the context of a client request.

What is the use of job queues?

A job queue contains an ordered list of jobs waiting to be processed by a subsystem. The job queue is the first place that a submitted batch job goes before becoming active in a subsystem. The job is held here until a number of factors are met.

Does NodeJS queue request?

node-request-queue is created for request package, which is different than express . You can accomplish the queue using simplest promise queue library p-queue. It has concurrency support and looks much more readable than any other libraries.

What is a queue in Node JS?

A queue is a data structure used in Node.js to appropriately organize asynchronous operations. These operations exist in different forms, including HTTP requests, read or write file operations, streams, and more. Handling asynchronous operations in Node.js can be challenging.

Is it possible to create a task queue system in node?

It’s possible to develop a sophisticated Node task queue system to manage queue data structures in a couple of hundred lines of code. The queue-mongodb module described here uses MongoDB for data storage, but the same concepts could be adopted by any SQL or NoSQL database. The code is available on GitHub and npm.

What is async queue in Node JS?

Node.js async.queue () Method Last Updated : 11 Jun, 2021 The async module is is designed for working with asynchronous JavaScript in NodeJS. The async.queue returns a queue object which is capable of concurrent processing i.e processing multiple items at a single time.

What is the role of Node JS in asynchronous activity?

Note that Node.js is responsible for every asynchronous activity, because JavaScript can block the thread with its single-threaded nature. It is also responsible for adding functions to the callback queues after completing the background operations. JavaScript has nothing to do with the callback queue.


2 Answers

Yes, you can do that. Perhaps your current code looks something like this (I'm assuming you're using promises):

app.get('/', function (req, res) {
  scaffold().then(function() {
    res.send('done');
  });
});

We'll use promise-queue to simplify things. First we need to create a queue:

var queue = new Queue(1);

The argument is the number of items the queue will run concurrently. Since you want no concurrency, we use 1. Now, we need to pass in a promise factory instead of running a promise:

app.get('/', function (req, res) {
  queue.add(function() {
    return scaffold().then(function() {
      res.send('done');
    });
  });
});

If we just passed in a promise, it would start immediately. That' why we must pass in a function that returns the promise.

If you want to respond to the request immediately and not wait for the task to finish, you can move that outside of the promise factory. This also allows us to avoid the extra anonymous function since now scaffold is the promise factory that we want to queue.

app.get('/', function (req, res) {
  queue.add(scaffold);
  res.send('done');
});
like image 139
Aaron Dufour Avatar answered Oct 20 '22 00:10

Aaron Dufour


This isn't the "strictly node.js" answer, but I'd highly recommend looking into Beanstalkd as a way of solving this problem (using fivebeans as the nodejs library).

Put each of your tasks in a named job queue, only pull a new job out when you're ready to work on it, and put the next step in the next queue when it's ready. We use this in production to do a complicated series of transcoding steps every time a new video is uploaded, where each step depends on one or more previous steps having completed.

like image 28
Meekohi Avatar answered Oct 19 '22 22:10

Meekohi