How to structure node app to handle cpu intensive tasks?

Question

I've learned that cpu intensive tasks should be moved out of server and server should be responsible for talking to client and responding fast not really for sending out batch emails, or calculating users in database with certain quality or resizing image etc.

so not having done this how can i have my nodejs stay in contact and control processes outside of nodejs code.

For example,

User uploads a million rows big csv file.

I want to analyze it and return.

I could make user wait for nodejs to analyze and get back with result.

Works but doesn't scale what if there are 3000 users, all advantageous of event loop go out of window.

So how can i handle this.

I read that i should let other outside processes to handle this, how? could these processes be written in better faster language. What if i want to stream progress to client (30%) done, this will require communication between process and nodejs.

I think GO and Nodejs make best combo if i can figure this out.

Answer 1

In Node, if you need to perform CPU intensive work you should use a child_process. This will keep the main process free to continue to handle requests and respond to clients while your CSV is being analyzed.

Of course for 3000 users you wouldn't want to spawn 3000 separate processes as this wouldn't scale so designing how to control/create your child processes should all be considerations when building out your Node app.

If you want to report on progress you can use messages between the Parent and Child processes to communicate status of the CSV analyzation with child.send() (Parent to Child Communication) and process.send() (Child to Parent Communication). See this example from the Node Docs for communicating between Parent and Child processes.