Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Will Node.js get blocked when processing large file uploads?

Will Node.js get blocked when processing large file uploads?

Since Node.js only has one thread, is that true that when doing large file uploads all other requests will get blocked?

If so, how should I handle file uploads in nodejs?

like image 638
nilveryboring Avatar asked Apr 10 '14 07:04

nilveryboring


People also ask

Is NodeJS suitable for large applications?

It is suitable for large enterprise projects that do complex and complicated computations and data processing. The comparison in terms of development time between Node. js and Java is that, Node. js is easier to learn than Java, leading to faster development when using Node.

Is NodeJS good for file upload?

Learn more. Show activity on this post. It's said that using Node. js is not advisable for file uploads, because it blocks the IO loop till completion, so I tried to stream-upload many concurrent files, and I found out that Sails.

Does NodeJS block?

Blocking is when the execution of additional JavaScript in the Node. js process must wait until a non-JavaScript operation completes. This happens because the event loop is unable to continue running JavaScript while a blocking operation is occurring.

CAN NodeJS handle high traffic?

Node. js is easily employed as a server-side proxy where it can handle a large amount of simultaneous connections in a non-blocking manner. It's especially useful for proxying different services with different response times, or collecting data from multiple source points.


1 Answers

All the I/O operations is handled by Node.js is using multiple threads internally; it's the programming interface to that I/O functionality that's single threaded, event-based, and asynchronous.

So the big upload of your example is performed by a separate thread that's managed by Node.js, and when that thread completes its work, your callback is put onto the event loop queue.

When you do CPU intensive task it blocks. Let's say we have a task compute() which needs to run almost continuously, and does some CPU intensive calculations.


Answer to the main question "How should I handle file uploads in nodejs?"
Check in your code (or the library) where you save file on the server, is it dependent on writefile() or writeFileSync()?
If it is using writefile() then its asynchronous; But if it is writeFileSync() its is synchronous version.


Updates: In response to a comment:

"the answer "No, it won't block" is correct but explanation is completely wrong. JS is in one thread AND I/O is in one (same) thread. Event loop / asynchronous processing / callbacks make this possible. No multiple threads required. " - by andrey-sidorov

There is no async API for file operations so Node.js uses a thread pool for that. You can see it in the code of libuv. You can look at the source for fs.readFile in lib/fs.js, you’ll see binding.read. Whenever you see binding in Node’s core modules you’re looking at a portal into the land of C++. This binding is made available using NODE_SET_METHOD(target, "read", Read). If you know any C, you might think this is a macro – it was originally, but it’s now a function.

Going back to ASYNC_CALL in Read, one of the arguments is read: the syscall read. But wait, doesn't this function block?

Yes, but that’s not the end of the story. An Introduction to libuv denotes the following:

"The libuv filesystem operations are different from socket operations. Socket operations use the non-blocking operations provided by the operating system. Filesystem operations use blocking functions internally, but invoke these functions in a thread pool and notify watchers registered with the event loop when application interaction is required."

Summary: Node API method writeFile() is asynchronous, but that doesn’t necessarily mean it’s non-blocking underneath. As the libuv book points out, socket (network) code is non-blocking, but filesystems are more complicated. Some things are event-based (kqueue), others use thread pool (as in this case).

Consider going through C code on which Node.js is developed, for more information:

  • Unix fs.c
  • Windows fs.c
like image 186
Amol M Kulkarni Avatar answered Sep 17 '22 15:09

Amol M Kulkarni