Nodejs - What is maximum thread that can run same time as thread pool size is four?

Question

Requirement is concurrent request 1000 per second and IO operation such as database queries on each request. As nodejs works on event loop and it will assign the IO operation to thread pool, but thread pool default size is 4, so at same time maximum 4 thread (IO operation) can work and rest has to wait in queue. Once any thread complete the execution, they can process.

Query 1 - Can we increase the thread pool size to N number as per requirement and does it improve the performance or it will down the performance?

Query 2 - How we can achieve above requirement in nodejs?

Query 3 - Nodejs is idle choice for this requirement or other suggestion such as golang

Answer 1

Network I/O operations on node.js runs on the main thread.

Yes, node.js spawns four threads in addition to the main thread but none of them are used for network I/O such as database operations. The threads are:

1. DNS resolver (because most OS provide only synchronous API for this)

2. File system API (because this is messy to do asynchronously cross-platform)

3. Crypto (Because this uses the CPU)

4. Zlib (zip compression)

Everything else do not use threads unless you yourself spawn worker_threads. See node's own documentation for more information about this: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/. Do not rely on information not from the node.js project itself such as youtube or medium articles that say node I/O uses thread pools - they don't know what they are talking about.

Increasing thread pool size will not do anything for network I/O because node.js simply does not have any code to make network I/O utilize the extra threads. If you want to spread the load across multiple processors you can use clustering. You can write your own clustering code or use the clustering mode of process managers such as pm2 to pass the connections to your processes.

How can node claim to be high performance if it uses only ONE THREAD!

The thing most non-systems programmers don't realise is that waiting for I/O takes exactly zero CPU time. Doing it by spawning threads means you are allocating lots and lots of RAM and are mostly using zero CPU time for all those threads (imagine spawning 1024 threads each not using the CPU at all). While those threads (or in the case of node.js the main thread) are waiting for 1000 replies from the db the OS queues those requests into a series of packets and send them down to your network card who in turn send them to the database one bit at a time - yes, I/O at it's core is not parallel (unless you use trunking on multiple network cards). So most of the heavy lifting is done by Ethernet while your process is paused by the OS (waiting).

What node.js does is that while the request is waiting it makes another request. This is what non-blocking means. Node does not wait for a request to complete before processing all the other requests. This means by default all requests you make in node.js are concurrent - they don't wait to for other requests to finish.

On the request completion side any responses received from the server triggers node to search the event queue (really it's just a set at this point because any item in the queue can complete at any time) and find the corresponding callback to call. Executing the callback do take CPU time but not waiting for the network request.

This is why systems like node.js can compete with multi-threaded systems. In fact in some cases can outperform multi-threaded systems because doing it on the same thread means you don't need locks (mutexes or semaphores) and you avoid the cost of context switching (when the OS puts one thread to sleep, copying all register values to RAM then wake up another thread copying register values back for the new process from RAM).

Answer 2

Networking in nodejs does not use the threadpool so switching that will not affect your network I/O throughput. Networking uses OS APIs that are already asynchronous and non-blocking.

Your Javascript that runs when an incoming request is processed does not use the thread pool either.

Disk I/O does use the threadpool, but if you're only accessing one physical disk drive, then you may not benefit much from making the threadpool larger because there's only one physical disk servo that can only be in one place at a time anyway so running 20 disk requests in parallel doesn't necessarily help you if they are all competing for the same disk head positioning. In fact, it might even make things worse as the OS tries to timeslice between all the different threads causing the disk head to move around more than optimal to serve each thread.

To serve 1000 requests per second, you will have to benchmark and test and find out where your bottlenecks are. If I had to guess I'd bet the bottleneck would be your database in which case reconfiguring nodejs settings isn't where you need to concentrate your effort. But, in any case, only once you've determined where the bottleneck is in your particular application can you properly figure out what options might help you with that bottleneck. Also, keep in mind that serving 1000 requests/sec means you can't be running Javascript per request that takes more than 1ms for each request. So, you will probably have to cluster your server too (typically one cluster per physical CPU core in your server hardware). So, if you have an 8-core server, you would set up an 8-node cluster.

For example if you are CPU limited in your nodejs process with the running of your own Javascript, then perhaps you want to implement nodejs clustering to get multiple CPUs all running different requests. But, if the real bottleneck is in your database, then cluster your nodejs request handlers won't help with the database bottleneck.

Benchmark, measure, come up with theories based on the data for what to change, design specific tests to measure that, then implement one of the theories and measure. Adjust based on what you measure. You can only really do this properly (without wasting a lot of time in unproductive directions) by measuring first, making appropriate adjustments and then measuring progress.

Answer 3

Network sockets in libuv are non blocking (i.e. not on that threadpool). Build a test harness first. You are more than likely fine with the defaults.

To increase the threadpool size, set the environment variable UV_THREADPOOL_SIZE=N up to 1024.

$ node --help
...
Environment variables:
...
UV_THREADPOOL_SIZE            sets the number of threads used in libuv's threadpool