handling millions of requests/second: how does load balancer(main server thread) works

Question

What will happen: If I write a server application backed with a thread pool of millions of threads and it gets millions of requests per second

I have worked on developing web services. The web service was deployed on 1000's of computers with a front end load balancer. The load balancer's job was to distribute the traffic amongst the servers that actually process the web requests. So my question is that since the process running inside load balancer itself HAS to be single threaded to listen to web requests on a port, how does it handle accepting millions of requests per second. the load balancer might be busy delegating a task, then what happens to the incoming request at that instance of time?

In my opinion, all clients will not be handled since there will only be single request handler thread to pass on the incoming request to the thread pool

This way no multi threaded server should ever work. I wonder how does facebook/amazon handles millions of requests per second.

Answer 1

You are right, it won't work. There is a limit to how much a single computer can process, which is nothing to do with how many threads it is running.

The way Amazon and Facebook etc handle it is to have hundreds or thousands of servers spread throughout the world and then they pass the requests out to those various servers. This is a massive subject though so if you want to know more I suggest you read up on distributed computing and come back if you have specific questions.

Answer 2

With the edit, the question makes much more sense. It is not hard to distribute millions of requests per second. A distribution operation should take somewhat in the viscinity of tens of nanoseconds and would merely consist of pushing the received socket into the queue. No biggie.

As soon as it's done, balancer is ready to accept the next request.