epoll performance

Question

Can anyone please help me to answer the questions about epoll_wait.

1. Is it overkill to use many threads that call epoll_wait on the same fds set to serve at about 100K active sockets? or will it just be enough to create only 1 thread to perform epoll_wait?

2. How many threads will wake up from epoll_wait when for example only one socket is ready to read data? i mean, can there be situation when 2 or more threads will wake up from epoll_wait but will have same fds in resulted events?

3. What is the best way to organize threads in server that works with many active clients (e.g. 50K+). The best way i think is: 1 I/O Worker Thread which perfroms epoll_wait and i/o operations. + Many Data processing threads which will process the data received from I/O worker thread (can take a long time, such as any game logic) and compose new data for I/O worker thread to send to client. Am I right in this approach, or can anyone help me to find out the best way to organize this?

Thanks in advance, Valentin

Answer 1

1. When using epoll, you want to size your thread total to the number of physical CPU cores (or hyperthread dispatch units) which you want to use for processing. Using only one thread for work means that at most one core will be active at a time.

2. It depends on the mode of the epoll file descriptor. Events can be "edge triggered", meaning that they only happen once atomically, or "level triggered" meaning that any caller gets an event if there is space in the buffer.

3. Not enough information to say. I'd suggest not having special purpose threads at all, for simplicity, and simply handling each event's "command" in the thread in which it is received. But obviously that depends on the nature of your application.

Answer 2

I recommend you this reading from 2006: http://www.kegel.com/c10k.html