Design pattern for threadpooling

Question

I currently have an application that has a large number of threads and this makes the application very large. Each thread is long-running, basically an infinite loop of polling for new emails then handling them. Each thread holds on to one SSL connection, which is why threading works well for the application.

I want to use thread pooling. The simplest approach is to just fix the number of threads then add say 10 users per thread but even at that point it does not seem to be balancing the work as uniformly as 1 user / thread as each loop is fairly long to process. Plus this isn't actually a thread pool.

My question is - what is the proper design pattern here (as it's certainly more intelligent than what I wrote above), and is there a C++ library that handles this well? It would also be helpful to point me to a Java utility as in my experience it's pretty easy to work out a design pattern from a Java utility.

Answer 1

One user/thread is probably a good starting point. Anything that blocks or can be blocked needs to run in its own thread to avoid holding up or being held up by other executing code. If the actual processing of an e-mail is going to run straight through, it can be placed in a "runnable", to use the Java term, and submitted to a thread pool. The Java term would be ThreadPoolExecutor, and it can't be that hard to write a simple one yourself. You'd want one thread for each likely CPU core to get the most work from your computer.

If the work per e-mail is not that great, it might be simpler and faster to skip the thread pool and just do the work on the reader thread. You've paid the cost to run it, you may as well get all the work out of it you can.

At the other extreme, if processing an e-mail involves blocking or getting blocked, you may want to fire up a new thread for each e-mail. You can end up with a lot of threads, but it will get the maximum amount of work out of your computer. You don't want to be falling behind on your e-mails with your CPU usage at only 5%.