Multithreaded network crawler threads limit

Question

I have access to an application written in .NET C# which connects (using mainly "raw" http requests, partially web services and xml requests, still over http) to numerous external systems and updates some stuff in them.

There can be alot of work queued at a given time and the naive approach to increase the throughput was to increase the threads count. The logic behind it was: since most of the time we are waiting for network replies, we can simultaneously wait for more network replies. The cpu and ram does not seem to be raching their limits.

Still creating around 300 threads makes everything work slower than with lower thread counts.

I am wondering is it an operating system limit (windows server 2012 r2), the .NET (4.5) limit or something else? How can I diagnose where is the bottleneck? (as I said the cpu and ram does not seem to be the problem)

I know the external systems might get overloaded and decrease overall performance, but lets assume this is negligible.

Answer 1

Creating a thread requires some CPU and RAM, 300 thread creations allocate at least 1 MB per thread plus stack allocations plus some other stuff.

You should use the thread pool for that. The threads in the pool are already created and are waiting to serve you.

In case of a long waiting for the network response you could use the asynchronous IO algorythm which wont require many threads.