.Net max concurrent timer threads

Question

I am trying to load a queue of interval processes. In other words I have a queue and I want each item in the queue to run on an individual interval.

My problem is that I can't seem to get more than 25 threads to run at one time. I'm using .Net 4.5 on a 64bit machine which has a default max thread count of 32768.

How do I make my app run as many concurrent threads as my machine can handle?

Here is an example app that replicates the actual problem in my production code:

class Program
{
    static void Main(string[] args)
    {
        System.Threading.ThreadPool.SetMaxThreads(200, 200);
        test t = new test();

        t.LoadUrls("http://www.google.com");

        while (1 == 1)
        {
            System.Threading.Thread.Sleep(1000);//refresh every 5 seconds
            Console.WriteLine(System.Diagnostics.Process.GetCurrentProcess().Threads.Count);
        }
    }


    public class test
    {

        public void LoadUrls(string url)
        {
            for (int i = 0; i < 100; i++)
            {
                System.Threading.Timer t = new System.Threading.Timer(new System.Threading.TimerCallback(RunInterval), url, 0, 1000);
                Console.WriteLine("Loaded {0} feeds.", i);
            }
        }
        private static void RunInterval(object state)
        {
            string url = state as string;
            string data = "";

            using (System.Net.WebClient cl = new System.Net.WebClient())
            {
                Console.WriteLine("getting data for " + url);
                data = cl.DownloadString(url);
            }

            //do something with the data

        }
    }
}

This code should theoretically run 198 threads after 2 seconds or so.

By the way, this worked beautifully in my prototype app; it was written in node. But, now I can't get it to work correctly in c#...

ANSWER: The problem was actually with garbage collection and wasn't a threadpool issue at all; the pool is more than capable of spooling all the threads I'm throwing at it. The trick is to use the single parameter constructor of the System.Threading.Timer; this will make the timer use itself as a semaphore, thus avoiding gc.

class Program
{
    static void Main(string[] args)
    {
        for (int i = 0; i < 100; i++)
        {
            test t = new test();
            t.url = "http://www.google.com?" + i;
            System.Threading.Timer ti = new System.Threading.Timer(new System.Threading.TimerCallback(t.RunInterval));
            ti.Change(0, 1000);
        }

        while (1 == 1)
            System.Threading.Thread.Sleep(int.MaxValue);
    }


    public class test
    {
        public string url { get; set; }
        public void RunInterval(object state)
        {
            Console.WriteLine("getting data for " + this.url);
            string data = "";

            using (System.Net.WebClient cl = new System.Net.WebClient())
            {
                data = cl.DownloadString(this.url);
            }
        }
    }
}

I'm not for sure why you would ever want a timer to be collected by the gc, but hey what do I know.

Answer 1

How do I make my app run as many concurrent threads as my machine can handle?

That's the wrong approach. Every thread is costly, create a few hundred and your system will seriously degrade.

Your code uses the ThreadPool. The pool has an algorithm to limit the number of threads. When you increase the Sleep() times you might see a few more threads.

A more direct way would be to set ThreadPool.MinThreads. But expect less performance, not more.

Answer 2

According to System.Threading.Timer

Use a TimerCallback delegate to specify the method you want the Timer to execute. The timer delegate is specified when the timer is constructed, and cannot be changed. The method does not execute on the thread that created the timer; it executes on a ThreadPool thread supplied by the system.

Then in System.Threading.Threadpool

There is one thread pool per process. Beginning with the .NET Framework 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads. The number of threads in the thread pool can be changed by using the SetMaxThreads method. Each thread uses the default stack size and runs at the default priority.

Answer 3

you want to use property called MaxDegreeOfParallelism, but you will need to modify your code a bit. and then test it and figure out the optimal number of threads you want to run in parallel. here is the link: http://msdn.microsoft.com/en-us/library/system.threading.tasks.paralleloptions.maxdegreeofparallelism.aspx