TaskFactory.StartNew -> System.OutOfMemoryException

Question

There around 1000 task running but sometimes i receive the following aout of memory exception thrown by task scheduler. What could be the reason and how to avoid it.

System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(Object parameter)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Object action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
   at System.Threading.Tasks.TaskFactory.StartNew(Action action, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler)
   at App.StartReadSocketTask()

Answer 1

Your (non x64) App has a max memory space of 2GB. Each Thread requires a minimum of 1 MB, typically you can expect OOM before you reach 1000 Threads.

In itself the Task class is supposed to address this (by using the ThreadPool). But when your Tasks take too long (> 500 ms) the TP will slowly add Threads, failing after a few minutes or longer.

The simplest solution might be to look in your code where this unbounded creation of Tasks occurs, and see if you can limit in a way that agrees with your solution. Like if you are using a Producer/Consumer Que, make it a bounded queue.

Otherwise, limit the MaxThreads but this is a blunt, application-wide instrument.

Answer 2

As i was experimenting with testing the limits of the parallel system i encountered this very issue myself. oleksii's comment is spot on (1k threads ~= 1GB of committed memory). It is important to note that this memory is reserved virtual address space NOT the amount of memory actually 'used'. Out of memory exceptions occur when the system cannot commit a contiguous chunk of virtual address space large enough to satisfy your request (insert 'memory fragmentation' rhetoric here). If you view the process in the windows task manager about the time it dies you may see as little as 80-120mb of 'used' memory. To see how much virtual address space is reserved show the "Memory - Commit Size" column in task manager.

To keep this short i was able to break past the ~1k thread limit by switching my build config from x86 to 64 bit. This increases the amount of virtual address space available from (roughly) 2GB to 6TB+ (depending on your OS version) and my OutOfMemoryException went away.

Here's the simple program I created that illustrates this artifact, be sure to run it as x86 and watch it die somewhere between 1k and 1.5k threads - then switch to 64 bit and it should run to completion without failing.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Threading;

namespace TaskToy
{
    class Program
    {
        static void Main( string[] args )
        {
            List<Task> lTasks = new List<Task>();
            int lWidth = 0;
            for ( int i = 0; i < 5000; i ++ )
            {
                lTasks.Add( new Task( (o) => {

                    Console.WriteLine( "B " + Interlocked.Increment( ref lWidth ) + " tid " + Thread.CurrentThread.ManagedThreadId );
                    Thread.Sleep( 60000 );
                    Console.WriteLine( "E " + Interlocked.Decrement( ref lWidth ) + " tid " + Thread.CurrentThread.ManagedThreadId );
                }, null, TaskCreationOptions.LongRunning ) );
            }

            Parallel.For( 0, lTasks.Count, ( i ) =>
            {
                lTasks[i].Start();
            } );

            Task.WaitAll( lTasks.ToArray() );
            Console.WriteLine( "DONE - press any key..." );
            Console.ReadKey( true );
        }
    }
}

P.S. The 'lWidth' variable indicates the current level of concurrency, that is, how many tasks are actually running at once.

Overall this was a fun academic experiment but it will likely be a 'few' years before running thousands of threads provides a useful return. It is likely advisable to limit the number of threads you spin up to something more practical - likely an order of magnitude less than 'thousands'.