Multithreaded application crashes with errors R6016 or 0xC0000005 at high memory usage

Question

This is multi-threaded console based application being developed in C++ using MSVC2010 on Windows XP. The application consumes high amount of memory. But when memory usage reaches around 2GB (Windows 32bit memory limit) it abruptly crashes at random places with either of these errors:

1. R6016- not enough space for thread data
2. 0xC0000005: Access violation reading location 0x02030000

Accompanied to this, there is also a string appears on console "runtime error" many many times. But not sure from where is it coming (My code does not print it).

Unfortunately there are no definitive stack traces or locations available where it crashes. It just crashes anywhere especially when memory usage goes around 2GB. I would expect CRT to return error or failure when it cannot allocate memory. In that case I've made sure the application would run smooth. But why on this earth does it display message boxes with these errors which led to quit the application.

I am struggling for many days on this and its really unbearable now. I tried:

1) To know the location of crash in code through debugger. But it just crashes anywhere. For e.g. Mostly at new memory allocation calls similar to:

char* ptr = new (std::nothrow) char[1024];

As we can see there is no reason for any application to crash here.

2) To narrow down the code path to find exact scenario under which it crashes: I couldn't succeeded in this also.

As a workaround I thought to not to touch memory usage of my application to 2GB. But then a question came to my mind, "What is the guarantee that it will not crash even at low memory usage. It could be just matter of time?"

Also, there is no definitive way to know in advance how much exact memory is available to allocate.

I am really exhausted out with this bug and desperately need help/guidance.

UPDATE:

I narrowed down that this is problem with QueueUserWorkItem (Windows function that queues a work item to a worker thread in the thread pool). When the call is made under no memory situation, it causes memory allocations inside thread to fail with "runtime error" Unfortunately, I am unable give here sample code reproducing this crash, because I call that function through a third party library and it is very difficult to extract code out of it for demo.

But to explain, I call the function (with flag WT_EXECUTELONGFUNCTION) and when it finishes it invokes the callback from where I call it again (so that it could process next work item).

As a workaround I am trying to put check on memory usage of the application ("Private Bytes") and not letting it to touch 2GB. Hopefully it should work.

Answer 1

There are several components:

1. If you're close to the 2GB addressable user space, then you can actually get out of memory issues...
2. The exception that you get isn't a generic "out of memory" problem. Actually it's a "problem when allocating thread local storage".
3. Memory management may rely on thread local storage.

From what I've found (http://social.msdn.microsoft.com/Forums/en-US/7b63a615-b906-42dc-87cb-da2638ed03ac/r6016-not-enough-space-for-thread-data?forum=vcgeneral):

This error can also happen when you create a thread by using CreateThread or any other function which doesn't use _beginthtread/_beginthreadex. To be precise, you won't get this error when the thread is created, you'll get this error later when you try to access some CRT functionality which requires a per thread data structure.