OutOfMemoryException when a lot of memory is available

Question

We have an application that is running on 5 (server) nodes (16 cores, 128 GB Memory each) that loads almost 70 GB data on each machine. This application is distributed and serves concurrent clients, therefore, there is a lot of sockets usage. Similarly, for synchronization between multiple threads, there are a few synchronization techniques being used, mostly using System.Threading.Monitor.

Now the problem is that while application is running and the data is traveling between these server nodes and between clients and servers, one or two server machines start receiving OutOfMemoryException even though there is 40+% memory still available. We have a feeling that this exception is coming from unmanaged code. Although, we are not directly making any unmanaged calls, we have seen that the last call in the OOM exception stack trace is always a framework call that internally calls unmanaged code.

Following are a couple of examples.

Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Monitor.ObjPulseAll(Object obj)
   ....

Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
   at System.Threading.Monitor.Wait(Object obj, TimeSpan timeout)
   ....

We are clueless here as to what is causing this issue. We have induced GC on these machines multiple times but that also doesn't seem to help.

Any help would be appreciated..

EDIT:

Following are some more details;

• Application is running in x64 process.
• Windows Server 2012 R2
• .NET Framework 4.5
• Server GC enabled
• AllowLargeObject flag is set.

EDIT2: Please note that this is not a memory leak. 70 GB process size is valid here.

Answer 1

Some of the preliminary questions that other users have suggested are cool, but have you considered being lazy and profiling your app?

I can think of Ants profiler from Redgate or dotmemory from JetBrains, links below.

http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/

https://www.jetbrains.com/dotmemory/