Free objects ~50% of .NET heap. How to know what's causing OOM

Question

I am trying to determine what's contributing to my application high memory usage. So I took process dump file. EEHeap command is showing about 2.8 GB in .NET memory heap.

0:000> !EEHeap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (00000000001e3030)
generation 0 starts at 0x00000000a5f21360
generation 1 starts at 0x00000000a5c45a60
generation 2 starts at 0x000000007fff1000
ephemeral segment allocation context: none
 segment     begin allocated  size
000000007fff0000  000000007fff1000  00000000beeca248  0x3eed9248(1055756872)
Large object heap starts at 0x000000027fff1000
 segment     begin allocated  size
000000027fff0000  000000027fff1000  000000028f7d42f8  0xf7e32f8(259928824)
Heap Size:               Size: 0x4e6bc540 (1315685696) bytes.
------------------------------
Heap 1 (00000000001ed9e0)
generation 0 starts at 0x00000001a6c77908
generation 1 starts at 0x00000001a69a8370
generation 2 starts at 0x000000017fff1000
ephemeral segment allocation context: none
 segment     begin allocated  size
000000017fff0000  000000017fff1000  00000001cc127e58  0x4c136e58(1276341848)
Large object heap starts at 0x000000028fff1000
 segment     begin allocated  size
000000028fff0000  000000028fff1000  000000029b55d8f8  0xb56c8f8(190236920)
Heap Size:               Size: 0x576a3750 (1466578768) bytes.
------------------------------
GC Heap Size:            Size: 0xa5d5fc90 (2782264464) bytes.

Strange thing is that the biggest entry for !Dumpheap is the free which is consuming ~1.3 GB

00000000001ac980      388   1373069656      Free

Some of these are in LOH.

0:000> !dumpgen 3 -stat
       Count      Total Size      Type
-------------------------------------------------
           2        233,376   System.Collections.Hashtable+bucket[]
           1        262,168   System.Double[]
           6      1,804,496   System.Int64[]
           1      1,868,972   System.Int32[]
          57     18,707,978   System.Byte[]
          31     69,830,576   System.Object[]
         102    346,151,516   **** FREE ****

There are no objects ready for finalization which tells me that Finalize thread just finished its job.

0:000> !FinalizeQueue
SyncBlocks to be cleaned up: 0
Free-Threaded Interfaces to be released: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
----------------------------------
------------------------------
Heap 0
generation 0 has 2748 finalizable objects (00000000047b6518->00000000047bbaf8)
generation 1 has 55 finalizable objects (00000000047b6360->00000000047b6518)
generation 2 has 6878 finalizable objects (00000000047a8c70->00000000047b6360)
Ready for finalization 0 objects (00000000047bbaf8->00000000047bbaf8)
------------------------------
Heap 1
generation 0 has 69 finalizable objects (0000000004807160->0000000004807388)
generation 1 has 6 finalizable objects (0000000004807130->0000000004807160)
generation 2 has 7818 finalizable objects (00000000047f7ce0->0000000004807130)
Ready for finalization 0 objects (0000000004807388->0000000004807388)

LOH seem to be heavily fragmented.

0:000> !HeapStat
Heap             Gen0         Gen1         Gen2          LOH
Heap0       419073768      2996480    633686624    259928824
Heap1       625673552      2946456    647721840    190236920
Total      1044747320      5942936   1281408464    450165744

Free space:                                                 Percentage
Heap0       397468560           24      1545984    183287672SOH: 37% LOH: 70%
Heap1       611153984           24     16750104    162864048SOH: 49% LOH: 85%
Total      1008622544           48     18296088    346151720

I don't know if I am connecting all the dots here, but my thought is that in next GC, all the free object will be removed from memory and .NET heap memory should go down based on Free objects that are in Gen 0, 1, 2 (because those will be compressed). Is it a fair statement or do you see any other issues that I may be missing? Of course LOH fragmentation is a issue but I don't know how much that account for application utilization.

EDIT: Applicaton has started failing with OOM now. Based on some comments below, memory reported against Free object doesn't go back to OS. How can I know what is causing OOM

Answer 1

The Large Object Heap does not do automatic defragmentation, so it is indeed possible to get an OOM while you have lots of free space on the LOH. There is no easy way out of this hole. There are three general approaches:

1) Reduce the amount of LOH (>85,000 bytes) allocations your application performs, pool large objects instead of letting them become collected, and try to perform a small number of very big allocations instead of a large number of medium ones.

2) Use .NET 4.5.1's LOH compaction. It's a stopgap, not a true solution, but if fragmentation is the only memory-related problem you have, it can help for now.

3) Move to a 64-bit process. You will still have a fragmentation problem, but you will not run out of memory so easily.