w3wp.exe Garbage Heap is FULL of Free object space and keeps on growing

Question

My IIS app (.NET 4.0, IIS7) is continually growing in memory, and eventually falls over, as if I have a memory leak.

So I took a DMP while it was up around 1.7GB and cracked it open in WinDbg.

The dumpheap -stat command showed that while I had a fair amount of objects in the heap, the majority (and bit that is growing when looking at sequential DMPs) is labelled as "Free" (>800MB):

000007feef55b768 46091 2212368 System.Data.DataRowView
000007fe9739dda8 10355 2236680 Newtonsoft.Json.Serialization.JsonProperty
000007fef4260610 33062 2644960 System.Signature
000007fef4242250 41809 4682608 System.Reflection.RuntimeMethodInfo
000007fef424f058 69232 8847997 System.Byte[]
000007fef4241b28 11 9437680 System.Double[]
000007fef4237ca0 15 9505176 System.DateTime[]
000007fef424c168 32559 11009308 System.Char[]
000007fef424dc30 17271 11795376 System.Int32[]
000007feef555c48 908 17936672 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]
000007feef554f58 853 22056952 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]
000007feef5514b0 541444 51978624 System.Data.DataRow
000007fef424aee0 1550958 132599872 System.String
000007fef422f1b8 183607 178409288 System.Object[]
0000000000d8b2d0 234017 844500226 Free

So I then ran "!dumpheap -type Free" which gave me a lot of tiny Free objects (94 bytes each to be exact!) and towards the end a couple of larger "Free" objects:

00000003098a59b0 0000000000d8b2d0 134 Free
00000003098c19d0 0000000000d8b2d0 102 Free
00000003098ffa00 0000000000d8b2d0 54 Free
0000000309a41d98 0000000000d8b2d0 5961750 Free
000000041f8a1000 0000000000d8b2d0 24 Free
000000042001b4d0 0000000000d8b2d0 16933078 Free
00000004211bf7c8 0000000000d8b2d0 7702 Free
00000004212c1600 0000000000d8b2d0 35173374 Free
00000004236b3be0 0000000000d8b2d0 66886 Free
0000000423cc41e8 0000000000d8b2d0 10778318 Free
0000000424768928 0000000000d8b2d0 2254734 Free
00000004249ec128 0000000000d8b2d0 21166350 Free
000000042600f1e0 0000000000d8b2d0 51366 Free
000000042621bac8 0000000000d8b2d0 114007238 Free

Statistics:
              MT    Count    TotalSize Class Name
000007fef31e7460        1           32 System.Net.SafeLocalFree
0000000000d8b2d0   234017    844500226      Free
Total 234018 objects

Noting the repeating address of "d8b2d0" I ran a "!gcroot d8b2d0" to which I got the following result:

0:000> !gcroot d8b2d0
Found 0 unique roots (run '!GCRoot -all' to see all roots).

So... after all that... I have masses of Free objects stuck on my heap that the GC is not releasing. This builds up over about 2-3 hours. There is no other sign of any memory leaks from actual typed objects. It happens over almost all of our VMs in production.

Does anyone have any idea how to deal with the GC leaving so much Free object space on the heap?

It's had me running in circles for days. So hat's off to the genius that can help me figure this one out!

Answer 1

Thanks to the suggestion on using PerfView by magicandre1981, and after quite a few very large memory dumps, I've located this issue down to the use of DataRow.DefaultIfEmpty() in a Linq expression. It was part of a support function for carrying out a LeftJoin between two DataTables.

With the DefaultIfEmpty() in the DataRow enumeration as part of the Linq expression, I could recreate memory sky rocketing to 4GB+ in minutes. Most of this survives through to the Gen2 GC collection through sheer volume of ItemArrayObjects being created (both small and large object heap). This in turn lead to an ever climbing committed memory value. The GC2 collection just can't keep up on the small heap, and the large heap suffers from poor (if any) reuse.

We've taken this call out and the same recreation never gets more than 50MB in committed memory. We now need to see what the impact of this is on our DataTable joins, but the memory heap mystery has been solved!

Thanks for the help folks.