Menu
Our WCF service hosted in IIS crashes (w3wp.exe ~ 1.6 GB) as the user load increases. We have got a dump through Debug Diag and ran this command in WinDbg. This is the output:
0:000> !address -summary
Failed to map Heaps (error 80004005)
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unclassified> 6007 57a86000 ( 1.370 Gb) 85.37% 68.48%
Free 268 19519000 ( 405.098 Mb) 19.78%
Image 874 80da000 ( 128.852 Mb) 7.84% 6.29%
Stack 8022 64ce000 ( 100.805 Mb) 6.14% 4.92%
TEB 2674 a72000 ( 10.445 Mb) 0.64% 0.51%
NlsTables 1 24000 ( 144.000 kb) 0.01% 0.01%
ActivationContextData 4 b000 ( 44.000 kb) 0.00% 0.00%
CsrSharedMemory 1 7000 ( 28.000 kb) 0.00% 0.00%
PEB 1 1000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 16360 5dd0f000 ( 1.466 Gb) 91.37% 73.30%
MEM_IMAGE 1188 8454000 ( 132.328 Mb) 8.05% 6.46%
MEM_MAPPED 36 974000 ( 9.453 Mb) 0.58% 0.46%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT 14613 51e17000 ( 1.279 Gb) 79.75% 63.97%
MEM_FREE 268 19519000 ( 405.098 Mb) 19.78%
MEM_RESERVE 2971 14cc0000 ( 332.750 Mb) 20.25% 16.25%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 8423 475ea000 ( 1.115 Gb) 69.51% 55.76%
PAGE_EXECUTE_READ 155 69f1000 ( 105.941 Mb) 6.45% 5.17%
PAGE_READWRITE|PAGE_GUARD 5319 1f24000 ( 31.141 Mb) 1.90% 1.52%
PAGE_READONLY 364 11bc000 ( 17.734 Mb) 1.08% 0.87%
PAGE_WRITECOPY 179 a16000 ( 10.086 Mb) 0.61% 0.49%
PAGE_EXECUTE_READWRITE 109 1fd000 ( 1.988 Mb) 0.12% 0.10%
PAGE_EXECUTE_WRITECOPY 64 149000 ( 1.285 Mb) 0.08% 0.06%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unclassified> 66f0000 c041000 ( 192.254 Mb)
Free 71c97000 4109000 ( 65.035 Mb)
Image 51b0e000 da4000 ( 13.641 Mb)
Stack c90000 3d000 ( 244.000 kb)
TEB 7f370000 1000 ( 4.000 kb)
NlsTables 7ffb0000 24000 ( 144.000 kb)
ActivationContextData 70000 5000 ( 20.000 kb)
CsrSharedMemory 7f6f0000 7000 ( 28.000 kb)
PEB 7ffd4000 1000 ( 4.000 kb)
What I don't understand is unclassified. They are all huge! I ran !address and again got lot of unclassified. I am a beginner in WinDbg so don't know how to proceed from here with regards to memory leak.
!heap -s and !gchandles gives the following output and these numbers look very small.
0:000> !gchandles
GC Handle Statistics:
Strong Handles: 101224
Pinned Handles: 23
Async Pinned Handles: 2
Ref Count Handles: 4
Weak Long Handles: 0
Weak Short Handles: 0
Other Handles: 0
Statistics:
MT Count TotalSize Class Name
67370460 1 12 System.Web.Hosting.AppManagerAppDomainFactory
79b5f8b4 1 16 System.Threading.RegisteredWaitHandle
7aa02d34 1 20 System.Net.TimerThread+TimerQueue
79b76668 1 20 System.Threading._ThreadPoolWaitOrTimerCallback
79b9a934 1 24 System.Threading.AutoResetEvent
79ba1b18 1 28 System.SharedStatics
6733fc00 3 36 System.Web.Hosting.ISAPIRuntime
79b9f5e8 5 60 System.Object
79b9fde8 1 84 System.ExecutionEngineException
79b9fd9c 1 84 System.StackOverflowException
79b9fd50 1 84 System.OutOfMemoryException
79b9fc0c 1 84 System.Exception
7988736c 2 136 System.Threading.OverlappedData
79b9fe34 2 168 System.Threading.ThreadAbortException
79ba3670 10 360 System.Security.PermissionSet
79ba0ec8 4 448 System.AppDomain
79b88df4 19 1520 System.Runtime.Remoting.ServerIdentity
6736eb50 54 1728 System.Web.NativeFileChangeNotification
67375ecc 489 95844 System.Web.HttpContext
79b9ffcc 2645 126960 System.Threading.Thread
79b56c28 10536 683588 System.Object[]
79baa808 87474 1749480 System.Threading._TimerCallback
Total 101253 objects
0:000> !heap -s
LFH Key : 0x866023d1
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 020b0000 - 020b0000 (size 00000000)
Virtual block: 02250000 - 02250000 (size 00000000)
00080000 00000002 65536 42512 42512 1765 248 1 2 21d36 L
00180000 00008000 64 12 12 10 1 1 0 0
00260000 00001002 1088 628 628 24 0 1 0 0 L
00500000 00000002 1024 20 20 2 1 1 0 0 L
00640000 00001002 256 28 28 1 1 1 0 0 L
00680000 00001002 256 12 12 4 1 1 0 0 L
006c0000 00001002 256 12 12 4 1 1 0 0 L
00700000 00001002 1280 1172 1172 42 10 1 0 0 L
00740000 00001002 256 12 12 4 1 1 0 0 L
00ad0000 00001002 1280 296 296 2 1 1 0 0 L
00b10000 00001002 256 12 12 4 1 1 0 0 L
00b50000 00001002 256 12 12 4 1 1 0 0 L
00b90000 00001002 256 12 12 4 1 1 0 0 L
00bd0000 00001002 256 12 12 4 1 1 0 0 L
00c10000 00001002 256 12 12 4 1 1 0 0 L
00c50000 00001002 256 12 12 4 1 1 0 0 L
00dc0000 00001002 256 12 12 4 1 1 0 0 L
00e00000 00001002 256 12 12 4 1 1 0 0 L
00e40000 00001002 256 12 12 4 1 1 0 0 L
00e80000 00001002 256 12 12 4 1 1 0 0 L
00ec0000 00001002 256 12 12 4 1 1 0 0 L
00f00000 00001002 256 12 12 4 1 1 0 0 L
01050000 00001002 7424 4628 4628 14 4 1 0 0 L
01090000 00001002 256 36 36 1 1 1 0 0 L
01110000 00001002 7232 3192 3192 70 41 1 0 0 L
01120000 00001002 256 12 12 4 1 1 0 0 L
01160000 00001002 256 176 176 2 1 1 0 0 L
011a0000 00001002 256 24 24 3 1 1 0 0 L
012e0000 00001002 256 12 12 3 1 1 0 0 L
01320000 00001002 256 12 12 4 1 1 0 0 L
01360000 00001002 256 12 12 4 1 1 0 0 L
013a0000 00001002 256 12 12 4 1 1 0 0 L
013e0000 00001002 256 72 72 3 1 1 0 0 L
01420000 00001002 256 12 12 4 1 1 0 0 L
01460000 00001002 256 168 168 3 1 1 0 0 L
014a0000 00001002 256 12 12 4 1 1 0 0 L
014e0000 00001002 256 12 12 4 1 1 0 0 L
01520000 00001002 256 12 12 4 1 1 0 0 L
01560000 00001002 256 12 12 4 1 1 0 0 L
01670000 00001002 256 12 12 4 1 1 0 0 L
016b0000 00001002 256 12 12 4 1 1 0 0 L
016f0000 00001002 256 12 12 4 1 1 0 0 L
01730000 00001002 256 12 12 4 1 1 0 0 L
01770000 00001002 256 12 12 4 1 1 0 0 L
017b0000 00001002 256 12 12 4 1 1 0 0 L
017f0000 00001002 256 12 12 4 1 1 0 0 L
01830000 00011002 256 12 12 4 1 1 0 0 L
01870000 00001002 1088 120 120 7 1 1 0 0 L
01880000 00001002 64 12 12 4 1 1 0 0 L
01890000 00001002 256 12 12 4 1 1 0 0 L
019d0000 00001002 256 12 12 4 1 1 0 0 L
01a10000 00001002 256 116 116 0 0 1 0 0 LFH
01a50000 00001002 256 12 12 4 1 1 0 0 L
01b50000 00001002 256 200 200 4 0 1 0 0 L
01bb0000 00001002 3136 1500 1500 69 41 1 0 0 L
01bc0000 00041002 256 12 12 4 1 1 0 0 L
01c00000 00001002 3136 1500 1500 69 41 1 0 0 L
01c10000 00041002 256 12 12 4 1 1 0 0 L
01c50000 00041002 256 16 16 1 1 1 0 0 L
01e10000 00041002 256 64 64 1 1 1 0 0 L
01ed0000 00001002 1088 424 424 20 1 1 0 0 L
1af40000 00001002 256 152 152 0 0 1 0 0 L
1b060000 00001002 64 48 48 40 2 1 0 0 L
1bab0000 00001002 1280 688 688 28 2 2 0 0 L
-----------------------------------------------------------------------------
!eeheap -gc gives the following output:
0:000> !eeheap -gc
Number of GC Heaps: 4
------------------------------
Heap 0 (000e9678)
generation 0 starts at 0x3415e208
generation 1 starts at 0x340b6894
generation 2 starts at 0x026f0038
ephemeral segment allocation context: none
segment begin allocated size
026f0000 026f0038 066eda74 0x3ffda3c(67099196)
336c0000 336c0038 3471a7c4 0x105a78c(17147788)
Large object heap starts at 0x126f0038
segment begin allocated size
126f0000 126f0038 1272cd38 0x3cd00(249088)
Heap Size: Size: 0x5094ec8 (84496072) bytes.
------------------------------
Heap 1 (000ea8b0)
generation 0 starts at 0x39fd9748
generation 1 starts at 0x39f13224
generation 2 starts at 0x066f0038
ephemeral segment allocation context: none
segment begin allocated size
066f0000 066f0038 0a6ef668 0x3fff630(67106352)
38e00000 38e00038 3a63e8bc 0x183e884(25421956)
Large object heap starts at 0x146f0038
segment begin allocated size
146f0000 146f0038 146f0048 0x10(16)
Heap Size: Size: 0x583dec4 (92528324) bytes.
------------------------------
Heap 2 (000ebdb8)
generation 0 starts at 0x3087fedc
generation 1 starts at 0x307e8868
generation 2 starts at 0x0a6f0038
ephemeral segment allocation context: none
segment begin allocated size
0a6f0000 0a6f0038 0e6ee9f4 0x3ffe9bc(67103164)
2f3b0000 2f3b0038 31acc084 0x271c04c(41009228)
Large object heap starts at 0x166f0038
segment begin allocated size
166f0000 166f0038 166f0048 0x10(16)
Heap Size: Size: 0x671aa18 (108112408) bytes.
------------------------------
Heap 3 (000ed4d0)
generation 0 starts at 0x6d1c892c
generation 1 starts at 0x6d110038
generation 2 starts at 0x0e6f0038
ephemeral segment allocation context: none
segment begin allocated size
0e6f0000 0e6f0038 126ee814 0x3ffe7dc(67102684)
2b1a0000 2b1a0038 2d14311c 0x1fa30e4(33173732)
6d110000 6d110038 6d75f814 0x64f7dc(6617052)
Large object heap starts at 0x186f0038
segment begin allocated size
186f0000 186f0038 186f0048 0x10(16)
Heap Size: Size: 0x65f10ac (106893484) bytes.
------------------------------
GC Heap Size: Size: 0x175de850 (392030288) bytes.
878
About unclassified, a lot of posts on the Internet show that in late versions of WinDBG unclassified entries has just replaced the things that were mapped to different regions before. In previous versions of debugger you had these RegionUsageIsVAD, RegionUsageImage.
On my side, I also have a lot or unclassified entries in !address -summary output, but it doesn't prevent me from future debugging.
No, go back to your other results:
According to my experience with WinDBG, if eeheap shows ~300Mb of memory when MEM_COMMIT gives 1.3Gb, it could be a native memory leak.
See CodeProject on how you could catch it.
Note, that you possibly won't have stacks when running !heap -p -a, so you need to run your process with proper gflags before debug. You can read more about it
Then, you may possibly got another, simpler situation with repeated strings. I've run into such situation one time and described it.
161
WCF (Windows Communication Foundation) is a .NET technology. .NET does not make use of the Windows Heap Manager, so using any !heap command will not work for .NET. No wonder that it's small.
Instead, the .NET framework has its own memory manager due to garbage collection. You already used the correct command !eeheap -gc to display information about the .NET heaps. The last line of that command says:
GC Heap Size: Size: 0x175de850 (392030288) bytes.
So .NET is responsible for almost 400 MB of <unclassified> (<unknown>) memory. The rest could be caused by
VirtualAlloc() of courseIf these values remain constant, there's nothing to worry about. But if it increases, this might indicate a leak. I personally consider those 87000 timer callbacks as a potential source. What are all those timer callbacks doing? Are they ever able to finish their work?
79baa808 87474 1749480 System.Threading._TimerCallback
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
