Debugging I/O completion threads leak

Question

I'm trying to find a possible I/O completion threads leak in a .NET app. From time to time I'm sampling the number of available I/O completion threads using ThreadPool.GetAvailableThreads. At some point the number of available threads starts to degrade from 1000 to 1 (in several minutes).

I don't see any memory leak, errors, delay in execution or anything like it once it gets to 1 available thread.

In windbg, when running !threads I see ~700 threads, non of the them comes from the threadpool, most of them are marked with XXXX (as far as I know it means the GC didn't collect the thread yet), for example: XXXX 2a4 1630 37d54610 1400 Enabled 00000000:00000000 004b70d8 0 Ukn (or MTA)

How can debug this one? how can I see the callstacks of the completion threads?

When running !threadpool, I get the following:

CPU utilization: 31% Worker Thread: Total: 2047 Running: 2047 Idle: 0 MaxLimit: 2047 MinLimit: 2 Work Request in Queue: 0 -------------------------------------- Number of Timers: 4 -------------------------------------- Completion Port Thread:Total: 1000 Free: 1 MaxFree: 4 CurrentLimit: 999 MaxLimit: 1000 MinLimit: 2

However,

Answer 1

Run !sos.name2ee mscorlib.dll System.Threading.Thread and make a note of the MethodTable. Then run !sos.dumpheap -mt [MT]. This will give you a list of the thread objects. Then run !sos.gcroot [THREADOBJ] to see what's holding on to them.