Analyzing output of !threadpool and !threads in windbg

Question

I have generated dumps on four servers and am analyzing the output of !threadpool and !threads. I noticed the roughly consistent following output:

0:024> !threadpool
CPU utilization 0%
Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 200 MinLimit: 2
Work Request in Queue: 0
Number of Timers: 27
Completion Port Thread:Total: 2 Free: 0 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2

!threads -special
ThreadCount: 32
UnstartedThread: 0
BackgroundThread: 19
PendingThread: 0
DeadThread: 13
Hosted Runtime: no

My questions are:
1)How can I determine what the source of those 27 timers are?
2)What does 13 dead threads mean?
3)One of my threads is marked as having a lock. If i switch to that thread and run !clrstack, i see the following - is it related to my timers?

0:027> !clrstack
OS Thread Id: 0x14cc (27)
*** WARNING: Unable to verify checksum for System.ni.dll
Child-SP         RetAddr          Call Site
000000000ca6e1a0 000007fef5c06477 System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)
000000000ca6e200 000007fef673175b System.Net.TimerThread.ThreadProc()
000000000ca6e2d0 000007fef67c95fd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000000ca6e320 000007fef84d1552 System.Threading.ThreadHelper.ThreadStart()

Answer 1

1)How can I determine what the source of those 27 timers are?

Try to find instances of TimerCallback (for Threading.Timer):

!dumpheap -type TimerCallback

Then dump the callback properties (where callback address is the "Address" from the dumpheap output):

!do <callback address>

Then dump the Value address of the _target property:

!do <_target address>

That should spit out the object that holds a reference to the TimerCallback, which should lead you to where the timer was created.

I recommend checking out Tess Ferrandez's debugging labs, if you haven't already.

2)What does 13 dead threads mean?

My understanding is that a dead thread refers to a C++ thread which no longer has an active OS thread, but still has references and thus cannot be destroyed (C++ threads use ref counting).

A C# thread holds a reference to a C++ thread, and if your managed code keeps a reference to a C# thread, then that could be your problem.

This post on Yun Jin's blog might be of some interest to you.

3)One of my threads is marked as having a lock. If i switch to that thread and run !clrstack, i see the following - is it related to my timers?

That looks like a (System.Threading) timer thread waiting for its interval to elapse.