.Net app deadlock and GC

Question

I'm working on a .net system, which under some special conditions (.net 2.20 AND .net 4.0 installed) generates a deadlock in one of (win) services. Using WinDbg i could find out that:

In thread 7:

0:007> !clrstack
OS Thread Id: 0xfec (7)
ESP       EIP     
03c9dbd8 7c90e514 [HelperMethodFrame_PROTECTOBJ: 03c9dbd8] System.Reflection.Assembly._nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)
03c9dc70 79400e07 System.Reflection.Assembly.nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)
03c9dc90 7935c5e0 System.Reflection.Assembly.InternalLoad(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef, Boolean)
03c9dcb8 7939b788 System.Reflection.Assembly.InternalLoadFrom(System.String, System.Security.Policy.Evidence, Byte[], System.Configuration.Assemblies.AssemblyHashAlgorithm, Boolean, System.Threading.StackCrawlMark ByRef)
03c9dcdc 7939b725 System.Reflection.Assembly.LoadFrom(System.String, System.Security.Policy.Evidence)
03c9dce4 79398fdb System.Activator.CreateInstanceFrom(System.String, System.String, Boolean, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, System.Object[], System.Security.Policy.Evidence)
03c9dd10 79399653 System.AppDomain.CreateInstanceFrom(System.String, System.String)
03c9dd1c 793b8e39 System.AppDomain.CreateInstanceFromAndUnwrap(System.String, System.String)
03c9df48 79e7c0e3 [GCFrame: 03c9df48] 
03c9df64 79e7c0e3 [GCFrame: 03c9df64] 
03c9e0d4 79e7c0e3 [ContextTransitionFrame: 03c9e0d4] 
03c9e1d8 79e7c0e3 [GCFrame: 03c9e1d8] 
03c9e1bc 79e7c0e3 [GCFrame: 03c9e1bc] 
03c9f43c 79e7c0e3 [CustomGCFrame: 03c9f43c] 
03c9f450 79e7c0e3 [CustomGCFrame: 03c9f450] 
03c9f464 79e7c0e3 [CustomGCFrame: 03c9f464] 
03c9f478 79e7c0e3 [CustomGCFrame: 03c9f478] 
03c9f48c 79e7c0e3 [CustomGCFrame: 03c9f48c] 
03c9f4a0 79e7c0e3 [CustomGCFrame: 03c9f4a0] 
03c9f4b4 79e7c0e3 [CustomGCFrame: 03c9f4b4] 
03c9f880 79e7c0e3 [TPMethodFrame: 03c9f880] System.AppDomain.CreateInstanceFromAndUnwrap(System.String, System.String)
03c9f894 00a64498 Roche.RIG.CLink.DomainObject.Create[[System.__Canon, mscorlib]](System.String, Roche.RIG.CLink.CoreSystem.Interfaces.IBaseCoreSystem, Roche.RIG.CLink.TypeLifeManagementAction)
03c9f8dc 00acf220 Roche.RIG.SmartCom.DomainMgr.Load[[System.__Canon, mscorlib]](System.String)
03c9f92c 00acf075 Roche.RIG.SmartCom.DomainMgr.GetTunnel()
03c9f968 00acef05 Roche.RIG.SmartCom.CoreSystem.StartupTunnel(System.Object)
03c9f9b4 793e25cf System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
03c9f9bc 79366b3d System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
03c9f9d4 793e2734 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
03c9f9e8 793e26ac System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
03c9fb78 79e7c0e3 [GCFrame: 03c9fb78]

and in the other thread (thread 3):

0:003> !clrstack
OS Thread Id: 0xb08 (3)
ESP       EIP     
00d4eac4 7c90e514 [GCFrame: 00d4eac4] 
00d4faa4 7c90e514 [PrestubMethodFrame: 00d4faa4] Roche.RIG.SmartCom.CoreSystem.StartupTasks(System.Object)
00d4fab4 793e25cf System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
00d4fabc 79366b3d System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00d4fad4 793e2734 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
00d4fae8 793e26ac System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
00d4fc78 79e7c0e3 [GCFrame: 00d4fc78]

Output of !locks

0:007> !locks

CritSec +157d20 at 00157d20
LockCount          1
RecursionCount     1
OwningThread       fec
EntryCount         1
ContentionCount    1
*** Locked

CritSec wininet!GlobalCacheCritSect+0 at 7723ba38
LockCount          4
RecursionCount     4
OwningThread       b08
EntryCount         1
ContentionCount    1
*** Locked

CritSec +1a1310 at 001a1310
LockCount          0
RecursionCount     1
OwningThread       b08
EntryCount         0
ContentionCount    0
*** Locked

Scanned 1212 critical sections

Output of syncblk

0:003> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
-----------------------------
Total           370
CCW             1
RCW             319
ComClassFactory 0
Free            0

Output of ~*kv 90

    0:007> ~*kv 90

       0  Id: ebc.edc Suspend: 1 Teb: 7ffdd000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0012f090 7c90d9da 7c801879 000001a4 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    0012f094 7c801879 000001a4 00000000 00000000 ntdll!NtReadFile+0xc (FPO: [9,0,0])
    0012f0fc 77df348b 000001a4 0012f1c8 00000216 kernel32!ReadFile+0x16c (FPO: [Non-Fpo])
    0012f128 77df331f 000001a4 0012f1c8 00000216 advapi32!ScGetPipeInput+0x2a (FPO: [Non-Fpo])
    0012f19c 77df3628 000001a4 0012f1c8 00000216 advapi32!ScDispatcherLoop+0x3f (FPO: [Non-Fpo])
    0012f3fc 006ea31e 0019c3f8 a82bfd52 00000000 advapi32!StartServiceCtrlDispatcherW+0xe3 (FPO: [Non-Fpo])
    *** WARNING: Unable to verify checksum for System.ServiceProcess.ni.dll
    WARNING: Frame IP not in any known module. Following frames may be wrong.
    0012f418 67a241d2 0106174c 01061ab0 0019c400 0x6ea31e
    00000000 00000000 00000000 00000000 00000000 System_ServiceProcess_ni+0x41d2

       1  Id: ebc.d1c Suspend: 1 Teb: 7ffdc000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0082fe38 7c90df4a 7c809590 00000003 0082fe64 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    0082fe3c 7c809590 00000003 0082fe64 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
    0082fed8 7c80a115 00000003 0082ff1c 00000000 kernel32!WaitForMultipleObjectsEx+0x12c (FPO: [Non-Fpo])
    0082fef4 79ed25eb 00000003 0082ff1c 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
    0082ff54 79ed2548 a8b35c0c 00000000 79ece48c mscorwks!DebuggerRCThread::MainLoop+0xe9 (FPO: [Non-Fpo])
    0082ff84 79ed2477 a8b35c3c 79eb4145 79ece48c mscorwks!DebuggerRCThread::ThreadProc+0xe5 (FPO: [Non-Fpo])
    0082ffb4 7c80b729 00000000 79eb4145 79ece48c mscorwks!DebuggerRCThread::ThreadProcStatic+0x9c (FPO: [Non-Fpo])
    0082ffec 00000000 79ed2431 00000000 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

       2  Id: ebc.104 Suspend: 1 Teb: 7ffdb000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    009ffcd0 7c90df4a 7c809590 00000002 009ffcfc ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    009ffcd4 7c809590 00000002 009ffcfc 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
    009ffd70 7c80a115 00000002 7a3b9248 00000000 kernel32!WaitForMultipleObjectsEx+0x12c (FPO: [Non-Fpo])
    009ffd8c 79e719ab 00000002 7a3b9248 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
    009ffdac 79ef8cd9 0016e5d0 009ffeb0 0016edc8 mscorwks!WKS::WaitForFinalizerEvent+0x77 (FPO: [Non-Fpo])
    009ffdc0 79ebab37 009ffeb0 00000000 00000000 mscorwks!WKS::GCHeap::FinalizerThreadWorker+0x79 (FPO: [Non-Fpo])
    009ffdd4 79ebaad3 009ffeb0 009ffe5c 79f495f5 mscorwks!Thread::DoADCallBack+0x32a (FPO: [Non-Fpo])
    009ffe68 79eba9f7 009ffeb0 a8ae5d2c 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 (FPO: [Non-Fpo])
    009ffea4 79ebbe48 009ffeb0 00000000 00158500 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a (FPO: [Non-Fpo])
    009ffecc 79ebbe12 79ef8c6c 00000008 009fff14 mscorwks!ManagedThreadBase_NoADTransition+0x32 (FPO: [Non-Fpo])
    009ffedc 79ebb42b 79ef8c6c a8ae5c9c 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd (FPO: [Non-Fpo])
    009fff14 79ebaf8e 00000000 8706da80 804fb078 mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb (FPO: [Non-Fpo])
    009fffb4 7c80b729 0016edc8 00730074 00610020 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [Non-Fpo])
    009fffec 00000000 79ebaf48 0016edc8 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

       3  Id: ebc.b08 Suspend: 1 Teb: 7ffd9000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    00d4d7a0 7c90df5a 7c91b24b 00000610 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
    00d4d7a4 7c91b24b 00000610 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
    00d4d82c 7c901046 00157d20 7a08a990 00157d20 ntdll!RtlpWaitForCriticalSection+0x132 (FPO: [Non-Fpo])
    00d4d834 7a08a990 00157d20 79e83b8f 00157d20 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0])
    00d4d83c 79e83b8f 00157d20 a8e57bf0 001dac60 mscorwks!UnsafeEEEnterCriticalSection+0x18 (FPO: [1,0,0])
    00d4d878 79e75c62 00d4d88c 79e75c56 00157d20 mscorwks!CrstBase::Enter+0x13c (FPO: [Non-Fpo])
    00d4d880 79e75c56 00157d20 00d4d89c 79e75c4a mscorwks!EEEnterCriticalSection+0xb (FPO: [Non-Fpo])
    00d4d88c 79e75c4a 7a3c38b0 00157d20 00d4df10 mscorwks!CExecutionEngine::AcquireLock+0xb (FPO: [Non-Fpo])
    00d4d89c 79eec124 00157d20 8000000a 7a195ccc mscorwks!ClrEnterCriticalSection+0x11 (FPO: [Non-Fpo])
    00d4d8a8 7a195ccc a8e57c98 0385d008 8000000a mscorwks!CCriticalSection::Lock+0x17 (FPO: [0,0,4])
    00d4df10 7a196399 00000001 00000001 80070002 mscorwks!CDebugLog::DumpDebugLog+0x5b (FPO: [Non-Fpo])
    00d4df24 7a17d78f 001dac60 00000001 00000001 mscorwks!CDebugLog::Flush+0x53 (FPO: [Non-Fpo])
    00d4df94 79ef3441 03858a40 79eef384 00000000 mscorwks!CAssemblyName::BindToObject+0x674 (FPO: [Non-Fpo])
    00d4e00c 79ef3261 00173918 00d4dfe0 03858a30 mscorwks!FusionBind::RemoteLoad+0x14d (FPO: [Non-Fpo])
    00d4e094 79ef3078 00173918 00179dd0 00d4e464 mscorwks!FusionBind::LoadAssembly+0x127 (FPO: [Non-Fpo])
    00d4e340 79ef0503 00158500 00000001 00d4e464 mscorwks!AssemblySpec::FindAssemblyFile+0xdc (FPO: [Non-Fpo])
    00d4e4ac 79ee1a9a 00d4e4d4 00000001 00000000 mscorwks!AppDomain::BindAssemblySpec+0x18d (FPO: [Non-Fpo])

00d4e544 79ee1958 23000010 00000000 a8e5466c mscorwks!PEFile::LoadAssembly+0x95 (FPO: [Non-Fpo])
00d4e5e4 79f07846 00158500 23000010 a8e545a8 mscorwks!Module::LoadAssembly+0xee (FPO: [Non-Fpo])
00d4e620 79ef70d9 006f4c1c 010000a3 00000000 mscorwks!Assembly::FindModuleByTypeRef+0x113 (FPO: [Non-Fpo])
00d4e670 79e82979 00d4e704 00000000 00000000 mscorwks!ClassLoader::LoadTypeDefOrRefThrowing+0xfc (FPO: [Non-Fpo])
00d4e6a0 79e9ecb3 00d4e704 006f4c1c 010000a3 mscorwks!ClassLoader::LoadTypeDefOrRefOrSpecThrowing+0xa0 (FPO: [Non-Fpo])
00d4e734 79e9bab0 006f4c1c 0a000148 00d4e9c4 mscorwks!MemberLoader::GetDescFromMemberDefOrRefThrowing+0x210 (FPO: [Non-Fpo])
00d4e9bc 79efa4a9 00000001 0a000148 00d4eb08 mscorwks!MemberLoader::GetMethodDescFromMemberDefOrRefOrSpecThrowing+0x219 (FPO: [Non-Fpo])
00d4ea48 79efaf4b 006f4c1c 0a000148 00d4eb40 mscorwks!MemberLoader::GetMethodDescFromMemberDefOrRefOrSpecNT+0x76 (FPO: [Non-Fpo])
00d4eb50 79efae1f 006f4c1c 0a000148 006f5718 mscorwks!CEEInfo::findMethodInternal+0x154 (FPO: [Non-Fpo])
00d4ebd0 79061fd6 00d4f5bc 006f4c1c 8a000101 mscorwks!CEEInfo::findMethod+0x107 (FPO: [Non-Fpo])
00d4ebe8 7906535a 8a000148 006f4c1c 006f5718 mscorjit!Compiler::eeFindMethod+0x22 (FPO: [Non-Fpo])
00d4ecb8 79063324 00000028 0a000148 baadbeef mscorjit!Compiler::impImportCall+0xa7 (FPO: [Non-Fpo])
00d4f290 790635c3 00b52c70 807e6cc2 00b53250 mscorjit!Compiler::impImportBlockCode+0x29ae (FPO: [Non-Fpo])
00d4f314 7906355b 00b52c70 00000000 00b50010 mscorjit!Compiler::impImportBlock+0x20c (FPO: [Non-Fpo])
00d4f32c 79063494 00b52bb8 00b50010 79064e1c mscorjit!Compiler::impImport+0xe5 (FPO: [Non-Fpo])
00d4f338 79064e1c 00b50010 00d4f39c 790614e6 mscorjit!Compiler::fgImport+0x20 (FPO: [0,0,4])
00d4f344 790614e6 00d4f464 00d4f6c8 00d4f458 mscorjit!Compiler::compCompile+0xb (FPO: [Non-Fpo])
00d4f39c 79061236 006f4c1c 00d4f5b0 00d4f63c mscorjit!Compiler::compCompile+0x2df (FPO: [Non-Fpo])
00d4f430 7906118c 00d4f5b0 00d4f63c 00d4f464 mscorjit!jitNativeCode+0xb8 (FPO: [Non-Fpo])
00d4f468 79efd49f 790b22a0 00d4f5b0 00d4f63c mscorjit!CILJit::compileMethod+0x3d (FPO: [Non-Fpo])
00d4f4d4 79efd415 00191950 00d4f5b0 00d4f63c mscorwks!invokeCompileMethodHelper+0x72 (FPO: [Non-Fpo])
00d4f518 79efd3a7 00191950 00d4f5b0 00d4f63c mscorwks!invokeCompileMethod+0x31 (FPO: [Non-Fpo])
00d4f570 79ef9c64 00191950 00d4f5b0 00000000 mscorwks!CallCompileMethodWithSEHWrapper+0x84 (FPO: [Non-Fpo])
00d4f928 79ef9a68 006f5718 001d9938 00107210 mscorwks!UnsafeJitFunction+0x230 (FPO: [Non-Fpo])
00d4f9cc 79ef9905 001d9938 00000000 a8e559ac mscorwks!MethodDesc::MakeJitWorker+0x1c1 (FPO: [Non-Fpo])
00d4fa24 79e79091 00000000 a8e559fc 00d4fc78 mscorwks!MethodDesc::DoPrestub+0x486 (FPO: [Non-Fpo])
00d4fa74 003c1efe 00d4faa4 a82bfd52 010646c4 mscorwks!PreStubWorker+0xeb (FPO: [Non-Fpo])
WARNING: Frame IP not in any known module. Following frames may be wrong.
00d4fa8c 793e25cf 00000000 011509f0 01063ff4 0x3c1efe
00d4faac 793e25cf 01150a08 79366b3d 00d4fae0 mscorlib_ni+0x3225cf
00d4fab4 79366b3d 00d4fae0 006f2010 011509f0 mscorlib_ni+0x3225cf
00d4fac8 793e2734 011509f0 00000000 0206101c mscorlib_ni+0x2a6b3d
00d4fae0 793e26ac 00d4fb04 00d4fb20 0019f9e0 mscorlib_ni+0x322734
00d4faf4 79e7c0e3 a8e558a4 7915faa4 00d4fb84 mscorlib_ni+0x3226ac
00000000 00000000 00000000 00000000 00000000 mscorwks!CallDescrWorker+0x33

   4  Id: ebc.b54 Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr  Args to Child              
00fafe14 7c90daaa 77e765e3 000002bc 00faff74 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
00fafe18 77e765e3 000002bc 00faff74 00000000 ntdll!NtReplyWaitReceivePortEx+0xc (FPO: [5,0,0])
00faff80 77e76caf 00faffa8 77e76ad1 001994d8 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x12a (FPO: [Non-Fpo])
00faff88 77e76ad1 001994d8 00000000 00d4ebe8 rpcrt4!RecvLotsaCallsWrapper+0xd (FPO: [Non-Fpo])
00faffa8 77e76c97 0017a9b0 00faffec 7c80b729 rpcrt4!BaseCachedThreadRoutine+0x79 (FPO: [Non-Fpo])
00faffb4 7c80b729 00199218 00000000 00d4ebe8 rpcrt4!ThreadStartRoutine+0x1a (FPO: [Non-Fpo])
00faffec 00000000 77e76c7d 00199218 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

   5  Id: ebc.194 Suspend: 1 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr  Args to Child              
0382fce8 7c90df4a 7c929cb6 00000015 0382fd30 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0382fcec 7c929cb6 00000015 0382fd30 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
0382ffb4 7c80b729 00000000 7c80ae40 0000ffff ntdll!RtlpWaitThread+0x13d (FPO: [Non-Fpo])
0382ffec 00000000 7c929b8f 00000000 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

   6  Id: ebc.e9c Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr  Args to Child              
03a2fec8 7c90df4a 7c809590 00000003 03a2fef4 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
03a2fecc 7c809590 00000003 03a2fef4 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
03a2ff68 7c80a115 00000003 76a61348 00000000 kernel32!WaitForMultipleObjectsEx+0x12c (FPO: [Non-Fpo])
03a2ff84 769c87bd 00000003 76a61348 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
03a2ffb4 7c80b729 00000000 00000000 00000000 userenv!NotificationThread+0x5f (FPO: [Non-Fpo])
03a2ffec 00000000 769c8761 00000000 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

   7  Id: ebc.fec Suspend: 1 Teb: 7ff4f000 Unfrozen
ChildEBP RetAddr  Args to Child              
03c9ccd8 7c90df5a 7c91b24b 0000060c 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
03c9ccdc 7c91b24b 0000060c 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
03c9cd64 7c901046 0023ba38 771c9bb8 7723ba38 ntdll!RtlpWaitForCriticalSection+0x132 (FPO: [Non-Fpo])
03c9cd6c 771c9bb8 7723ba38 00000003 038609b0 ntdll!RtlEnterCriticalSection+0x46 (FPO: [1,0,0])
03c9cd8c 7722065c 03869218 00000000 00237788 wininet!CConMgr::CreateUniqueFile+0x21 (FPO: [Non-Fpo])
03c9cdb0 7722127e 03869218 00000000 00237788 wininet!CreateUrlCacheEntryA+0x5d (FPO: [Non-Fpo])
03c9ce04 7a195a5a 03872320 00000000 7a195fe4 wininet!CreateUrlCacheEntryW+0x183 (FPO: [Non-Fpo])
03c9ce64 7a195dcb 03872320 00000000 7a195fe4 mscorwks!CreateUrlCacheEntryW_NoThrow+0x30 (FPO: [Non-Fpo])
03c9d4e0 7a196399 00000001 00000001 80070002 mscorwks!CDebugLog::DumpDebugLog+0x130 (FPO: [Non-Fpo])
03c9d4f4 7a17d78f 03864b60 00000001 00000001 mscorwks!CDebugLog::Flush+0x53 (FPO: [Non-Fpo])
03c9d564 79ef3441 038659f8 79eef384 00000000 mscorwks!CAssemblyName::BindToObject+0x674 (FPO: [Non-Fpo])
03c9d5dc 79f785cb 038654c0 03c9d5b0 038659e8 mscorwks!FusionBind::RemoteLoad+0x14d (FPO: [Non-Fpo])
03c9d664 79ef3078 038654c0 038659b0 03c9da34 mscorwks!FusionBind::LoadAssembly+0x223 (FPO: [Non-Fpo])
03c9d910 79ef0503 03862758 00000001 03c9da34 mscorwks!AssemblySpec::FindAssemblyFile+0xdc (FPO: [Non-Fpo])
03c9da7c 79edf037 03c9db70 00000001 00000000 mscorwks!AppDomain::BindAssemblySpec+0x18d (FPO: [Non-Fpo])
03c9dae4 79f1d707 0000000b 00000000 03c9dc3c mscorwks!AssemblySpec::LoadDomainAssembly+0x114 (FPO: [Non-Fpo])
03c9db08 79f22e94 0000000b 00000000 03c9dc3c mscorwks!AssemblySpec::LoadAssembly+0x1d (FPO: [Non-Fpo])
03c9dc54 79400e07 00000000 00000001 03c9dcdc mscorwks!AssemblyNative::Load+0x240 (FPO: [Non-Fpo])
03c9dca8 7939b788 00000000 03c9dcdc 03c9dcec mscorlib_ni+0x340e07
03c9dcdc 79398fdb 03837a00 00000000 03c9dd2c mscorlib_ni+0x2db788
03c9dcec 79399653 00000000 00000000 00000000 mscorlib_ni+0x2d8fdb
03c9dd2c 79e7c068 03c9e154 00000001 03c9e224 mscorlib_ni+0x2d9653
03c9ddac 79f290ba 03c9e154 00000001 03c9e224 mscorwks!CallDescrWorkerWithHandler+0xa3 (FPO: [Non-Fpo])
03c9ddcc 79f29061 03c9e150 00000001 03c9e224 mscorwks!DispatchCallBody+0x1e (FPO: [Non-Fpo])
03c9de30 79f50be1 03c9e150 00000001 03c9e224 mscorwks!DispatchCallDebuggerWrapper+0x3d (FPO: [Non-Fpo])
03c9ded8 79f558f9 03c9e150 00000001 03c9e224 mscorwks!DispatchCall+0xb3 (FPO: [Non-Fpo])
03c9e094 79f54e3c 03c9e268 03c9e13c 79f54e03 mscorwks!CrossDomainChannel::MarshalAndCall_Wrapper+0x4ad (FPO: [Non-Fpo])
03c9e0a0 79f54e03 03c9e268 03c9e130 79f495f5 mscorwks!MarshalAndCall_Wrapper2+0xf (FPO: [Non-Fpo])
03c9e13c 79f55507 00000002 79fa10af 03c9e268 mscorwks!MakeCallWithAppDomainTransition+0xbc (FPO: [Non-Fpo])
03c9f61c 79f51053 03c9f858 abf855e4 03c9f6f0 mscorwks!CrossDomainChannel::MarshalAndCall+0x7a6 (FPO: [Non-Fpo])
03c9f66c 79f50fd8 03c9f858 abf8553c 03c9f880 mscorwks!CrossDomainChannel::ExecuteCrossDomainCall+0x7d (FPO: [Non-Fpo])
03c9f6b4 79f50ea0 03c9f880 03c9f858 abf85bc8 mscorwks!CrossDomainChannel::CheckCrossDomainCall+0xdd (FPO: [Non-Fpo])
03c9f840 003c253d 03c9f880 03837a00 03c9f858 mscorwks!CTPMethodTable::OnCall+0xa6 (FPO: [Non-Fpo])
WARNING: Frame IP not in any known module. Following frames may be wrong.
03c9f8e4 79e790d2 0115229c 01064430 00000000 0x3c253d
03c9f98c 793e25cf 00000000 01150994 01063ff4 mscorwks!PreStubWorker+0x139 (FPO: [Non-Fpo])
03c9f9ac 793e25cf 011509ac 79366b3d 03c9f9e0 mscorlib_ni+0x3225cf
03c9f9b4 79366b3d 03c9f9e0 006f2010 01150994 mscorlib_ni+0x3225cf
03c9f9c8 793e2734 01150994 00000000 0206101c mscorlib_ni+0x2a6b3d
03c9f9e0 793e26ac 03c9fa04 03c9fa20 03837a00 mscorlib_ni+0x322734
03c9f9f4 79e7c0e3 746e696f 00007265 03c9fa84 mscorlib_ni+0x3226ac
00000000 00000000 00000000 00000000 00000000 mscorwks!CallDescrWorker+0x33

   8  Id: ebc.c84 Suspend: 1 Teb: 7ff4e000 Unfrozen
ChildEBP RetAddr  Args to Child              
03d9fe08 7c90d21a 7c8023f1 00000000 03d9fe3c ntdll!KiFastSystemCallRet (FPO: [0,0,0])
03d9fe0c 7c8023f1 00000000 03d9fe3c 00000000 ntdll!NtDelayExecution+0xc (FPO: [2,0,0])
03d9fe64 79f4c26f 000001f4 00000000 abe85d10 kernel32!SleepEx+0x61 (FPO: [Non-Fpo])
03d9fe98 79f270ec 000001f4 00000000 abe85d5c mscorwks!EESleepEx+0xa3 (FPO: [Non-Fpo])
03d9fed4 79f27090 000001f4 00000000 03d9ffb4 mscorwks!__DangerousSwitchToThread+0x75 (FPO: [Non-Fpo])
03d9fee4 79f4c122 000001f4 12b60d16 0000000a mscorwks!__SwitchToThread+0xd (FPO: [Non-Fpo])
03d9ffb4 7c80b729 00000000 7c910222 7c91019b mscorwks!ThreadpoolMgr::GateThreadStart+0xa1 (FPO: [Non-Fpo])
03d9ffec 00000000 79f4c08d 00000000 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

   9  Id: ebc.e34 Suspend: 1 Teb: 7ff4c000 Unfrozen
ChildEBP RetAddr  Args to Child              
0411febc 7c90df4a 7c809590 00000002 0411fee8 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0411fec0 7c809590 00000002 0411fee8 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
0411ff5c 7c80a115 00000002 0411ff9c 00000000 kernel32!WaitForMultipleObjectsEx+0x12c (FPO: [Non-Fpo])
0411ff78 60f83739 00000002 0411ff9c 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
0411ffac 60f83763 0411ffec 7c80b729 60f8c2b8 mqutil!CCancelRpc::ProcessEvents+0x7b (FPO: [Non-Fpo])
0411ffb4 7c80b729 60f8c2b8 7c912d78 00000000 mqutil!CCancelRpc::CancelThread+0xd (FPO: [Non-Fpo])
0411ffec 00000000 60f83756 60f8c2b8 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

  10  Id: ebc.3d8 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr  Args to Child              
0426fb38 7c90df5a 7c8025db 000000cc 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0426fb3c 7c8025db 000000cc 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])
0426fba0 79e718ff 000000cc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xa8 (FPO: [Non-Fpo])
0426fbe4 79e718c8 000000cc ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [Non-Fpo])
0426fc34 79e7187d ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [Non-Fpo])
0426fc48 79f6694c ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [Non-Fpo])
0426fd14 79ebaf8e 03862390 7c90d06a 7c90e45f mscorwks!AppDomain::ADUnloadThreadStart+0x308 (FPO: [Non-Fpo])
0426ffb4 7c80b729 03848570 00000000 03837a00 mscorwks!Thread::intermediateThreadProc+0x49 (FPO: [Non-Fpo])
0426ffec 00000000 79ebaf48 03848570 00000000 kernel32!BaseThreadStart+0x37 (FPO: [Non-Fpo])

# 11  Id: ebc.a10 Suspend: 1 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr  Args to Child              
00b9ffc8 7c951e40 00000005 00000004 00000001 ntdll!DbgBreakPoint (FPO: [0,0,0])
00b9fff4 00000000 00000000 00905a4d 00000003 ntdll!DbgUiRemoteBreakin+0x2d (FPO: [Non-Fpo])

Sorry for the long paste...

Now, the code has a long list of dependencies, some managed (.net 1.0, 1.1, 2.0) and some unmanaged. None of the 'syncblk', 'locks' or 'dlk' (from sosex) show anything, but the deadlock is there and it is reported by DebugDiag analyzes on the memory dump (and the service is locked not doing any processing).

Considering my poor knowledge of WinDbg, how can I find out the actual cause of this deadlock? Or make sure I can avoid it?

The issue only seem to appear when .net 4.0 is also installed. I know that the framework is loading the latest version of mscoreei.dll, but this seems to only be a catalyst not the actual cause. And the issue in all scenarios seem to be due to a deadlock between a dll load and GC (and it might be triggered by the load of another win service). All managed threads are marked as PreEmptive GC "Enabled".

The system is WinXP+ SP3 + .net 2.0 + SP1/SP2 and + .net 4.0.

Thanks, florin

Answer 1

The command you want when diagnosing deadlocks is the !locks command - you should also make sure that you have the Windows symbols available (see [here][1] for instructions on how to do this).

Usually I do a !locks to get a list of critical sections:

CritSec DeadLockDemo!CritSecOne+0 at 004A0620
LockCount          1
RecursionCount     1
OwningThread       3d8
EntryCount         1
ContentionCount    1
*** Locked

...

You then do a ~*kv 90 to show a list of all threads in the process - the one threads you are looking for are the ones that have 004A0620 in the arguments list - this identifies the critical section that the thread is looking for.

Using this information (the list of threads waiting for a given critical section and the thread that owns that critical section), you can build up a tree of threads that are waiting for each other.

Update:

I've had a look but I'm not sure whats going on - thread b08 appears to be waiting on thread fec for some sort of lock used by mscorwks!CDebugLog::DumpDebugLog (which seems reasonable enough), however thread fec is stuck waiting on b08 during a call to wininet!CConMgr::CreateUniqueFile.

I'm struggling to see what thread b08 has done to obtain a lock that prevents thread fec continuing - it seems to me like thread b08 is the culprit, but it doesn't seem to be actually running anything other than .Net code.

The only thing that I can think of is that this is a Mixed mode assembly issue - you should check again to make sure that none of the assemblies you are using are mixed mode.

Answer 2

You should also take a look at sosex and its very useful !dlk command which checks out deadlocks ( the !locks command tracks native deadlocks, not managed ones).

Looking at the stack trace, and from what you've said, it can be a case of mixed (native/managed) assembly loading deadlock. Maybe http://msdn.microsoft.com/en-us/library/ms173266.aspx could help here.

Answer 3

Ended up opening a support ticket to MS. It was a combination of: 1st time assembly loading (ngen), mixed assemblies, fusion log being enabled. The workaround was to put merge code from 2 windows services.

Hope in the near future to be able to rewrite some code and get rid of some dependencies (unmanaged c++, .net 1.0 & 1.1)