.Net 2.0 Windows Service hangs during garbage collection

Question

I have written a .Net (2.0) Windows Service used to send out pager messages to users (via serial port connections to 3rd party hardware).

The service periodically queries a database (OSI PI Historian), parses the values and decides whether to send a message based on the paresed values.

The service uses a lookup object (the values of which come from a SQL Server database and is refreshed every minute) to get user address values and the message string to send.

The service uses a number of Threading.Timers to trigger the periodic database calls and message sends.

The service is installed on a Windows 2003 machine which has .Net 2 SP2 installed.

The service works fine for about a week and then it hangs. No exceptions are recorded within the log (log4net).

I have take a number of dumps from the server and they all exhibit the same characteristics - a thread triggers GC but hangs indefinitely:

06aaf06c 7c827b99 77e61d1e 00004acc 00000000 ntdll!KiFastSystemCallRet
06aaf070 77e61d1e 00004acc 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
06aaf0e0 79e8c5f9 00004acc ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
06aaf124 79e8c52f 00004acc ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af
06aaf174 79e8c54e ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117
06aaf188 79f2f88f ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17
06aaf1a8 79f2f8ca 7a3b9020 00000000 ffffffff mscorwks!SVR::gc_heap::user_thread_wait+0x50
06aaf1b8 79f2f806 00000000 7a3b8b18 00080101 mscorwks!WKS::gc_heap::wait_to_proceed+0xe
06aaf1cc 79f92f5a 00000000 00000000 00000000 mscorwks!WKS::gc_heap::garbage_collect+0x247
06aaf1f8 79f94e26 00000000 00000000 00000030 mscorwks!WKS::GCHeap::GarbageCollectGeneration+0x1a9
06aaf284 79f926ce 052c75c8 00000030 00000000 mscorwks!WKS::gc_heap::try_allocate_more_space+0x15b
06aaf298 79f92769 052c75c8 00000030 00000000 mscorwks!WKS::gc_heap::allocate_more_space+0x11
06aaf2b8 79e73291 052c75c8 0000002e 00000000 mscorwks!WKS::GCHeap::Alloc+0x3b
06aaf2d4 79e7d8d4 0000002e 00000000 00000000 mscorwks!Alloc+0x60
06aaf310 79e99056 0000000f b456b75e 00001ce3 mscorwks!SlowAllocateString+0x29
06aaf3b4 792bb2c1 00000000 0108082b 00001f40 mscorwks!FramedAllocateString+0xa1
    Stack shortened for bravity...

Managed stack:

ESP       EIP     
06aaf364 7c82847c [HelperMethodFrame: 06aaf364] 
06aaf3bc 792bb2c1 System.String.CreateStringFromEncoding(Byte*, Int32, System.Text.Encoding)
06aaf3dc 792aaf2a System.Text.EncodingNLS.GetString(Byte[], Int32, Int32)
06aaf3fc 6522d131 System.Data.SqlClient.TdsParserStateObject.ReadStringWithEncoding(Int32, System.Text.Encoding, Boolean)
06aaf41c 656ca93e System.Data.SqlClient.TdsParser.ReadSqlStringValue(System.Data.SqlClient.SqlBuffer, Byte, Int32, System.Text.Encoding, Boolean, System.Data.SqlClient.TdsParserStateObject)
06aaf448 6522d925 System.Data.SqlClient.TdsParser.ReadSqlValue(System.Data.SqlClient.SqlBuffer, System.Data.SqlClient.SqlMetaDataPriv, Int32, System.Data.SqlClient.TdsParserStateObject)
06aaf474 656208b7 System.Data.SqlClient.SqlDataReader.ReadColumnData()
06aaf484 65620962 System.Data.SqlClient.SqlDataReader.ReadColumn(Int32, Boolean)
06aaf4b4 65221415 System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32)
06aaf4c8 652213af System.Data.SqlClient.SqlDataReader.GetValue(Int32)
06aaf4f8 65220aec System.Data.SqlClient.SqlDataReader.get_Item(System.String)
06aaf504 03616b81 AlarmEventCollator.DataAccess.GetAllAlarmDetails()
06aaf564 03616a28 AlarmEventCollator.AlarmBuilder.buildLookups()
06aaf590 0361aa89 AlarmEventCollator.AlarmBuilder.pollLookups(System.Object)
06aaf5d0 792a83ff System.Threading._TimerCallback.TimerCallback_Context(System.Object)
06aaf5d8 792e019f System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
06aaf5f0 792a836b System.Threading._TimerCallback.PerformTimerCallback(System.Object)
06aaf77c 79e71b4c [GCFrame: 06aaf77c] 

The GC usually happens in the database parsing methods. All the other managed threads appear to hang - which would be consistent with the GC process.

I would appreciate any pointers. More information available on request.

Answer 1

How many GC threads do we have in a .NET process running the Server version of the GC on a dual-core machine?

Two, one per processor, or rather one per logical processor, so it would have been 4 if it was hyper threaded. In a process running the workstation version of the GC we would have no dedicated GC threads, instead garbage collection runs on the thread initiating the GC as there is no point in switching to a different thread for garbage collection when you only have one proc/one thread doing the GC.

Chris Lyon has a good writeup about GC modes and also an interesting post about GC latency modes comming in Orcas.

Why is this important to you? Since different GC modes are optimized for different things, your memory usage, GC latency etc. may vary a lot depending on what GC mode you are using. For example a windows service by default gets the workstation GC, but if there is a lot of througput (lots of short lived allocs), you are probably a lot better off running the server GC for memory usage and perf.