Windows Form application freeze randomly when run overnight

Question

I have a window form application, and it has multiple threads running that would invoke on the Main UI thread to update the UI. Occasionally on development machine, the application main UI thread will stop running, and the application no longer responses. It seems to happen if I left the application running overnight. However, I have users who run this window form application though remote desktop, and this problem happen a lot more often if the application is left running overnight with no user interaction.

I have find an article seems to be describing this problem, but I don't have enough Windows development knowledge to figure out why the application would freeze.

The only information I got is the following stack trace, indicating the main UI thread is waiting for some kind of operation.

This problem has been bothering me for quite some time now. I would appreciate any suggestions or comments.

Thanks!

Main UI thread stack trace:

mscorlib.dll!System.Threading.WaitHandle.WaitOne(long timeout, bool exitContext) + 0x2f bytes
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) + 0x25 bytes
System.Windows.Forms.dll!System.Windows.Forms.Control.WaitForWaitHandle(System.Threading.WaitHandle waitHandle = {System.Threading.ManualResetEvent}) Line 4268 C#
System.Windows.Forms.dll!System.Windows.Forms.Control.MarshaledInvoke(System.Windows.Forms.Control caller, System.Delegate method, object[] args, bool synchronous) Line 7614 C#
System.Windows.Forms.dll!System.Windows.Forms.Control.Invoke(System.Delegate method, object[] args) Line 7178 + 0x11 bytes C#
System.Windows.Forms.dll!System.Windows.Forms.WindowsFormsSynchronizationContext.Send(System.Threading.SendOrPostCallback d, object state) Line 89 C#
System.dll!Microsoft.Win32.SystemEvents.SystemEventInvokeInfo.Invoke(bool checkFinalization = true, object[] args = {object[2]}) + 0x62 bytes
System.dll!Microsoft.Win32.SystemEvents.RaiseEvent(bool checkFinalization = true, object key = {object}, object[] args = {object[2]}) + 0x10f bytes
System.dll!Microsoft.Win32.SystemEvents.OnUserPreferenceChanging(int msg, System.IntPtr wParam, System.IntPtr lParam) + 0x77 bytes
System.dll!Microsoft.Win32.SystemEvents.WindowProc(System.IntPtr hWnd = 2032836, int msg = 8218, System.IntPtr wParam = 47, System.IntPtr lParam = 100019840) + 0x2ca bytes
[Native to Managed Transition]
[Managed to Native Transition]
System.Windows.Forms.dll!System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(int dwComponentID, int reason = 4, int pvLoopData = 0) Line 2106 + 0x8 bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(int reason = 4, System.Windows.Forms.ApplicationContext context = {System.Windows.Forms.Application.ModalApplicationContext}) Line 3377 + 0x1b bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoop(int reason, System.Windows.Forms.ApplicationContext context) Line 3261 + 0xa bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.RunDialog(System.Windows.Forms.Form form) Line 1488 C#
System.Windows.Forms.dll!System.Windows.Forms.Form.ShowDialog(System.Windows.Forms.IWin32Window owner) Line 6120 + 0x8 bytes C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleAppMainForm.ShowOpenScheduleForm.AnonymousMethod() Line 829 + 0xd bytes C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleAppMainForm.PromptUserToSaveSchedule(System.Action oAfterPromptUserToSaveCallBack = {Method = Cannot evaluate expression because the code of the current method is optimized.}) Line 1858 + 0xb bytes C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleAppMainForm.ShowOpenScheduleForm() Line 859 + 0xb bytes C#
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Delegate.DynamicInvokeImpl(object[] args) + 0x55 bytes
System.Windows.Forms.dll!System.Windows.Forms.Control.InvokeMarshaledCallbackDo(System.Windows.Forms.Control.ThreadMethodEntry tme) Line 7266 + 0xb bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(object obj) Line 7228 + 0x7 bytes C#
mscorlib.dll!System.Threading.ExecutionContext.runTryCode(object userData) + 0x51 bytes
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x67 bytes
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x45 bytes
System.Windows.Forms.dll!System.Windows.Forms.Control.InvokeMarshaledCallback(System.Windows.Forms.Control.ThreadMethodEntry tme) Line 7213 + 0xffffffc5 bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Control.InvokeMarshaledCallbacks() Line 7297 + 0xb bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Control.WndProc(ref System.Windows.Forms.Message m) Line 13848 C#
System.Windows.Forms.dll!System.Windows.Forms.ScrollableControl.WndProc(ref System.Windows.Forms.Message m) Line 1491 C#
System.Windows.Forms.dll!System.Windows.Forms.ContainerControl.WndProc(ref System.Windows.Forms.Message m) Line 1898 C#
System.Windows.Forms.dll!System.Windows.Forms.Form.WndProc(ref System.Windows.Forms.Message m) Line 7515 C#
System.Windows.Forms.dll!System.Windows.Forms.Control.ControlNativeWindow.OnMessage(ref System.Windows.Forms.Message m) Line 14051 C#
System.Windows.Forms.dll!System.Windows.Forms.Control.ControlNativeWindow.WndProc(ref System.Windows.Forms.Message m) Line 14106 C#
System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.Callback(System.IntPtr hWnd, int msg = 49512, System.IntPtr wparam, System.IntPtr lparam) Line 647 + 0xa bytes C#
System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.DefWndProc(ref System.Windows.Forms.Message m = {System.Windows.Forms.Message}) Line 814 + 0x1d bytes C#
System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.WndProc(ref System.Windows.Forms.Message m) Line 1409 C#
Infragistics2.Win.UltraWinToolbars.v8.1.dll!Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.FormSubClasser.WndProcImpl(ref System.Windows.Forms.Message m) + 0x17f5 bytes
Infragistics2.Win.UltraWinToolbars.v8.1.dll!Infragistics.Win.UltraWinToolbars.UltraToolbarsManager.FormSubClasser.WndProc(ref System.Windows.Forms.Message m) + 0x5 bytes
System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.Callback(System.IntPtr hWnd, int msg = 49512, System.IntPtr wparam, System.IntPtr lparam) Line 647 + 0xa bytes C#
[Native to Managed Transition]
[Managed to Native Transition]
System.Windows.Forms.dll!System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(int dwComponentID, int reason = -1, int pvLoopData = 0) Line 2106 + 0x8 bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(int reason = -1, System.Windows.Forms.ApplicationContext context = {System.Windows.Forms.ApplicationContext}) Line 3377 + 0x1b bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.ThreadContext.RunMessageLoop(int reason, System.Windows.Forms.ApplicationContext context) Line 3261 + 0xa bytes C#
System.Windows.Forms.dll!System.Windows.Forms.Application.Run() Line 1457 C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleApp.LoadData() Line 318 + 0x5 bytes C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleApp.Run() Line 170 + 0x9 bytes C#
Schedule.exe!ME.APTS.ScheduleApp.ScheduleApp.Main() Line 126 + 0xb bytes C#

Answer 1

Yes, this is a fairly infamous threading problem caused by the SystemEvents class. I never got a solid diagnostic for it but the 90% odds are that this is triggered by an initialization problem in your app.

The root problem is that SystemEvents gets initialized on-demand by the first form in your app that has controls that are interested in the events it generates. If that first form is not created in the main thread then SystemEvents is helpless to guess at which thread is the UI thread in your program. Eventually, when a notification is received (like UserPreferenceChanging), it tries to fire the event on that thread, but it isn't around anymore. The fallback code in the SynchronizationContext class raises the event on a threadpool thread instead. That inevitably invokes Threading Hell by running UI code on a thread that didn't create the window. Lots of things can go wrong when that happens. Deadlock is a particularly common outcome when restoring the desktop after the workstation was locked.

Not the only possible way this can go wrong, it is inevitable if you create any form on another thread. Now SystemEvents cannot possibly raise the event on the correct thread of course, somebody is going to lose. A blog post that demonstrates a debugging technique is here. Yes, ugly. Ideally a control knows to deal with this and marshal the notification itself. But that was forgotten knowledge at .NET 2.0, DataGridView, NumericUpDown, DomainUpDown, ToolStrip+MenuStrip and the ToolStripItem derived classes don't do this. I should note that RichTextBox and ProgressBar are suspicious, the rest are okay.

Review the startup sequence of your app. Creating your own splash screen is a good lead, do favor using the built-in support that the WindowsFormsApplicationBase class provides. If you do it yourself then keep it very simple, just a bitmap. And as noted, any place where you might create your own form on a worker thread is a recipe for trouble. Always do it the other way around, run the expensive code on a worker and keep the UI on the main thread.