How to avoid race condition between Process termination notification and standard output redirection events?

Question

In a very controlled fashion, I queue jobs in a ProcessJobManager that will process them in up to X concurrent processes at a time.

After starting each process, I add it to a List<ActiveProcessJob> and store a WaitHandle for the process in my ActiveProcessJob's CompleteEvent property by constructing a new ManualResetEvent( false ) and assigning its SafeWaitHandle property to new SafeWaitHandle( my_process.Handle, false ).

I wait for processes to finish (as well as for any 'new_jobs_queued') via code like:

WaitHandle[] wait_handles =
    active_jobs
    .Select<ActiveProcessJob,WaitHandle>( j => j.CompleteEvent )
    .Union( new WaitHandle[]{ new_jobs_queued} ).ToArray();
WaitHandle.WaitAny( wait_handles );

This properly detects the termination of one or more processes (or items added to the queue); however, I am also redirecting the standard output stream and calling Process.BeginOutputReadLine to ensure Process.OutputDataReceived events are fired.

The problem is that the process termination is often detected and handled (it is removed from the active_jobs list before the event handler for the Process.OutputDataReceived event fires for the last time. In that case, the handler can't reference the Process because it has already been removed from the queue.

I almost need to know when a process is "about to exit", because otherwise I have no idea when to expect the last OutputDataReceived event, which obviously runs on a separate thread from my WaitAny call that waits for processes to terminate.

Perhaps Process.Exit is guaranteed to be called after the last Process.OutputDataReceived event and before the Process.HasExited method returns true? I need some certainty like that.

Answer 1

Since the redirected output stream is running in a separate thread from the one that handles the process termination signal and is in no other way synchronized with it, it's possible that the thread which handles the termination signal will perform all its work before the thread that handles the standard output event runs or finishes.

Process.Exit is not guaranteed to run before all output events have finished, probably due to an oversight where it doesn't wait for the end of the stream before calling RaiseOnExited, except when you call Process.WaitForExit, which I've determined from decompiling the code with .NET Reflector.

UPDATE: I actually found this confirmed in the documentation, which seems to suggest you must call WaitForExit TWICE to be sure it finished:

"When standard output has been redirected to asynchronous event handlers, it is possible that output processing will not have completed when this method returns. To ensure that asynchronous event handling has been completed, call the WaitForExit() overload that takes no parameter after receiving a true from this overload."

Original I thought a workaround might be to call Process.CancelOutputRead to remove the associated event listener before proceeding to remove the reference to the process from the job queue, but that may not work since Process.CancelOutputRead doesn't actually do any synchronous cleanup; it just sets a flag. Aside from just accepting that the output event handler may run after the process termination handler has completed and deal with the error, it seems that the only real workaround is to call WaitForExit() after the process termination signal has been received.