UI Automation events stop being received after a while monitoring an application and then restart after some time

Question

We are using Microsoft's UIAutomation framework to develop a client that monitors events of a specific application and responds to them in different ways. We've started with the managed version of the framework, but due to delay issues, moved to the native version wrapped in UIACOMWrapper. After more issues with performance inside our (massive) WPF application, we decided to move it to a separate terminal application (transfer the events to our WPF app through UDP) which seemed to fix all the performance issues. The only problem is that it seems that every several minutes, the events for TabSelection, StructureChanged, WindowOpened and WindowClosed stop being captured for a few minutes. Surprisingly PropertyChanged events are still received and handled while this happens. I will post the relevant code of our event monitor, but this is probably irrelevant as we have seen similar behavior when using Microsoft's own AccEvent utility. I can't post the code of the monitored application as it is proprietary and confidential as well, I can say that it is a WinForms application that hosts WPF windows and also quite massive. Has anyone seen this sort of behavior while working with the UI Automation framework? Thank you for your time.

Here's the monitor code (I know the event handling is on the UI Automation threads here but moving it to a dedicated thread did not change anything):

        public void registerHandlers()
    {
        //Register on structure changed and window opened events 
        System.Windows.Automation.Automation.AddStructureChangedEventHandler(
            this.getMsAutomationElement(), System.Windows.Automation.TreeScope.Subtree, this.handleStructureChanged);
        System.Windows.Automation.Automation.AddAutomationEventHandler(
            System.Windows.Automation.WindowPattern.WindowOpenedEvent,
            this.getMsAutomationElement(),
            System.Windows.Automation.TreeScope.Subtree,
            this.handleWindowOpened);
        System.Windows.Automation.Automation.AddAutomationEventHandler(
            System.Windows.Automation.WindowPattern.WindowClosedEvent,
            System.Windows.Automation.AutomationElement.RootElement,
            System.Windows.Automation.TreeScope.Subtree,
            this.handleWindowClosed);

        this.registerValueChanged();
        this.registerTextNameChange();
        this.registerTabSelected();
        this.registerRangeValueChanged();
    }

    private void registerRangeValueChanged()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
                    this.getMsAutomationElement(),
                    System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                    System.Windows.Automation.RangeValuePattern.ValueProperty);
        }
    }

    private void unregisterRangeValueChanged()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
                this.getMsAutomationElement(),
                this.handlePropertyChange);
    }

    private void registerValueChanged()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
                this.getMsAutomationElement(),
                System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                System.Windows.Automation.ValuePattern.ValueProperty);
        }
    }

    private void unregisterValueChanged()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
                            this.getMsAutomationElement(),
                            this.handlePropertyChange);
    }

    private void registerTextNameChange()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
            this.getMsAutomationElement(),
            System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                System.Windows.Automation.AutomationElement.NameProperty);
        }
    }

    private void unregisterTextNameChange()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
        this.getMsAutomationElement(),
        this.handlePropertyChange);
    }
    private void handleWindowOpened(object src, System.Windows.Automation.AutomationEventArgs e)
    {
        Console.ForegroundColor = ConsoleColor.Magenta;
        Console.WriteLine(DateTime.Now.ToShortTimeString() + " " + "Window opened:" + " " + 
            (src as System.Windows.Automation.AutomationElement).Current.Name);

        System.Windows.Automation.AutomationElement element = src as System.Windows.Automation.AutomationElement;
        //this.sendEventToPluginQueue(src, e, element.GetRuntimeId(), this.getAutomationParent(element).GetRuntimeId());
        //Fill out the fields of the control added message
        int[] parentId = this.getAutomationParent(element).GetRuntimeId();
        this.copyToIcdArray(parentId,
            this.protocol.getMessageSet().outgoing.ControlAddedMessage.Data.controlAdded.parentRuntimeId);
        this.copyToIcdArray(element.GetRuntimeId(),
            this.protocol.getMessageSet().outgoing.ControlAddedMessage.Data.controlAdded.runtimeId);
        //Send the message using the protocol
        this.protocol.send(this.protocol.getMessageSet().outgoing.ControlAddedMessage);
    }

    private void copyToIcdArray(int[] runtimeId, ICD.UI_AUTOMATION.RuntimeId icdRuntimeId)
    {
        icdRuntimeId.runtimeIdNumberOfItems.setVal((byte)runtimeId.Count());
        for (int i = 0; i < runtimeId.Count(); i++)
        {
            icdRuntimeId.runtimeIdArray.getElement(i).setVal(runtimeId[i]);
        }
    }

    private void handleWindowClosed(object src, System.Windows.Automation.AutomationEventArgs e)
    {
        if (src != null)
        {
            Console.ForegroundColor = ConsoleColor.Cyan;
            Console.WriteLine(DateTime.Now.ToShortTimeString() + " " + "Window closed:" + " " +
                (src as System.Windows.Automation.AutomationElement).GetRuntimeId().ToString());

            System.Windows.Automation.AutomationElement element = src as System.Windows.Automation.AutomationElement;
            this.copyToIcdArray(element.GetRuntimeId(),
                this.protocol.getMessageSet().outgoing.ControlRemovedMessage.Data.controlRemoved.runtimeId);
            //Send the message using the protocol
            this.protocol.send(this.protocol.getMessageSet().outgoing.ControlRemovedMessage);

            //this.sendEventToPluginQueue(src, e, element.GetRuntimeId());
        }
    }

EDIT: I forgot to mention that I strongly suspect that the issue is that one of the UI-Automation event handler threads gets stuck somehow. The reason I believe this, is that when the problem occurred in my monitor, I started an instance of AccEvent and it received all the missing events that my monitor was not getting. This means that the events are being fired but not passed to my monitor.

EDIT2: I forgot to mention that this happens running in Windows 8 with the specific target application, I have not seen this phenomenon on my own Windows 7 machine with other applications. Another interesting thing is that it seems to happen periodically more or less, but regardless of when I subscribe to events, i.e. it can happen almost immediately after subscribing but then it takes several minutes to reoccur.

Answer 1

I'm afraid I don't know the cause of the delays that you're seeing, but here are some thoughts on this...

Everything I say below relates to the native UIA API in Windows, not the managed .NET UIA API. All improvements to UIA in recent years have been made to the Windows UIA API. So whenever I write UIA client C# code, I call UIA through a managed wrapper that I generate with the tlbimp.exe SDK tool.

That is, I first generate the wrapper with a command like...

"C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\x64\tlbimp.exe" c:\windows\system32\uiautomationcore.dll /out:Interop.UIAutomationCore.dll

Then I include a reference to the Interop.UIAutomationCore.dll in my C# project, add "using Interop.UIAutomationCore;" to my C# file, and then I can do things like...

IUIAutomation uiAutomation = new CUIAutomation8();

IUIAutomationElement rootElement = uiAutomation.GetRootElement();

uiAutomation.AddAutomationEventHandler(
    20016, // UIA_Window_WindowOpenedEventId
    rootElement,
    TreeScope.TreeScope_Descendants,
    null,
    this);

...

public void HandleAutomationEvent(IUIAutomationElement sender, int eventId)
{
    // Got a window opened event...
}

In Windows 7, there were some important constraints around UIA event handlers. It was easy to write event handlers which didn't account for those constraints, and that could lead to long delays when interacting with UIA. For example, it was important to not add or remove a UIA event handler from inside an event handler. So at the time, I intentionally made no UIA calls at all from inside my event handlers. Instead, I'd post myself a message or add some action to a queue, allow my event handler to return, and take whatever action I wanted to in response to the event shortly afterwards on another thread. This required some more work on my part, but I didn't want to risk hitting delays. And any threads I created would be running in an MTA.

An example of the action described above is in my old focus tracking sample up at https://code.msdn.microsoft.com/windowsapps/Windows-7-UI-Automation-6390614a/sourcecode?fileId=21469&pathId=715901329. The file FocusEventHandler.cs creates the MTA thread and queues messages to avoid making UIA calls inside the event hander.

Since Window 7, I know the constraints in UIA relating to threading and delays have been relaxed, and the likelihood of encountering delays has been reduced. More recently, there were some improvements between Windows 8.1 and Windows 10 in this area, so if it'd be practical to run your code on Windows 10, it would be interesting to see if the delays still repro there.

I know this is time consuming, but you might be interested in removing the interaction with UIA inside your event handlers and seeing if the delays go away. If they do, it'd be a case of determining which action seems to trigger the problem, and seeing if there's an alternative way of achieving your goals without performing the UIA interaction in the event handlers.

For example, in your event handler, you call...

this.getAutomationParent(element).GetRuntimeId();

I expect this will lead to two calls back into the provider app which generated the event. The first call is to get the parent of the source element, and the second call is to get the RuntimeId of that parent. So while UIA is waiting for your event handler to return, you've called twice back into UIA. While I don't know that that's a problem, I'd avoid it.

Sometimes you can avoid a cross-proc call back to the provider process by having some data of interest cached with the event itself. For example, say I know I'm going to want the RuntimeId of an element that raised a WindowOpened event. I can ask UIA to cache that data with the events I receive, when I register for the events.

int propertyRuntimeId = 30000; // UIA_RuntimeIdPropertyId

...

IUIAutomationCacheRequest cacheRequestRuntimeId = uiAutomation.CreateCacheRequest();
cacheRequestRuntimeId.AddProperty(propertyRuntimeId);

uiAutomation.AddAutomationEventHandler(
    20016, // UIA_Window_WindowOpenedEventId
    rootElement,
    TreeScope.TreeScope_Descendants,
    cacheRequestRuntimeId,
    this);

...

public void HandleAutomationEvent(IUIAutomationElement sender, int eventId)
{
    // Got a window opened event...

    // Get the RuntimeId from the source element. Because that data is cached with the
    // event, we don't have to call back through UIA into the provider process here.
    int[] runtimeId = sender.GetCachedPropertyValue(propertyRuntimeId);
}

On a side note, when practical, I always cache data when dealing with events or accessing elements through UIA, (by using calls such as FindFirstBuildCache(),) as I want to avoid as many cross-proc calls as possible.

So my advice would be:

1. Use the native Windows UIA API with a managed wrapper generated by tlbimp.exe.
2. Cache as much data as possible with the events, to avoid having to call back into the provider process unnecessarily later.
3. Avoid calls back into UIA from inside a UIA event handler.

Thanks,

Guy

Answer 2

I have seen this behavior in my project. The solution was unsubscribes and resubscribe to the events using a timer. In addition, I set off any action following the events in a new task (running in an STA thread pool).