When I try to use UI Automation for PowerPoint 2013, I can only get the first character/word when I use RangeFromPoint

Question

The code works for Word and Outlook but fails with PowerPoint in that only the first character or first word of the textbox ever gets selected. Is this a bug? Is there any workaround? Try this on a simple PowerPoint slide in PowerPoint 2013.

private static async Task<string> getText(double x, double y)
{
    string result = null;

    try
    {
        var location = new System.Windows.Point(x, y);
        AutomationElement element = AutomationElement.FromPoint(location);

        object patternObj;
        if (element.TryGetCurrentPattern(TextPattern.Pattern, out patternObj))
        {
            var textPattern = (TextPattern)patternObj;

            var range = textPattern.RangeFromPoint(location);
            range.ExpandToEnclosingUnit(TextUnit.Word);
            range.Select();

            var text = range.GetText(-1).TrimEnd('\r');
            return text.Trim();
        }
        else
        {
            return "no text found";
        }
    }
    catch (Exception ex)
    {
        return ex.Message;
    }
}

You cannot see it from the screenshot, but the mouse is on "first" not "stuck", but regardless of where the mouse is placed, it always is stuck. Maybe this is fixed in PowerPoint 2016?

When I look at the bounding box for the range it is always the whole element, rather than the selected word. That could be part of the problem of why RangeToPoint is not working.

Original posted in MSDN but no response...

Update. If I use

text = printRange(range, text);
while (range.Move(TextUnit.Word, 1) > 0)
{
    text += Environment.NewLine;
    text = printRange(range, text);
}

I get

Answer 1

This behavior is probably due to a limitation in PowerPoint 2013, and I expect you can't work around it using UIA. When you call RangeFromPoint(), the UIA provider hit beneath the mouse, (ie the one that's implementing IUIAutomationTextPattern::RangeFromPoint(),) is meant to return a degenerative (ie empty) range where the mouse cursor is. Then the UIA client can expand the returned range to get the surrounding character, word, line or paragraph.

However, as you point out, PowerPoint 2013 isn't doing that. I've just written the test code below, (using a managed wrapper for the native Windows UIA API generated by tlbimp.exe,) and found that PowerPoint apparently returns a TextRange for the entire text box beneath the cursor. When I ran the code, I found that I did get the expected word beneath the cursor in WordPad, Word 2013 and PowerPoint OnLine, but not PowerPoint 2013. I got the same results when I ran the Text Explorer tool that's part of the Inspect SDK tool. The image below shows Text Explorer reporting that the text returned from PowerPoint 2013 is the entire text in the a text box, when the mouse is hovering over one of those words.

(I should add that for the test code below to work at all, I think the current display scaling setting needs to be at 100%. I've not added code to account for some other scaling being active.)

I don't know if this is fixed in PowerPoint 2016, I'll try to look into that and let you know.

Thanks,

Guy

private void buttonGetTheText_Click(object sender, EventArgs e)
{
    labelText.Text = "No text found.";

    IUIAutomation uiAutomation = new CUIAutomation8();

    Point ptCursor = Cursor.Position;

    tagPOINT pt;
    pt.x = ptCursor.X;
    pt.y = ptCursor.Y;

    // Cache the Text pattern that's available through the element beneath
    // the mouse cursor, (if the Text pattern's supported by the element,) in
    // order to avoid another cross-process call to get the pattern later.
    int patternIdText = 10014; // UIA_TextPatternId
    IUIAutomationCacheRequest cacheRequestTextPattern =
        uiAutomation.CreateCacheRequest();
    cacheRequestTextPattern.AddPattern(patternIdText);

    // Now get the element beneath the mouse.
    IUIAutomationElement element = 
        uiAutomation.ElementFromPointBuildCache(pt, cacheRequestTextPattern);

    // Does the element support the Text pattern?
    IUIAutomationTextPattern textPattern =
        element.GetCachedPattern(patternIdText);
    if (textPattern != null)
    {
        // Now get the degenerative TextRange where the mouse is.
        IUIAutomationTextRange range = textPattern.RangeFromPoint(pt);
        if (range != null)
        {
            // Expand the range to include the word surrounding 
            // the point where the mouse is.
            range.ExpandToEnclosingUnit(TextUnit.TextUnit_Word);

            // Show the word in the test app.
            labelText.Text = "Text is: \"" + range.GetText(256) + "\"";
        }
    }
}