Android onAccessibilityEvent get X and Y position of TextView

Question

I am trying to get the X and Y position of a TextView on my screen using Accessibility service to find the TextView, is this possible? Everything I have found requires you first to touch the screen. Here is how I am getting my node information.

public class MyAccessibilityService extends AccessibilityService {

    @TargetApi(16)
    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
        AccessibilityNodeInfo source = event.getSource();
    }
}

Answer 1

You can arbitrarily search the accessibility view hierarchy from an accessibility service any time! Though, I do recommend doing so from within the context of some type of accessibility event, as you then ensure that there is screen content to be traversed! Doing so in random callbacks is finicky at best. Here is a reasonable accessibility config XML file to use for such purposes:

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:accessibilityEventTypes="typeWindowContentChanged|typeWindowsChanged|typeWindowStateChanged"
    android:accessibilityFlags="flagReportViewIds|flagIncludeNotImportantViews"
    android:canRetrieveWindowContent="true"
    android:accessibilityFeedbackType="feedbackGeneric"
    android:notificationTimeout="1000" 
    android:settingsActivity=".SettingsActivity"
    />

Below are some comments on a few of these fields specifically.

android:notificationTimeout="1000"

Only retrieve accessibility events of a given type once per second! With the events listed, any lower setting would be QUITE verbose. We are only relying on this to call our callback and ensure we have nodes. Once per second is JUST DANDY for these purposes. Tweak as necessary.

android:accessibilityEventTypes="typeWindowContentChanged|typeWindowsChanged|typeWindowStateChanged"

Roughly speaking, this is the subset of events that will allow you to catch all screen change events. Open a new window... Scan the view hierarchy!

android:accessibilityFlags="flagReportViewIds|flagIncludeNotImportantViews"

Flag include not important views will include MANY more views in the AccessibilityNodeInfo hierarchy. Specifically, many layout views and such that the Android OS normally would not deem necessary for accessibility purposes. I like to include this, as this is also a developer adjustable property and Android developers are notoriously unaware when it comes to accessibility. Best to just fetch everything and sort through yourself!

Okay, so there's your service config! Now, the rest is quite easy. What you want to do is recurse down through the root nodes children until you find a TextView node. I have set up a silly service below that would find the first TextView node on each screen update (unless they come more than once per second given the service config XML above), and then log its screen coordinates.

class MyAccessibilityService extends AccessibilityService {

    @Override
    public void onAccessibilityEvent(AccessibilityEvent e) {

        //You can actually call getRootInActiveWindow() any time!
        //Doing so here ensures that the screen is in a reasonable state
        //and not in the middle of rendering or something else silly.
        final AccessibilityNodeInfo textNodeInfo = findTextViewNode(getRootInActiveWindow());

        if (textNodeInfo == null) return;

        Rect rect = new Rect();

        textNodeInfo.getBoundsInScreen(rect);

        Log.i(A11yService.class.getSimpleName(), "The TextView Node: " + rect.toString());

    }


    public AccessibilityNodeInfo findTextViewNode(AccessibilityNodeInfo nodeInfo) {

        //I highly recommend leaving this line in! You never know when the screen content will
        //invalidate a node you're about to work on, or when a parents child will suddenly be gone!
        //Not doing this safety check is very dangerous!
        if (nodeInfo == null) return null;

        Log.v(A11yService.class.getSimpleName(), nodeInfo.toString());

        //Notice that we're searching for the TextView's simple name!
        //This allows us to find AppCompat versions of TextView as well
        //as 3rd party devs well names subclasses... though with perhaps
        //a few poorly named unintended stragglers!
        if (nodeInfo.getClassName().toString().contains(TextView.class.getSimpleName())) {
            return nodeInfo;
        }

        //Do other work!

        for (int i = 0; i < nodeInfo.getChildCount(); i++) {
            AccessibilityNodeInfo result = findTextViewNode(nodeInfo.getChild(i));

            if (result != null) return result;
        }

        return null;
    }

    //Required for a valid Accessibility Service.
    @Override
    public void onInterrupt() {}
}

You can also seek out the open source library I built, called Accessibility Service Utilities, that makes ALL of this stuff much easier! The A11yNodeInfoMatcher class is what you'd want.